Credible Interval vs Confidence Interval
Credible interval vs confidence interval: the real difference between the Bayesian and frequentist reading of the same range, with a full numeric example.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
A 95% frequentist confidence interval and a 95% Bayesian credible interval usually land on nearly identical numbers, and yet they answer two completely different questions. The first describes how a method behaves if you repeated it over and over; the second describes the actual probability of the true value, given what you observed. The sentence almost everyone reaches for, “there’s a 95% chance the true rate falls in here,” is only true of the second one. Applied to the first, it’s a misreading so common that studies testing trained researchers find it in the majority of answers. This guide draws the real line between the two, using one worked example computed by the same two statistical engines already running on this blog, the frequentist one and the Bayesian one, side by side, on the exact same pair of numbers.
What each interval actually answers
The mix-up starts with an unlucky coincidence: the two intervals usually cover roughly the same stretch of numbers, but the question each one is answering is entirely different.
The frequentist confidence interval answers: “if I reran this exact experiment an enormous number of times, drawing a fresh sample each time and building a new interval the same way, 95% of those intervals would contain the true value.” The confidence claim is about the procedure, not about the one specific interval sitting on your screen right now. Once that particular interval is calculated, it either already contains the true value or it doesn’t, there’s no probability left to spread around inside it. The true conversion rate isn’t treated as a random variable under the frequentist view, it’s a fixed (if unknown) number; what varies from sample to sample is the interval itself.
The Bayesian credible interval answers: “given the data I observed and the prior belief I started with, there’s a 95% probability the true value falls inside this exact interval.” Here the true conversion rate is treated as genuinely uncertain, and that uncertainty is modeled as a probability distribution, the posterior. The credible interval is, literally, the slice of that distribution holding 95% of the probability mass. This is the reading that most people, without meaning to, already assign to the ordinary confidence interval.
Why this mix-up is so common
The historical root is simple: the two intervals came out of different schools of statistics (the frequentist school, built by Jerzy Neyman in the 1930s; the Bayesian school, which updates beliefs with Bayes’ theorem), yet the plain-language name people give to the result is almost always the Bayesian version, even when the calculation underneath is frequentist. Saying “I’m 95% confident the value is in here” sounds completely natural to say out loud, it’s just that the sentence is only logically correct for the credible interval.
This isn’t a rhetorical exaggeration for the sake of the article. Hoekstra, Morey, Rouder and Wagenmakers asked 120 researchers and 442 psychology students to judge six statements about a confidence interval, all of them false by construction. Both researchers and students agreed, on average, with more than three of the six wrong statements, including the most common one of all: “there is a 95% probability that the true value lies within this particular interval.” According to the authors, the misreading was just as frequent among experienced statisticians as it was among beginners.
The second reason is numerical, and it’s the heart of this article: in many practical cases, especially with a weak Bayesian prior and a reasonably sized sample, the two intervals come out with almost identical values. When the numbers agree, it’s an easy (and wrong) leap to assume the meaning agrees too. The next section shows exactly that coincidence, with real numbers.
A side-by-side example, with real numbers
To keep this concrete instead of abstract, we’ll reuse the same pair of numbers from our statistical significance guide: the control (A) had 210 conversions out of 4,200 visitors; the variant (B) had 273 out of 4,200. Both blocks below came straight out of the two engines already running the calculators on this blog, the frequentist significance() function and the Bayesian bayesianAbTest() function (a Beta posterior with a uniform Beta(1,1) prior), with nothing rounded off by ear.
Each version’s rate: both intervals, side by side
| Version | Observed rate | Confidence interval (Wilson, 95%) | Credible interval (95%) |
|---|---|---|---|
| A (control) | 5.00% | 4.38% to 5.70% | 4.38% to 5.70% |
| B (variant) | 6.50% | 5.79% to 7.29% | 5.79% to 7.29% |
Notice that A’s two intervals virtually coincide (the difference only shows up in the fourth decimal place), and the same is true for B. That happens because the Bayesian prior used here is the flattest one possible, uniform Beta(1,1), and a sample of 4,200 visitors per side is large enough for the evidence to swamp whatever pull the prior has. Even so, the question each interval in this table is answering is still a different one: the confidence interval speaks about the method; the credible interval speaks about the actual probability that B’s true rate sits between 5.79% and 7.29%, given that you observed 273 conversions out of 4,200 visitors.
Where A and B actually diverge: the two engines disagree on the real question
Where the numeric coincidence stops is on the question “is B actually better than A?”. Running the frequentist engine on the difference between the two rates gives a z-score of 2.95, a two-sided p-value of 0.0031, and a 95% confidence interval on the difference of +0.50 to +2.50 percentage points, a significant result (well under the 0.05 threshold). That calculation answers: “if A and B were truly identical, the odds of seeing a gap this large by chance alone would be 0.31%.”
Running the Bayesian engine on the exact same pair of numbers reframes the answer entirely: the probability that B genuinely beats A comes out to 99.8%, the relative improvement between the posterior means is +29.9%, and the expected loss of deciding for B (the expected regret if A actually turns out to be better) is only 0.00 percentage points (the exact figure is 0.0002pp, rounded down by the calculator), against 1.50 percentage points of expected loss for deciding on A instead. That calculation answers the question most decision-makers are actually asking: “what are the odds I’m right choosing B, and how much am I risking if I’m wrong?”
Check both sides yourself with the same numbers (210 conversions out of 4,200 visitors for A; 273 out of 4,200 for B). First, the frequentist read:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Now the same pair of numbers, read the Bayesian way:
Beta-Binomial model with a uniform Beta(1,1) prior and a 95% credible interval. Deterministic calculation, updates live.
Notice that the two blocks agree on the central fact, B wins, comfortably, but only the second one hands back a sentence that reads like “there’s a 99.8% chance B beats A.” The first one will never phrase it that way, no matter how tempting the p-value makes it look.
Where this difference actually matters in practice
For most day-to-day decisions, the gap between the two concepts is academic: both agree on the direction of the effect, and the team moves forward. But there are three concrete situations where the difference stops being semantic and turns into a real communication risk:
- Reporting a result to a non-technical audience. A statistician knows “95% confidence” isn’t a probability statement about the true value, but a manager, an investor, or a client will almost always read it that way, because it’s the more natural interpretation in plain English. If the number in front of you is frequentist, communicating it as if it were Bayesian promises a certainty the math doesn’t back up.
- Comparing the risk of two decisions. “B has a 99.8% chance of being better, and the risk of getting it wrong by choosing B is only 0.0002 percentage points (0.00pp on the calculator)” is an actionable sentence that already bakes in the size of the possible mistake. The frequentist equivalent, “the p-value is 0.0031”, requires an extra interpretive step that most teams either skip or get wrong.
- Explaining why a test came back inconclusive. An overly wide confidence interval is easy to describe as “we don’t know,” but a wide credible interval lets you go further: it can say, for instance, “there’s a 62% chance B is better, but not confidently enough yet to decide,” a graded statement the p-value alone can’t offer.
| What you want to say | Can you say it with a confidence interval? | Can you say it with a credible interval? |
|---|---|---|
| “There’s a 95% chance the true value is in here” | No. It’s the most common misreading, per Hoekstra et al. (2014) | Yes, that’s exactly its definition |
| “If I reran this test many times, 95% of intervals built this way would contain the true value” | Yes, that’s the correct definition | No, that isn’t the question this interval answers |
| “B has an X% chance of being better than A” | Not directly (requires transforming the p-value, which most people don’t do correctly) | Yes, it’s the direct output of the model |
| “The expected cost of deciding wrong is Y percentage points” | Not part of the calculation | Yes, that’s the expected loss |
One more technical distinction: where each interval actually comes from
It’s worth spelling out where each number originates, because that’s what explains why the coincidence in the example above isn’t guaranteed in every scenario. The Wilson confidence interval used on this page is built off the normal curve that approximates the binomial distribution, with no prior belief involved at all, just the count of conversions and visitors. The credible interval comes from a Beta(1 + conversions, 1 + non-conversions) posterior, obtained by combining the uniform Beta(1,1) prior with that same observed data, then cut at the 2.5% and 97.5% percentiles of that distribution, the so-called equal-tailed cut, the most common approach and the one this blog’s Bayesian calculator uses. There’s a second way to cut a credible interval, the HPD (highest posterior density) interval, which searches for the narrowest possible interval that still holds 95% of the probability mass; for a symmetric posterior like the one in this example, the two cuts land almost on top of each other, but for a strongly skewed posterior (little data, a rate near 0% or 100%) they can pull apart noticeably. The weak prior and large sample in our worked example are exactly the condition under which the confidence interval and the credible interval converge numerically; with a stronger prior or a smaller sample, the two numbers can drift much further apart from each other, even while the interpretation of each one stays exactly the same.
Confidence interval and credible interval aren’t the same ruler with a different label
One sibling misunderstanding is worth closing out here too: neither interval is inherently “more rigorous” than the other. Our statistical significance guide already covers the z-test formula and the confidence interval on the difference; our Bayesian A/B testing guide covers how the choice of prior shapes the posterior that generates the credible interval. Both calculations start from the same real data and arrive at defensible verdicts, each one consistent within the logic of its own school. The mistake isn’t choosing the frequentist or the Bayesian route, it’s speaking one’s sentence while thinking in the other’s meaning, which is exactly what most people do without noticing.
Do this automatically on Donnu
You’ve just seen why “95% confidence” and “95% chance the true value is in here” aren’t the same sentence, even when the numbers look identical. That’s exactly the kind of detail that turns into an expensive misunderstanding in a results meeting, someone promises a stakeholder a certainty the confidence interval never actually offered. Donnu removes that risk at the source because it already runs on a native Bayesian engine: the verdict you get back is already phrased as the question everyone actually asks (“what are the odds B is better, and how much am I risking if I choose wrong?”), so your team never has to translate a p-value into a sentence it can’t support.
Start a free 14-day trial and also read the complete guide to A/B testing statistical significance, the Bayesian A/B testing guide, and Thompson Sampling explained, which uses this same Bayesian posterior to route traffic in real time. Prefer to calculate on your own? Use the confidence interval calculator or the Bayesian A/B test calculator on this blog. Read this guide in Portuguese: Intervalo de Credibilidade x Confiança.
References
- Hoekstra, R., Morey, R. D., Rouder, J. N. & Wagenmakers, E.-J. Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21(5), 1157-1164, 2014. pubmed.ncbi.nlm.nih.gov/24420726.
- Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D. & Wagenmakers, E.-J. The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103-123, 2016. pubmed.ncbi.nlm.nih.gov/26450628.
- GrowthBook. How to interpret a confidence interval, step by step. growthbook.io/insights/how-interpret-confidence-interval-step-by-step.
Frequently asked questions
- Do a confidence interval and a credible interval give you the same number?
- Often yes, or close enough, when the Bayesian prior is weak (a uniform Beta(1,1), for instance) and the sample is reasonably large: the two calculations converge on nearly identical numeric ranges. What changes completely is what that number is allowed to mean. That numeric coincidence is exactly what fuels the confusion between the two concepts.
- Can I say there is a 95% chance the true rate falls inside my confidence interval?
- Not formally, no. That is the Bayesian interpretation, and it belongs to the credible interval, not the frequentist confidence interval. Once a confidence interval is calculated, the true rate either falls inside it or it does not; there is no probability left to assign to that specific interval. The phrase "95% confidence" describes the method used to build the interval, not the interval itself.
- Why does this distinction matter if the numbers usually look similar?
- Because business decisions often hinge on a direct probability claim ("what are the odds this change is real?"), and only the credible interval answers that honestly. Using the wrong language with a stakeholder can make you sound more, or less, certain about a result than the math actually supports.
- Does the credible interval depend on which prior you choose?
- Yes, and that is the main structural difference between the two. A confidence interval uses no prior at all, only the observed data. A credible interval combines the data with a prior belief; with a weak, uniform prior like the Beta(1,1) used on this blog, that influence is small and fades quickly as the sample grows, but with a strong prior the two intervals can diverge by a lot more.
- Which one should I actually use for my A/B test?
- Both are defensible, and they are not in competition: the confidence interval is the standard of classical statistics and formal reporting, while the credible interval tends to communicate better with decision-makers, because it answers the exact question most people already assume they're getting. The real mistake is not picking one, it's speaking one's language while thinking in the other's meaning.