Bayesian Priors Explained: A/B Testing Guide
Bayesian priors in A/B testing explained with real numbers: informative vs uninformative priors, how each reshapes the posterior, and when to use each.
📚 This article is part of the guide A/B Testing Statistical Significance Guide.
Every Bayesian calculation starts with a choice that most articles on the topic quietly skip past: what belief you already held before seeing your test’s data. That belief is the prior, and it is not a technical footnote, it is the piece that decides how fast, or how slowly, your conversion rate estimate reacts to the traffic that comes in. This guide is a statistical deep dive that complements our Bayesian A/B testing guide, which already covers priors, posteriors, and the probability that B beats A at the level a working practitioner needs. It is also a companion to the multi-armed bandits vs A/B testing guide, the piece that explained why Thompson Sampling, the most common Bayesian algorithm in production bandits, depends on exactly this prior-to-posterior mechanism to decide, visitor by visitor, where to send traffic.
Here the focus is only the prior: what it is, the practical difference between an uninformative prior and an informative one, how each mathematically reshapes the calculated posterior, and, most importantly, when each type is actually worth using. Every number below is calculated, not described by feel.
What a prior is, without the hand-waving
In Bayesian inference you never start from absolute zero. Before you look at a single visitor in your test, you already hold, implicitly or explicitly, some belief about the plausible range of the conversion rate. The prior is that belief, formalized as a probability distribution over the possible values of the rate.
For conversion rates, a number between 0% and 100%, the Beta distribution is the standard choice, because it is conjugate with the Binomial model that describes conversions and non-conversions: when you update a Beta prior with binomial data, the posterior also comes out as a Beta, with a simple, exact formula, no simulation required. The update rule is:
In other words: you add the observed conversions to the alpha parameter and the observed non-conversions to the beta parameter, and that is your new belief distribution. It is the same arithmetic behind this blog’s Bayesian calculator and behind Thompson Sampling in bandits.
Uninformative prior vs informative prior
The choice of prior comes down, in practice, to one question: how much weight should this prior belief carry against the data that is arriving right now?
Uninformative (weak) prior: the classic example is Beta(1,1), also called the uniform prior. It assigns the same probability density to every conversion rate between 0% and 100%, meaning it does not favor any specific value before seeing data. In weight terms, it is equivalent to having observed just one conversion and one non-conversion before starting, practically nothing. This prior lets “the data speak for itself”: the posterior converges toward the observed rate very quickly.
Informative (strong) prior: a practical example would be Beta(20,180). That prior’s mean is alpha divided by the sum of alpha and beta, so 20 divided by 200, which is 10%. It embeds the belief that the conversion rate sits around 10%, with a confidence equivalent to having observed 20 conversions out of 200 prior visits (that is how you read the “weight” of a Beta prior: alpha plus beta acts like a virtual sample size, already lived through before the current test began). The larger that sum, the more concentrated the distribution is around its mean, and the more weight it carries against new data.
The figure below compares the two density curves side by side, before any update with real data:
Notice the asymmetry: the uniform prior is a nearly flat, low line with no preference, while the informative prior already draws a concentrated “opinion” before a single visitor has been counted.
How prior and observed data form the posterior
The path from prior to posterior always follows the same flow, and it helps to see it before diving into numbers:
Note that the formula is identical for both types of prior. The entire difference lives in the starting point: a Beta(1,1) starts from almost nothing, a Beta(20,180) starts already carrying the weight equivalent to 200 prior observations. That initial weight is what determines how much the posterior resembles the prior versus the new data.
A worked example: same evidence, two different priors
Let’s calculate this for real, not describe it. The scenario: a new test received 100 visitors and 8 conversions, a raw observed rate of 8.00%. We will update the two priors shown above with exactly this same data, using the conjugate update rule.
Uninformative prior, Beta(1,1): Posterior = Beta(1 + 8, 1 + 92) = Beta(9, 93). Posterior mean = 9 divided by (9 + 93) = 9/102 ≈ 8.82%.
Informative prior, Beta(20,180): Posterior = Beta(20 + 8, 180 + 92) = Beta(28, 272). Posterior mean = 28 divided by (28 + 272) = 28/300 ≈ 9.33%.
| Prior | Prior alpha, beta | Prior mean | Posterior after 8 conversions in 100 | Posterior mean |
|---|---|---|---|---|
| Uninformative | Beta(1, 1) | 50.00% | Beta(9, 93) | 8.82% |
| Informative | Beta(20, 180) | 10.00% | Beta(28, 272) | 9.33% |
With exactly the same evidence, the two posteriors land on different numbers. The weak prior’s posterior sits close to the raw observed rate (8.00%), nudged up only slightly by the leftover pull of the uniform prior. The strong prior’s posterior sits closer to 10%, the mean it already carried before any visitor was counted, because the prior’s weight (200 pseudo-observations) still outweighs the new data’s weight (100 real visitors) by a wide margin.
The same effect shows up in each posterior’s uncertainty. Using the variance of a Beta distribution and a normal approximation for the 95% interval:
| Posterior | Standard deviation | Approximate 95% interval |
|---|---|---|
| Beta(9, 93), weak prior | ≈ 2.80 percentage points | ≈ 3.3% to 14.3% |
| Beta(28, 272), strong prior | ≈ 1.68 percentage points | ≈ 6.0% to 12.6% |
Notice the detail that usually goes unnoticed: the strong prior’s posterior looks “more confident” (a narrower interval) simply because it adds 200 virtual observations to the 100 real ones, reaching an effective total of 300. That confidence is only justified if the 10% belief baked into the prior actually matched reality. If the true rate is really 8%, that narrower interval is, in practice, more confident around a slightly wrong value, which is exactly the risk of a badly calibrated prior, covered in detail further below.
Try it yourself: the same math, with your own numbers
The calculator below runs on the same Beta-Binomial engine as this example, with the product’s default prior: a uniform Beta(1,1), uninformative, the safe choice whenever there is no strong, reliable prior belief to embed. Paste the visitors and conversions for A and B from your own test and watch the posterior calculate live:
Beta-Binomial model with a uniform Beta(1,1) prior and a 95% credible interval. Deterministic calculation, updates live.
If you wanted to apply an informative prior such as the Beta(20,180) from this guide, the math behind it would follow the same update rule, just adding that initial weight of 20 conversions and 180 non-conversions to alpha and beta before adding the new data, exactly as we did by hand above. That is why a calculator with a fixed uniform prior tends to return a posterior closer to the raw observed rate than a calculator loaded with a strong prior would return for the same data.
How much traffic until the prior stops mattering
A direct way to see the difference between the two priors is to ask: in the same scenario (true rate of 8%), how many visitors are needed before each prior’s posterior converges near the true value? Carrying the update formula forward (posterior = prior plus data, with the observed rate holding at 8% of accumulated traffic), the table below shows each prior’s posterior mean at different sample sizes:
| Cumulative visitors | Conversions (≈8%) | Posterior mean, weak prior Beta(1,1) | Posterior mean, strong prior Beta(20,180) |
|---|---|---|---|
| 20 | 2 | 13.64% | 10.00% |
| 100 | 8 | 8.82% | 9.33% |
| 500 | 40 | 8.17% | 8.57% |
| 2,000 | 160 | 8.04% | 8.18% |
| 10,000 | 800 | 8.01% | 8.04% |
The pattern is clear and holds for any Beta prior: the larger the sum of the prior’s alpha and beta (the “virtual sample size” it carries), the more real visitors are needed for the data to overtake that initial belief. A weak prior yields to the data almost immediately. A strong prior only yields once real traffic accumulates that is comparable to, or larger than, the weight it already carried in.
When to use an informative prior, and when to use a weak one
The choice is not aesthetic, it is a risk decision. An informative prior tends to pay off when:
- A reliable, comparable track record exists. You have already run dozens of similar tests (same type of page, same type of change, similar audience) and have a real distribution of observed conversion rates from those past tests, not a guess.
- Traffic in the current test is low and the decision cannot wait. A well-calibrated prior reduces estimate variance in the early weeks, when very little data has arrived yet, at the cost of being slower to correct if the belief turns out wrong.
- The tested change is a small variation of something already tested, not a brand-new idea. Reusing knowledge from past CTA button tests for a new CTA button test is reasonable; using that same prior for a full checkout redesign is not.
A weak or uniform prior is the safer choice when:
- No reliable comparable track record exists, or the existing track record comes from a very different context (different audience, different offer, different traffic channel).
- The tested change is genuinely new, with no similar precedent to base a specific belief on.
- You prioritize letting the data speak for itself, even if that means a noisier initial estimate in the test’s early weeks.
When in doubt, a weak prior errs on the cheaper side: it takes a bit longer to sharpen the estimate in the first weeks, but it never pushes the decision toward a value you only “thought” was true.
The risk of a badly calibrated prior
The numeric example above showed the upside of a well-calibrated informative prior: less noise in a test’s early weeks. The downside shows up when the belief baked into the prior is simply wrong.
Imagine the same strong prior, Beta(20,180), with its 10% belief, except the new page’s true rate is actually 3%, much lower than the track record suggested (for example, because this specific test’s audience is colder, or the offer is less familiar). With 100 visitors and about 3 conversions (holding the 3% proportion), the posterior would be Beta(20 + 3, 180 + 97) = Beta(23, 277), with a mean of 23 divided by 300, or approximately 7.67%, more than double the true rate. A weak prior, updated with the same numbers (Beta(1 + 3, 1 + 97) = Beta(4, 98)), would land at a mean of 4 divided by 102, approximately 3.92%, much closer to reality.
This is the exact mechanism of bias from a badly calibrated prior: it is not a random error that cancels out quickly with more data, it is a systematic push toward the prior belief, one that only corrects at the speed with which real data outweighs the prior’s virtual weight. The stronger the wrong prior, the more expensive the mistake, and the longer a business decision can rest on an inflated or understated number without anyone noticing, because the posterior keeps looking “confident” (narrow interval) the entire time, just confident around the wrong value.
The practical rule: using an informative prior is a bet that the prior belief is correct. When it is, you gain speed and less noise. When it is not, you pay in systematic bias, and the stronger prior is precisely the one that takes longest to correct.
Automate This in Donnu
Choosing the right prior is only half the Bayesian work: the other half is running the calculation without getting the formula wrong, without mixing up alpha and beta, and without discovering too late that the chosen prior was pulling the decision the wrong way. Donnu already runs on a native Bayesian engine with a uniform prior by default, exactly the safest choice for most real tests, without requiring you to calculate posteriors by hand to trust the verdict.
Start a 14-day free trial and watch the Bayesian posterior get calculated correctly with every new visitor. To see this same prior-to-posterior mechanism at work in real-time traffic allocation, check out the multi-armed bandits vs A/B testing guide and the complete Bayesian A/B testing guide. If you’d rather review the frequentist foundation of significance first, see the guide to A/B testing statistical significance, or leia em português.
References
- Jeffreys, H. An Invariant Form for the Prior Probability in Estimation Problems. Proceedings of the Royal Society of London, Series A, 186, 453-461, 1946. royalsocietypublishing.org.
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. & Rubin, D. B. Bayesian Data Analysis, 3rd edition (full text available free). stat.columbia.edu/~gelman/book.
- VWO. Enhanced SmartStats, Bayesian powered sequential testing engine. vwo.com/why-us/technology/statistics.
- Lyst Engineering. Bayesian AB Testing (how to specify a Beta prior’s alpha and beta in production, with worked numeric examples). making.lyst.com.
Frequently asked questions
- What is a Bayesian prior, in simple terms?
- It is the belief about a conversion rate before you look at any data from the current test. It is represented as a probability distribution, almost always a Beta distribution for conversion rates, and it is combined mathematically with the observed data to produce the posterior, the updated belief after seeing the evidence.
- What is the difference between an informative and an uninformative prior?
- An uninformative prior, such as the uniform Beta(1,1), treats every conversion rate between 0% and 100% as equally likely before any data arrives, so the posterior ends up guided almost entirely by the observed evidence. An informative prior, such as a Beta(20,180) built from past tests, already embeds a belief concentrated around a specific value (near 10% in that example), and that belief only gives way gradually as more data arrives.
- Can a badly calibrated prior bias the outcome of a real test?
- Yes, and that is the main risk of using an informative prior without solid justification. If the prior belief is wrong, for example inherited from a test with a different audience or offer, the posterior gets pulled toward an incorrect value, and the stronger the prior, the more data is needed to correct that bias. A weak prior errs less in this respect, because it yields to evidence almost immediately.
- When does it make sense to use an informative prior instead of a uniform one?
- When there is a reliable, comparable track record, for example dozens of past tests on the same type of page measuring the same type of change, with an already known distribution of conversion rates. Outside that scenario, a weak or uniform prior is the safer choice, because it does not impose a prior opinion that may simply be wrong for the new case.
- What kind of prior does the Bayesian calculator on this blog use?
- By default, a uniform Beta(1,1) prior, uninformative: every possible conversion rate starts out equally likely, and the posterior is guided by the numbers you enter. It is the safe choice whenever you do not have a strong, reliable prior belief to embed in the calculation.