Bayesian A/B Testing: The Complete Guide (vs Frequentist)

Bayesian A/B testing is a method of analyzing experiments that treats conversion rates as probability distributions updated by data, producing outputs like “B has a 93% probability of beating A” instead of a frequentist p-value. Where the classic approach asks “how surprising would this data be if there were no real difference?”, the Bayesian approach asks “given what I believed before and what I’ve now observed, how likely is it that B is actually better, and by how much?”

This guide is the companion to our complete guide to A/B testing statistical significance, which covers the frequentist z-test and p-value in full. Here we cover the Bayesian side on its own terms: the mechanics of prior, likelihood, and posterior; the decision metrics that actually drive a ship or no-ship call; a fully worked numeric example; and the single biggest misconception about Bayesian testing, that you can peek whenever you like with no consequences. You can’t, not entirely, and pretending otherwise is where most vendor explanations quietly go wrong.

What is Bayesian A/B testing, exactly?

In a Bayesian A/B test, each variant’s true conversion rate is treated as an unknown quantity described by a probability distribution, not a single fixed number you’re trying to estimate with a point guess. You start with a prior distribution representing what you believed before seeing any data from this test. You observe visitors and conversions. You combine the two using Bayes’ rule to get a posterior distribution, your updated belief about each variant’s true rate given the evidence.

Once you have posteriors for A and B, you can answer the questions marketers actually ask directly: What is the probability B is better than A? If I pick B and I’m wrong, how much do I lose on average? What’s a plausible range for the real lift? None of these require translating a p-value into business language, because the output is already a probability statement about the thing you care about.

Frequentist vs Bayesian A/B testing: the core difference

The two schools differ in what “probability” even means. Frequentist statistics treats probability as a long-run frequency: a p-value describes how often you’d see data this extreme across many repeated experiments if there were truly no effect. Bayesian statistics treats probability as a degree of belief: the posterior directly describes how confident you should be that B beats A, given your prior and this specific dataset.

	Frequentist	Bayesian
Interpretation of probability	Long-run frequency across repeated experiments	Degree of belief given the data observed
Core output	p-value, significant or not	Posterior distribution, P(B beats A)
Reads like	“5% chance this pattern appears with no real effect”	“93% chance B is genuinely better than A”
Interval	Confidence interval (about the procedure)	Credible interval (about the parameter itself)
Peeking	Requires sequential-testing correction to stay valid	More interpretable at any N, but not immune to optional stopping
Sample size	Fixed in advance from baseline, MDE, alpha, power	No hard minimum, but posterior needs enough data to be precise
Prior information	Not used	Explicit input; can be uninformative or informative
Industry status	Long-standing academic and industry default	Growing, especially in product-led SaaS tools

Neither column is the “correct” one. A well-run frequentist test and a well-run Bayesian test on the same clean data will usually point to the same practical conclusion, they are different lenses on the same evidence, not competing claims about reality.

Prior, likelihood, posterior: the loop that does the work

Every Bayesian calculation runs the same three-part loop: start with a prior belief, update it with a likelihood (how probable the observed data is under different possible true rates), and land on a posterior belief that blends the two.

Why the Beta distribution models conversion rate

Conversion is a binary outcome per visitor, convert or don’t, which makes it a Binomial process. The Beta distribution is the mathematically convenient prior for this case because it’s conjugate to the Binomial: combining a Beta prior with Binomial data produces another Beta distribution as the posterior, with no numerical integration required. A Beta distribution has two parameters, written Beta(alpha, beta), where alpha roughly tracks accumulated successes (conversions) and beta tracks accumulated failures (non-conversions).

The update rule is genuinely just addition. Starting from a prior Beta(alpha_0, beta_0), after observing “s” conversions out of “n” visitors, the posterior is:

A Beta(1,1) prior is uniform, it says “any conversion rate from 0% to 100% is equally plausible before I see data,” which is the standard uninformative starting point. An informative prior like Beta(12,88) instead encodes “I already believe the rate is around 12 ÷ (12+88) = 12%, with a certain amount of confidence,” which is useful when you have solid historical data for a page.

Choosing a prior, briefly

Most A/B testing tools default to an uninformative prior like Beta(1,1) precisely because it lets the data dominate the conclusion, which matches how marketers intuitively expect a fresh test to behave. An informative prior built from a page’s real historical conversion rate can sharpen early results, but a prior that’s too confident can “drown” a modest amount of new data: if your prior is very tightly concentrated around 12% and your new variant is actually converting at 18%, it takes more data to overcome that strong prior than it would with a flat one. When in doubt, start uninformative and treat informative priors as an optimization for experienced teams who have validated their historical baseline is stable.

The decision metrics that actually drive shipping

Once you have posteriors for A and B, four numbers do the practical work:

• Probability to beat control, P(B beats A): the share of simulated or integrated outcomes where B’s sampled rate exceeds A’s. This is what “B has a 93% probability of beating A” means. Important nuance: it is a statement about direction, not magnitude, a variant can have a 93% chance of being better while the actual size of that improvement remains uncertain.
• Expected lift and its credible interval: the average relative improvement of B over A across the posterior, plus a range (commonly 95%) that plausibly contains the true lift. This is the number that turns “B is probably better” into “B is probably better, by somewhere between X% and Y%.”
• Expected loss: the average amount of conversion rate you’d give up, across every plausible scenario in the posterior, if you picked a variant and it turned out to be the wrong choice. Unlike P(B beats A), expected loss is risk-aware, a variant can have a high chance of winning but a large expected loss if the downside scenario is severe, which matters when a wrong call is costly to reverse.
• Credible interval: the direct probability statement about the effect itself, for example “there is a 95% probability the true lift is between +0.5 and +9.5 percentage points, given this prior and this data.”

A related idea worth knowing is the region of practical equivalence (ROPE): instead of asking “is there any difference at all,” you define a band around zero that counts as “practically the same” (say, plus or minus 0.2 percentage points) and check how much of the posterior lift falls inside versus outside that band. It’s a way of encoding “statistically different” versus “different enough to matter” directly into the decision rule.

A fully worked example, start to finish

Control (A) had 200 conversions out of 2,000 visitors. Variant (B) had 230 conversions out of 2,000 visitors. Let’s walk the Bayesian read of these exact numbers, using an uninformative Beta(1,1) prior for both.

Step 1, raw rates. A converts at 200 ÷ 2,000 = 10.0%. B converts at 230 ÷ 2,000 = 11.5%. Relative lift on the raw numbers: (11.5 − 10.0) ÷ 10.0 = +15%. As always, the question is whether that gap is real or noise.

Step 2, build the posteriors. With a Beta(1,1) prior, the update rule from above gives:

Each posterior is a full distribution over “what the true conversion rate might be,” not a single number. The mean of a Beta(alpha, beta) distribution is alpha ÷ (alpha + beta), which gives posterior means of 201 ÷ 2002 ≈ 10.04% for A and 231 ÷ 2002 ≈ 11.54% for B, both extremely close to the raw rates because 2,000 visitors is enough data for the flat prior’s influence to be negligible.

Step 3, P(B beats A). In practice this is computed by drawing, say, 100,000 random samples from each posterior and counting how often the B sample exceeds the A sample (the Python snippet below does exactly this). For this dataset, that comes out to roughly P(B beats A) ≈ 94%, a solid but not overwhelming signal that B is the better variant.

Step 4, expected lift and credible interval. Averaging the relative lift (sample_B − sample_A) ÷ sample_A across those same simulated pairs gives an expected relative lift of roughly +15%, matching the raw-rate estimate, with a 95% credible interval on that lift of roughly −4% to +38%. The interval includes a real negative slice, which is consistent with a P(B beats A) in the low-to-mid 90s rather than the high 90s: there’s still a meaningful chance B is actually slightly worse.

Step 5, expected loss. Expected loss for choosing B, averaged over every simulated scenario where B underperforms A, comes out small in absolute conversion-rate terms, well under half a percentage point, because the scenarios where B loses tend to be scenarios where it loses by a little, not a lot. This is the number a risk-averse team should check before shipping: a high P(B beats A) with a low expected loss is a comfortable call; a high P(B beats A) with a large expected loss (which happens when the downside case is severe even if unlikely) deserves more caution.

The decision. With P(B beats A) around 94% and a small expected loss, many teams would lean toward shipping B while explicitly flagging that the bar for “confident” hasn’t fully been cleared, and a team with a stricter risk tolerance might reasonably choose to keep collecting data instead. The credible interval still leaves real room for a marginal or even flat outcome. That nuance, “probably a real win, with a quantified and non-trivial sliver of doubt,” is exactly what a single p-value or a blunt “significant / not significant” label doesn’t communicate.

For comparison: the frequentist read on the same numbers

The same 200/2,000 vs 230/2,000 dataset, run through a standard two-proportion z-test, gives a useful side-by-side reference. Paste the numbers into the calculator below to see the p-value, confidence interval, and verdict our statistical significance guide walks through in full:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

You’ll typically find the z-test lands well above the conventional 0.05 threshold on this dataset, technically “not significant” under a strict frequentist read, even though the Bayesian posterior above already leans fairly confidently toward B. That gap between the two verdicts on the same data is a good illustration of why “probably true, moderate confidence” is often a more honest summary than a hard pass/fail line in either framework.

When can you stop? Peeking, and the honest truth

This is the part vendor marketing pages tend to oversimplify in one direction or the other, so it’s worth stating carefully.

The myth: “Bayesian testing lets you peek at results anytime with zero consequences, unlike frequentist testing.” This gets repeated because it captures something real, then overstates it.

What’s actually true: A Bayesian posterior is a valid, correctly calibrated summary of the evidence at any sample size, you’re never “cheating the math” by looking. That’s a genuine, useful property frequentist fixed-horizon tests don’t share without extra machinery. But if your decision rule is “keep checking P(B beats A) and stop the very first moment it crosses 95%,” you are running an optional stopping procedure, and optional stopping inflates your practical error rate regardless of which statistical framework computes the number you’re stopping on. The posterior itself isn’t biased by looking at it; your decision procedure of looking repeatedly and stopping at the first favorable moment is what introduces the bias, and that’s a property of the stopping rule, not of Bayesian math specifically.

The nuanced version: Bayesian testing is more interpretable at any N, the posterior always means what it says. It is not automatically immune to the practical effects of optional stopping unless your stopping rule was specifically designed to account for it (sequential Bayesian methods, pre-registered expected-loss thresholds, or similar always-valid constructions). A team that peeks constantly and stops the moment P(B beats A) looks good is still more likely to call false wins than a team that commits to a stopping rule in advance, in exactly the same spirit as frequentist peeking inflating false positives.

Practical stopping rules that hold up:

• Pre-commit to a P(B beats A) threshold (commonly 95% or higher) combined with a minimum sample size, rather than stopping on threshold alone.
• Use an expected-loss tolerance: stop when expected loss falls below a level you’ve decided in advance is an acceptable risk, since this accounts for the cost of being wrong, not just the odds of being right.
• Define a ROPE in advance and stop when enough of the posterior lift falls outside it in one direction.
• For a rigorous “check whenever you like” guarantee, use a sequential testing framework purpose-built for that: group sequential designs or always-valid confidence sequences, which spend your error budget deliberately across looks instead of assuming any-time peeking is free.

Do you need a sample size in the Bayesian world?

The “Bayesian tests need no sample size” claim is a myth worth retiring. It’s true there’s no rigid pre-registered N the way a frequentist fixed-horizon test requires before it’s considered valid, you can compute a posterior after your very first visitor. But a posterior from 20 visitors per variant is enormously wide and unstable, P(B beats A) can swing from 60% to 90% with the next handful of conversions. What you actually need is enough data for the posterior to narrow to a range precise enough to act on, which in practice tracks the same underlying drivers as frequentist sample size: your baseline rate, the effect size you care about detecting, and how much uncertainty you’re willing to tolerate before deciding. Running both a Bayesian and frequentist sample-size calculation on the same assumptions usually lands in a similar traffic ballpark, because both are ultimately limited by the same amount of real-world evidence.

As with frequentist testing, also plan to run the test across full business cycles, at least one to two weeks, so weekday/weekend and payday effects don’t distort the posterior. A wide credible interval that’s actively narrowing as data comes in is a sign you’re on track; one that’s stayed wide for weeks is a sign your traffic can’t support the effect size you’re chasing without a bigger, bolder change.

Run the numbers yourself: a Python beta-binomial snippet

The Beta-Binomial update and the Monte Carlo estimate of P(B beats A), expected lift, and expected loss are simple enough to run yourself with nothing but the Python standard library. This is a reference snippet to run locally, it is not a live tool embedded on this page.

import random

def simulate_beta(alpha, beta_param, n_samples, rng):
    # Uses random.betavariate, part of Python's standard library.
    return [rng.betavariate(alpha, beta_param) for _ in range(n_samples)]

def bayesian_ab_test(conv_a, n_a, conv_b, n_b, prior_alpha=1, prior_beta=1, n_samples=100_000, seed=42):
    rng = random.Random(seed)

    # Posterior = Beta(prior_alpha + conversions, prior_beta + non-conversions)
    post_a = (prior_alpha + conv_a, prior_beta + (n_a - conv_a))
    post_b = (prior_alpha + conv_b, prior_beta + (n_b - conv_b))

    samples_a = simulate_beta(*post_a, n_samples, rng)
    samples_b = simulate_beta(*post_b, n_samples, rng)

    b_wins = sum(1 for a, b in zip(samples_a, samples_b) if b > a)
    prob_b_beats_a = b_wins / n_samples

    lifts = [(b - a) / a for a, b in zip(samples_a, samples_b)]
    expected_lift = sum(lifts) / n_samples

    # Expected loss of choosing B: average shortfall in the scenarios where A was actually better.
    loss_if_b = [max(a - b, 0) for a, b in zip(samples_a, samples_b)]
    expected_loss_b = sum(loss_if_b) / n_samples

    return {
        "prob_b_beats_a": prob_b_beats_a,
        "expected_relative_lift": expected_lift,
        "expected_loss_b": expected_loss_b,
    }

result = bayesian_ab_test(conv_a=200, n_a=2000, conv_b=230, n_b=2000)
print(result)

Running this with the worked example’s numbers reproduces the same ballpark as Step 3 above: a P(B beats A) in the low-to-mid 90s percent range, an expected relative lift near +15%, and a small expected loss for choosing B. If you have scipy available, scipy.stats.beta offers the same distribution with closed-form mean, variance, and interval methods, useful if you’d rather compute exact quantiles than simulate them.

Beyond two variants, and beyond conversion rate

Everything above assumes one control, one variant, and a binary conversion outcome, but two extensions come up constantly in practice. With A/B/n tests (a control plus several variants), you can compute “probability to beat control” for each variant separately, or “probability of being the best of all variants,” and the two questions can have different answers, more variants in play means more chances for one to look good by chance, so treat multi-variant results with the same caution a frequentist multiple-comparisons correction implies. For continuous outcomes like revenue per visitor instead of a binary convert/don’t-convert, the Beta-Binomial pairing above doesn’t directly apply, that typically calls for a different conjugate model (commonly Gamma-Exponential style approaches) or simulation-based methods rather than the simple update shown here.

Bayesian A/B testing and multi-armed bandits

Bayesian A/B testing and multi-armed bandits (commonly implemented via Thompson Sampling) both build on the same posterior machinery, but they answer different questions. A/B testing is “test-and-ship”: run a fixed experiment, reach a confident conclusion, then send all traffic to the winner. A bandit is “explore-exploit”: it continuously shifts more traffic toward whichever variant currently looks best, using the same posterior sampling idea, without ever formally “concluding” the test. Bandits tend to minimize regret (traffic wasted on a losing variant) during the test itself, which suits short-lived campaigns or many simultaneous variants, while a fixed A/B test is generally preferred when you want a clean, well-documented, statistically defensible answer to report to a team or stakeholder before rolling out a permanent change.

When to choose Bayesian vs frequentist

Neither approach is a universal upgrade over the other, the choice usually comes down to who is reading the result and what’s at stake:

• Choose Bayesian when your audience is non-technical decision-makers who find “93% probability B is better” more actionable than a p-value, when you want an interpretable read at any point in the test (while still avoiding careless optional stopping), or when expected loss as a risk-aware stopping rule fits how your team thinks about downside.
• Choose frequentist when you need results defensible against long-standing academic or regulatory convention, when your organization already has fixed-horizon sample-size discipline built into its process, or when you’re publishing results externally where p-values remain the expected format.
• Either works when your test is clean (proper randomization, no SRM, full business cycles) and you have enough traffic. A poorly designed test produces a misleading answer in both frameworks; the statistical school doesn’t rescue bad data collection.

Common pitfalls and misconceptions

• Reading a credible interval like a confidence interval, or vice versa. A 95% credible interval is a direct probability statement about the parameter; a 95% confidence interval is a statement about the long-run behavior of the interval-construction procedure. They often look numerically similar with an uninformative prior, but they answer different questions.
• Treating P(B beats A) as the whole story. A variant can have a 90%+ chance of winning while the expected lift is tiny or the expected loss is uncomfortably large in the unlucky scenario. Check lift and loss alongside the win probability, not instead of it.
• An overconfident prior quietly overriding real data. A strong informative prior needs meaningfully more data to move than a flat one does, always sanity-check what a Beta(1,1) run on the same data would have said.
• Ignoring Sample Ratio Mismatch (SRM). If your traffic split arrived at 60/40 instead of the intended 50/50 with no good explanation, something is broken in randomization or tracking, and no posterior computed from that data can be trusted, however confident it looks.
• Novelty effects. A variant sometimes performs well for the first few days simply because it’s new and curious visitors interact with it more, an effect that a Bayesian posterior updates toward as more data arrives, but that early instability can look like early confidence if you’re not tracking how the posterior has moved over time.

Do this automatically on Donnu

You’ve just walked through what it actually takes to run a Bayesian A/B test honestly: building the posteriors correctly, reading P(B beats A) alongside expected loss instead of in isolation, and treating “you can peek anytime” as a nuanced truth rather than a free pass. Donnu builds Bayesian statistics into its reporting by default, with a lightweight snippet that never blocks your page and account-level data isolation, so you get an honest read without hand-rolling the Beta-Binomial math yourself.

Start a free 14-day trial and run your next experiment with statistics that hold up to scrutiny. For the frequentist side of this same coin, see our complete guide to A/B testing statistical significance, and for the full experiment workflow, see how to run an A/B test step by step.

References

• Kohavi, R., Tang, D. & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion site at experimentguide.com.
• Robinson, D. Understanding the Beta distribution (using baseball statistics). varianceexplained.org.
• Georgiev, G. Statistical Methods in Online A/B Testing. analytics-toolkit.com.
• Miller, E. How Not To Run an A/B Test (the peeking problem), 2010. evanmiller.org.