Bayesian A/B Test Calculator

This is the Bayesian read of your test. It answers "what is the probability that B is better than A" and "how much do I risk if I decide now", instead of the p-value. If you want the classic market standard (p-value, confidence interval and a yes or no verdict at 95%), use the significance calculator. Both read the same data through different lenses: pick this one when you want to decide by probability and risk.

How to use it

1. Enter visitors and conversions for A (the control) and for B (the variation).
2. Read the probability that B beats A in the hero: it is the direct chance that B is the better version.
3. Check the risk of choosing B (the expected loss): how much conversion you lose, on average, if you decide for B and it is worse.
4. Look at the posterior rates of A and B with the 95% credible interval, plus the relative lift.
5. Decide when the probability is high and expected loss is low. Both together, not one alone.

How it works: the Bayesian math

Each variation becomes a distribution of belief over the true conversion rate, not a single number. With the Beta-Binomial model, each arm's posterior is a Beta distribution:

Beta(1,1) is the uniform prior (before data, every rate is equally plausible). From the two posteriors, the probability that B wins is the chance a draw from B's distribution exceeds one from A, computed by integration: P(B > A) = ∫ f_B(x)·F_A(x) dx. The expected loss of choosing B is the average amount A beats B in the cases where A wins, E[(p_A − p_B)⁺].

Worked example (reproduces the default result)

With the pre-filled values: A with 100 conversions in 1,000 visitors and B with 120 in 1,000. The posteriors are Beta(101, 901) for A and Beta(121, 881) for B. The posterior means land at 10.08% (A) and 12.08% (B), a relative lift of +19.8%. Integrating the two distributions, the probability that B beats A is 92.3% (so 7.7% for A). The risk of choosing B, the expected loss, is only 0.05 percentage points. That is exactly what the tool shows above when you open the page: a 92% chance B is better, with near-zero average damage if we are wrong.

How to read it, and where it fools you

The probability that B wins is a direct statement: 92% means a 92% chance B is the better version, given what you observed. Expected loss is its companion: even at 92%, there is an 8% chance of being wrong, and expected loss measures the average size of that mistake. When it is tiny, deciding for B is cheap even under doubt. That is why the two numbers travel together: probability answers "how likely" and expected loss answers "how costly if I am wrong".

Limits to keep in mind: the math assumes a binary metric (converted or not), well-randomized traffic between A and B, and independence between the groups. It does not cover revenue per visitor (higher variance), strong seasonality, or several variations at once. And being Bayesian is not a license to peek without rules: set a probability threshold and an expected-loss threshold in advance, and close whole weekly cycles. For the verdict in the classic standard, cross-check with the significance calculator.

Best practices when deciding by probability

The strength of the Bayesian read is turning the test into a risk decision, not a blunt yes or no. To use that well, line up these points before you turn the test on.

• Combine two stop triggers: a high probability (say 95%) and an expected loss below a small amount you accept risking.
• Size the test anyway. The Bayesian view reads better with volume, so plan the sample with the sample size calculator.
• One primary metric per test. Chasing several at once erodes confidence in the decision.
• If the probability stays stuck near 50% and loss stays high, the test is telling you it cannot decide yet. Collect more instead of forcing a read.

Frequently asked questions

What does the Bayesian view answer that the p-value does not?

The question the business actually asks: what is the probability that B is better than A. The p-value answers something different and counterintuitive (the chance of seeing this result if there were no difference at all). The Bayesian view gives you "92% chance B wins" directly, plus how much you risk losing by deciding now, which is what helps you call it.

What is expected loss in a Bayesian A/B test?

It is how much conversion rate you risk losing, on average, if you pick the wrong variation. If the expected loss of choosing B is 0.05 percentage points, deciding for B is safe even if it turns out worse, because the average damage is tiny. Many teams stop the test when expected loss drops below a small threshold they set in advance.

Which prior does the calculator use?

A uniform Beta(1,1) prior, which assumes nothing about the rate before seeing data: every rate from 0 to 100% starts equally likely. With a reasonable number of visitors, the data dominates and the prior choice barely moves the result. It is the neutral, most common choice for A/B testing.

Can I stop the test as soon as the probability passes 95%?

The Bayesian approach tolerates looking at results over time better than the frequentist one, but peeking without rules still inflates risk. The safe path is to combine two triggers set before you start: a high probability (say 95%) and an expected loss below your threshold. Only stop when both hit, and still close whole weekly cycles.

How is this different from the significance calculator?

The significance calculator is frequentist: p-value, confidence interval and a yes or no verdict at 95%. This one is Bayesian: a direct probability that B wins and an expected loss, with no hard threshold. Use significance when you want the classic market standard, and this one when you want to decide by probability and risk. Both read the same data through different lenses.

Does it work for very low rates or little traffic?

It does, but the lower the rate and volume, the wider the uncertainty (the credible interval). With few people, the probability stays near 50% and expected loss stays high, an honest signal that you cannot decide yet. Collect more before you call it.

Keep going

Compared by probability? Cross-check the classic verdict with the significance calculator and size your next test with the sample size calculator. The full context is in the guide on what A/B testing is.