Expected Loss in Bayesian A/B Testing

Expected loss is the stopping rule in Bayesian A/B testing that answers a more honest question than “what is the probability that B is better”: how much would you lose, on average, in the metric that matters, if you picked the wrong variation? Instead of deciding purely on the odds of being right, you decide on the size of the risk still on the table. Once that risk drops below what the business can tolerate, you stop, without waiting for a p-value to cross 0.05. This guide builds on our Bayesian priors guide and the Bayesian treatment already covered in the complete guide to A/B testing: here the focus is narrowly on the stopping rule itself, with numbers computed by the same Bayesian engine that powers the calculator on this blog, not estimated by ear.

What probability alone does not tell you

The Bayesian calculator on this blog already shows two numbers side by side: the probability that B beats A, and the “risk of choosing B,” which is exactly the expected loss. Most guides to Bayesian A/B testing stop at the first number, treating probability as if it were enough to act on. It is not.

Probability answers “what is the chance I am wrong.” Expected loss answers “if I am wrong, what does it cost.” Those are different questions, and the second one is the one a business actually needs answered: two tests can land on the same win probability and still carry completely different risk, depending on the size of the effect in play and how much sample is backing the estimate.

We ran two real simulations through the same Bayesian engine that powers this blog to show the gap in numbers, not just in theory:

The reason is simple: Test 2 has a small sample (110 per variant) backing a large effect (20% versus 30%). Uncertainty about the true size of that effect is still wide, so in the scenarios where betting on B turns out wrong, it can turn out very wrong. Test 1 has a large sample backing a small effect: even in the scenarios where B is not actually better, the likely gap is tiny. Win probability barely tells the two cases apart; the size of the risk does.

The math behind expected loss

The starting point is the same one from the Bayesian priors guide: each variant carries a Beta posterior, updated from a prior and the observed data (conversions and non-conversions). From those two posteriors you can compute not just who is probably ahead, but the size of the expected error in each direction:

In plain English: take the gap between A’s true rate and B’s true rate in every possible scenario (weighted by each variant’s posterior), but only count that gap when it is positive, meaning A genuinely beats B. The plus sign on the exponent means exactly that: zero out the difference whenever B is the actual winner, and only add up the scenarios where choosing B would have been a mistake. The result is the average of that possible mistake, integrated over all the uncertainty that still remains. A mirror-image expected loss exists for the other direction, choosing A when B was the real winner, and the Bayesian calculator on this blog reports both.

Notice what this definition already buys you: if B is clearly ahead and the sample is already large, the scenarios where A wins become rare, and when they do happen the gap tends to be small, so expected loss collapses toward zero. If the race is still tight, or the sample is small, the scenarios where the choice is wrong stay plausible and, depending on the effect size in play, expensive, so expected loss stays high. That is exactly the behavior the simulation above captured with real numbers.

A worked example, end to end

Let’s run the same scenario used in this blog’s sample size calculation: a 5% baseline rate, a 10% relative minimum detectable effect (target of 5.5%), 95% confidence, and 80% power. This blog’s sample size engine returns 31,234 visitors per variant, which, at a weekly traffic of 10,000 visitors (5,000 per variant), takes 44 days to reach the planned sample, the point at which a classic frequentist test would finally declare a winner.

But the real conversion rate of each variant does not jump straight to its final value: it reveals itself gradually, visitor by visitor. Running the same Beta-Binomial posterior day after day, with A’s true rate at 5.00% and B’s at 5.50% (the effect the test was designed to detect), expected loss from choosing B shrinks like this:

Day	Visitors per variant	P(B beats A)	Expected loss from choosing B	Expected loss in weekly revenue*
3	2,143	77.39%	0.0894 pp	$1,340.44
7	5,000	86.86%	0.0295 pp	$441.93
10	7,143	91.14%	0.0153 pp	$229.33
14	10,000	94.35%	0.0076 pp	$114.10
16	11,429	95.72%	0.0051 pp	$77.20
21	15,000	97.39%	0.0026 pp	$38.37
30	21,429	99.03%	0.0007 pp	$10.56
44	31,429	99.76%	0.0001 pp	$1.86

*Hypothetical conversion to revenue: an average order value of $150 applied to the 10,000-visitor weekly traffic, just to give expected loss a business unit. Swap in your own average order value and traffic.

Choosing a stopping threshold

The number that decides when to stop does not come out of a statistical formula on its own: it is a business risk decision, and it needs to be set before the test begins, the same way you fix a sample size before running a frequentist test. Chris Stucchio, in the whitepaper describing the Bayesian engine behind VWO, proposes exactly this logic: pick a tolerable expected loss threshold, expressed in the business unit that actually matters, and stop once expected loss falls below it.

In the example above, we used a practical threshold: 0.1% of expected weekly revenue. With 10,000 weekly visitors and a $150 average order value, reference weekly revenue lands around $78,750, so 0.1% of it is roughly $78.75 a week. Converting that figure back into the unit the Bayesian calculator reports (percentage points of conversion rate), the threshold works out to about 0.0053 percentage points. That is exactly the number expected loss crosses on day 16, against the 44 days a fixed sample plan would take to reach the same confidence.

That is the practical payoff of deciding by expected loss instead of waiting on a p-value or a fixed sample: you stop as soon as the residual risk is small enough for your business to tolerate, not sooner, not later. Worth repeating a point already covered in the Bayesian priors guide: this entire calculation depends on the posterior being well calibrated, and a poorly chosen prior distorts expected loss exactly the way it distorts win probability.

It is also worth noting, in passing, what happens with the same pair of numbers used in the significance example from the complete guide to A/B testing: A with 210 conversions out of 4,200 visitors, B with 273 out of 4,200. There, the p-value came out to roughly 0.003. Running that same pair of numbers through the Bayesian engine, the probability that B beats A is 99.84% and the expected loss from choosing B is just 0.0002 percentage points, essentially negligible. Both readings agree on the winner; the Bayesian one just adds a direct ruler for how much risk was left.

A second example: SaaS, trial to paid

The same reasoning applies outside e-commerce. Take a SaaS product testing an onboarding change, aiming to lift trial-to-paid conversion from 20% to 21.6% (an 8% relative effect), at a more modest 700 trials a week. This blog’s sample size engine calls for 10,101 trials per variant, which, at that volume, would take 203 days for the fixed sample plan to complete, too long for most product roadmaps.

Applying the same 0.1% threshold, now against a reference weekly monthly-recurring-revenue base for a week of trials (a hypothetical $97/month ARPU, generating a reference of roughly $14,123/month), the equivalent threshold in percentage points comes out to around 0.0208. Expected loss from choosing B crosses that threshold on day 70, at 3,500 trials per variant and a 95.04% probability that B beats A, against the 203 days the fixed plan would need.

Scenario	Baseline	Relative effect	Fixed-plan sample	Fixed-plan duration	Expected loss crosses on	Time saved
E-commerce (conversion)	5.00%	+10%	31,234/variant	44 days	day 16 (11,429/variant)	28 days, ~64% faster
SaaS (trial to paid)	20.00%	+8%	10,101/variant	203 days	day 70 (3,500/variant)	133 days, ~66% faster

In both scenarios, deciding by expected loss instead of waiting out the full fixed sample plan cut roughly two-thirds of the testing time, without giving up rigor: in both cutoff points, the probability that B beats A was already above 95%, and the residual risk, measured in the business’s own unit, was already small enough for the tolerance set before the test began.

Expected loss is not a silver bullet

It is worth the honesty that most Bayesian material skips: expected loss solves a real problem, but it is not immune to every risk of a poorly designed test.

It can still be fooled by peeking. Checking the result repeatedly and stopping the first moment expected loss crosses the threshold still occasionally catches a lucky streak, especially with little accumulated sample. David Robinson’s piece on whether Bayesian A/B testing is immune to early stopping makes exactly this point: the posterior’s interpretation stays valid at any N, but stopping at the very first crossing, with no additional discipline, still inflates the odds of a premature call. The safer practice, in the same spirit as our guide on the peeking problem in A/B testing, is to require a minimum sample before even checking the criterion, and to confirm expected loss stays below the threshold across more than one reading, not just the first time it crosses.

It inherits sensitivity to the prior. Like any result that comes out of a posterior, expected loss computed with a poorly calibrated prior (say, a strong prior inherited from a different context, covered in detail in the Bayesian priors guide) ends up systematically skewed in the same direction as the prior’s bias. When in doubt, the uniform prior, the default on this blog’s calculator, is the safer choice.

The threshold is a business decision, not a statistical truth. There is no universal “tolerable expected loss” value. It depends on how much risk your business is willing to accept in exchange for deciding faster, and it needs to be set, and documented, before the test begins, exactly the way you would fix a sample size for a frequentist test.

To lay the three criteria that decide an A/B test side by side, and where each one can mislead:

Criterion	What it answers	Strength	Weakness
P-value (frequentist)	What is the chance of seeing this difference, or a larger one, if A and B were actually equal	Industry standard, comparable across studies	Does not say the size of the risk of being wrong, only the chance the effect is noise
Probability P(B beats A)	What is the chance B is genuinely better than A	Intuitive for decision makers, speaks in terms of belief rather than a hypothetical infinite repetition	The same probability value can hide either an expensive or a cheap mistake; it does not tell the two apart
Expected loss	If I choose wrong, how much do I lose, on average, in the metric that matters	Combines the chance of being wrong with the size of the mistake in a direct business unit	The tolerance threshold needs careful judgment; the formula alone does not choose it for you

See the risk of your own decision

The calculator below already shows expected loss next to the win probability, exactly the pair of numbers this guide compared. Paste your own test’s visitors and conversions for A and B and read the “risk of choosing B” field, the same expected loss calculated here, end to end:

A (control)

VisitorsConversions

B (variation)

VisitorsConversions

-probability that B beats A

Rate of A (posterior)-

Rate of B (posterior)-

Probability A wins-

Risk of choosing B (expected loss)-

Relative lift (means)-

Beta-Binomial model with a uniform Beta(1,1) prior and a 95% credible interval. Deterministic calculation, updates live.

If you want to apply the revenue threshold to your own case, convert the tolerance you set (for example, 0.1% of expected revenue for the period) into the same unit the calculator reports, by dividing the tolerable amount by the period’s traffic multiplied by your own average order value or ARPU, the same way we did in the two examples above.

Automate this on Donnu

Computing expected loss by hand means running a numerical integration over two Beta posteriors for every new visitor, and most teams simply do not do it, deciding by ear or waiting on a p-value that might never arrive in time for the roadmap. Donnu already runs on a native Bayesian engine that calculates win probability and expected loss in real time with every new conversion, no spreadsheet, no manual integration, and no waiting for a fixed sample plan to finish before it is safe to decide.

Start a free 14-day trial and watch the risk of your own decision shrink visitor by visitor. To understand the foundation behind this calculation, read the Bayesian priors guide, and if your primary metric is noisy (revenue per user, session length), see how CUPED reduces A/B test variance before any stopping rule even enters the picture. Prefer to review the frequentist foundation first? See the A/B testing statistical significance guide.

Leia em português: perda esperada em teste A/B bayesiano.

References

• Stucchio, C. Bayesian A/B Testing at VWO (technical whitepaper defining expected loss and the stopping threshold used in this guide). vwo.com/downloads/VWO_SmartStats_technical_whitepaper.pdf.
• GrowthBook. Statistical Details, Bayesian Engine (production definition of “risk”/expected loss: how much you expect to lose if the chosen variation is actually worse). docs.growthbook.io/statistics/details.
• Robinson, D. Is Bayesian A/B Testing Immune to Peeking? Not Exactly (why expected loss and Bayesian probability still require discipline against early stopping). varianceexplained.org.