The Winner's Curse: Why Your Wins Shrink After Launch

The lift you measured on a winning A/B test is, on average, larger than the lift you will get after launch, and the size of that gap is set almost entirely by the statistical power of the design. This is the winner’s curse: you did not pick the variant at random, you picked it for crossing a threshold, and crossing a threshold on a small sample requires help from noise. This guide covers why the bias is structural rather than a data problem, how much inflation to expect at each power level, a worked example where an apparent plus 23 percent replicates at plus 8 percent, and the three ways to report a number that survives contact with reality. It is part of our complete guide to A/B testing and pairs with minimum detectable effect.

The winner’s curse is selection, not measurement

Your estimate of the lift is unbiased. That is a real property of the two-proportion test and it is not in dispute. Run the same experiment a thousand times against a true lift of plus 5 percent relative and the average of those thousand estimates lands on plus 5 percent.

But you do not report the average of a thousand runs. You report the runs that crossed p below 0.05, and you throw away the rest as inconclusive. On an underpowered design, most runs against a real plus 5 percent effect do not cross. The ones that do are the ones where random variation happened to push the observed difference upward. Conditioning on significance selects for upward noise, so the average surviving estimate is larger than the truth.

Gelman and Carlin gave this quantity a name and a method in Perspectives on Psychological Science in 2014. They call the overestimation factor the exaggeration ratio, or expected Type M (magnitude) error: the expectation of the absolute value of the estimate divided by the true effect size, conditional on statistical significance. Alongside it they define the Type S (sign) error rate: the probability that a significant estimate points the wrong way. Their summary of the situation is blunt: statistically significant results in a noisy setting are highly likely to be in the wrong direction and invariably overestimate the absolute values of any actual effect sizes, often by a substantial factor.

How much inflation, at each power level

The method is a direct calculation, not a rule of thumb. Take the true effect, take the standard error of your design, work out what fraction of repeated runs would clear the significance threshold, and average the absolute estimate over just those runs. We ran that calculation in closed form across the power range, at alpha 0.05 and two-sided:

Power of the design against the true effect	Type S error rate: significant winner points the wrong way	Exaggeration ratio: measured effect divided by true effect
10 percent	4.50 percent	3.71 times
20 percent	0.53 percent	2.26 times
30 percent	0.11 percent	1.81 times
50 percent	0.01 percent	1.41 times
80 percent	under 0.01 percent	1.12 times
90 percent	under 0.01 percent	1.06 times

Two checks that this table is right, both against figures published by Gelman and Carlin. At a true effect 2.8 standard errors from zero, which is the definition of 80 percent power, they report power 0.80, a Type S error rate of 1.2 times ten to the minus six, and an exaggeration factor of 1.12; our calculation returns 0.7996, 1.21 times ten to the minus six, and 1.125. For their worked example with a true effect of 2 percent and a standard error of 8.1 percent, they report power 0.06, Type S 24 percent and exaggeration 9.7 from a 10,000-run simulation; our closed form returns 0.057, 24 percent and 9.5, the small difference being simulation noise in theirs.

The shape of the table is the whole message. Above 80 percent power there is essentially nothing to correct. Below 30 percent power the correction is larger than most teams’ entire annual improvement. And most A/B tests on small and mid-sized sites are far below 30 percent power against realistic effects, which is the point made in A/B testing for low-traffic sites.

The mechanism generalizes beyond experimentation. Ioannidis, in his 2005 PLoS Medicine essay on why most published research findings are false, sets out the same structure in terms of positive predictive value and lists the conditions that make a claimed finding less likely to be true: smaller studies, smaller effect sizes, more relationships tested with less preselection, and more teams chasing statistical significance. An experimentation program with a large backlog, small traffic and a low prior success rate ticks every box.

Worked example: plus 23 percent that was really plus 8

A team on a 3 percent baseline runs a checkout change for two weeks and gets 12,000 visitors per variant. Paste the numbers below into the calculator:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Round 1	Visitors	Conversions	Rate
A, control	12,000	360	3.000 percent
B, variant	12,000	444	3.700 percent

The calculator returns plus 0.700 percentage points, plus 23.33 percent relative, z = 3.01, p-value 0.0026, with a 95 percent interval of plus 0.245 to plus 1.155 percentage points. Significant, and not marginally so.

Now look at what the design could see. At 12,000 per variant on a 3 percent baseline, 95 percent confidence and 80 percent power, the minimum detectable effect is 0.617 percentage points, or plus 20.57 percent relative. The observed effect barely exceeds the smallest effect the test was built to detect, which is the classic signature of a design that can only produce large-looking wins. If the true effect were the far more plausible plus 5 percent relative, this design would have had 9.9 percent power to detect it, which the table above maps to an expected exaggeration of about 3.7 times and a 4.5 percent chance of the sign being wrong outright.

The team runs a confirmation, sized for the smallest lift they would actually ship, plus 10 percent relative, which needs 53,211 per variant. They round to 54,000:

Round 2, confirmation	Visitors	Conversions	Rate
A, control	54,000	1,620	3.000 percent
B, variant	54,000	1,750	3.241 percent

The calculator returns plus 0.241 percentage points, plus 8.02 percent relative, z = 2.28, p-value 0.0229, interval plus 0.033 to plus 0.448 percentage points. The minimum detectable effect at this size is plus 9.69 percent relative, so the design is now resolving the range that matters.

The change is real and worth shipping. The number is a third of what the first run said. The ratio between the two point estimates is 23.33 divided by 8.02, about 2.9 times, sitting neatly inside the range the exaggeration table predicts for a design in that power band. Nothing went wrong in round one. It measured exactly what an underpowered design measures when it produces a win.

Why the roadmap total never arrives

Take a program that shipped ten winners this quarter, each measured at plus 5 percent on its own funnel step. The slide says plus 50 percent. Two multiplications stand between that slide and reality.

Inflation. If those tests ran around 20 percent power against their true effects, each measured plus 5 percent corresponds to a true effect nearer plus 2.2 percent, by the ratio in the table above. Ten of those is not plus 50 percent, it is closer to plus 22 percent before anything else is applied.

Dilution. An effect measured on the subset of users who reached that funnel step has to be scaled by that subset’s share of traffic before it can be added to a company-level metric. Kohavi and colleagues state the rule directly in their KDD 2014 rules of thumb: metrics should be diluted by their segment size, and a 10 percent improvement to a 1 percent segment has an overall impact of approximately 0.1 percent. The same paper reports that at a site like Bing, the experiments that succeed improve key metrics by 0.1 to 1.0 percent once diluted to overall impact.

Those two together are usually enough to close the gap between the roadmap and the dashboard without needing any further explanation. The practical fix is to stop adding point estimates: attribute revenue from the confirmed, diluted number, as covered in attributing revenue to your A/B test winner, and treat the program’s aggregate through its measured win rate rather than through a sum of best cases.

Three ways to report an honest number

Approach	What you report	Cost	When to use it
Lower bound of the interval	The conservative end of the 95 percent interval instead of the point estimate	Free	Always, as the planning number in any forecast
Confirmation run	The replicated estimate from a second run sized for your real threshold	Weeks of traffic	Before any decision that commits budget or headcount to the result
Shrinkage estimator	The point estimate pulled toward the distribution of effects your program actually produces	Requires a prior from your own history	Programs with enough completed tests to know their own effect distribution
Power the test properly in the first place	An estimate that needs almost no correction, at 80 percent power or above	Sample, decided before launch	Whenever the traffic exists; this is the only option that removes the problem rather than patching it

The first row is the one nobody does and everybody could. In the worked example above, round one’s lower bound was plus 0.245 percentage points, which in relative terms is about plus 8.2 percent, and that is almost exactly what the confirmation run found. The honest number was already sitting in the first report. It was just not the number anybody quoted.

Common mistakes

Mistake	What it produces
Forecasting revenue from the point estimate of an underpowered win	A forecast inflated by a factor other departments will plan against
Powering a confirmation run against the first run’s estimate	A rerun too small to resolve the honest effect
Reading a smaller replication as “the test failed”	Discarding a real improvement because it stopped being spectacular
Stopping the test when the line crosses	The winner’s curse applied repeatedly rather than once; see the peeking problem
Adding shipped lifts across the roadmap	Compounding both inflation and dilution in the same number
Comparing your lift to public case studies	Those are also selected winners, so they carry the same bias and usually worse
Treating the bias as a tooling defect	It is a property of the selection rule; every tool that reports significance has it
Concluding that low-traffic sites should not test	The right conclusion is to test bigger changes, not to stop; see minimum detectable effect

Automate this with Donnu

The winner’s curse is not fixed by a better p-value, it is fixed by making power and uncertainty impossible to ignore at the moment of the decision. Donnu A/B computes the minimum detectable effect for the sample you actually have and shows it beside the result, so a lift that barely exceeds it is visibly a lift the design could only report as large. It reports the confidence interval as the headline alongside the point estimate rather than underneath it, so the conservative planning number is always in view. And cloning an experiment with a fresh randomization seed takes one click, which is what makes a confirmation run a normal step instead of a project.

Start a free 14-day trial and see the interval next to every result you are about to ship.

References

• Gelman, A. and Carlin, J. Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 2014. Source of the exaggeration ratio and Type S error definitions and of the retrodesign method used for our table, of the published values power 0.80, Type S 1.2 times ten to the minus six and exaggeration 1.12 at a true effect 2.8 standard errors from zero, of the worked example returning power 0.06, Type S 24 percent and exaggeration 9.7, and of the finding that significant results in noisy settings invariably overestimate effect magnitude. stat.columbia.edu.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the dilution rule that metrics should be scaled by segment size, with a 10 percent improvement to a 1 percent segment giving approximately 0.1 percent overall, and of the observation that successful experiments at a site like Bing move key metrics by 0.1 to 1.0 percent once diluted. exp-platform.com.
• Ioannidis, J. P. A. Why Most Published Research Findings Are False. PLoS Medicine, 2(8), 2005. Source of the positive predictive value framing and of the conditions that make a claimed finding less likely to be true, including smaller studies, smaller effect sizes and more teams chasing statistical significance. journals.plos.org.
• Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T. and Xu, Y. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. KDD 2012. Source of the case in which a surprising result was rerun on a larger sample and many of the effects disappeared, the empirical form the winner’s curse takes in practice. exp-platform.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on statistical power, effect size estimation and replication. Companion material at experimentguide.com.

Read also: Minimum detectable effect · How many visitors an A/B test needs · The peeking problem · Average A/B test win rate · Free statistical power calculator · Leia em português