Twyman's Law: When a Result Is Too Good to Be True

A spectacular A/B test result is evidence about your instrumentation before it is evidence about your product. Twyman’s law, as Kohavi and colleagues state it in their KDD 2014 rules of thumb, is that any figure that looks interesting or different is usually wrong, and the operational consequence is that an exceptionally strong win should trigger a bug hunt and a confirmation run, not a launch. This guide covers where the law comes from, the Bayes arithmetic that makes it rigorous rather than folkloric, four documented cases where a big win turned out to be a measurement artifact, a worked example of a statistically flawless result that should still not be shipped, and how to size the confirmation run that settles it. It is part of our complete guide to A/B testing and pairs with A/A testing.

Twyman’s law, and why it is an operating procedure

Kohavi, Deng, Longbotham and Xu, writing up seven rules of thumb generalized from thousands of controlled experiments at Amazon, Booking.com, LinkedIn and multiple Microsoft properties for KDD 2014, dedicate a rule to the asymmetry in how teams read results. Their description of the human failure mode is worth quoting because it is the mechanism, not the moral: we are inclined to resist and question negative results to our great new feature that is being tried, so we drill deeper to find the cause. However, when the effect is positive, the inclination is to celebrate rather than drill deeper and look for anomalies.

Their answer is to install a rule that removes the discretion. When results are exceptionally strong, they call out Twyman’s law: any figure that looks interesting or different is usually wrong. The paper credits the formulation to Twyman by way of a note by A. S. C. Ehrenberg in the Journal of the Royal Statistical Society, and the authors note the same reflex is standard elsewhere in science: no modern editor celebrates a submitted proof that P equals NP, they send it to a reviewer to find the error.

The rule is not scepticism for its own sake. It is a claim about base rates, and it is testable.

The Bayes arithmetic that makes the law rigorous

The same paper formalizes it. Let alpha be the significance level, beta the type II error rate, and pi the prior probability that the alternative hypothesis is true. Then the posterior probability of a true positive given a statistically significant result is

P(true positive | significant) = pi × (1 − beta) ÷ [ pi × (1 − beta) + (1 − pi) × alpha ]

That is the entire mechanism. The p-value threshold is fixed at 0.05 and power is fixed at 80 percent in every row below. The only thing that changes is how often ideas of this kind actually work, and it changes the answer completely.

Prior probability the idea works	Posterior probability the win is real, given p below 0.05
1 in 3	88.9 percent
1 in 5	80.0 percent
1 in 10	64.0 percent
1 in 50	24.6 percent
1 in 100	13.9 percent
1 in 500	3.1 percent

We recomputed this table from the formula above at alpha 0.05 and power 0.8. Two of its rows are the paper’s own published figures and match exactly: Kohavi and colleagues report 89 percent for a prior of 1 in 3, which they describe as the average success rate reported across multiple experiments at Microsoft, and 3.1 percent for a prior of 1 in 500, which is the rate they give for a breakthrough result at Bing. The rest of the table is the same arithmetic extended across the range.

The practical translation: the rarer the result you are claiming, the more evidence it takes to claim it, and a p-value below 0.05 is not more evidence when the claim gets rarer. It is exactly the same evidence.

Four wins that were measurement artifacts

The reason to take the base rate seriously is that the alternative explanations are documented and mundane. All four cases below are from the Microsoft papers.

A 64 percent drop in clicks that was good news. Kohavi and colleagues (KDD 2014) describe an Office Online team testing a page redesign with a strong call to action. Actual purchases were hard to track, so the team used clicks on revenue-generating links as a surrogate, assuming clicks times conversion rate equals revenue. The redesign produced a 64 percent reduction in clicks per user. The shocking size is what made people look, and the explanation was that the assumption of a stable click-to-purchase conversion rate was wrong: the treatment page showed the price, so it attracted fewer clicks from better-qualified users who converted at a much higher rate.

More clicks from a slower page. JavaScript was added to the Bing results page, which normally slows things down and should have cost engagement. Clicks per user went up. Following Twyman’s law, they investigated. Click tracking is beacon-based and some browsers drop the call when the user navigates away; the extra JavaScript improved click-tracking fidelity, not actual clicking.

A CDN migration that improved everything. Over several months in 2013, Bing moved its content delivery network from Akamai to its own Bing Edge. Home page clickthrough improved, feature usage went up, abandonment came down. The same root cause: better click-tracking fidelity. To quantify it, they replaced beacon-based click tracking with redirects, a method with negligible click loss, and found the click loss rate for some browsers dropped by more than 60 percent. A large portion of the gains over time was an artifact.

A dramatic improvement that double-counted. An MSN experiment tried a better auto-suggest algorithm, and searches on Bing referred from MSN improved dramatically. The new code was effectively issuing two searches when a user selected a suggestion; the browser disconnected one of them, so only one results page was displayed, but both were counted.

The 2012 puzzling-outcomes paper adds the pattern that closes the loop, in a case where metrics unrelated to the change moved in unexpected directions with high statistical significance: we reran the experiment on a larger sample to increase statistical power, and many of the effects disappeared. The cause was carryover from an earlier experiment on the same user buckets, an effect they measured as still visible about three weeks later in one case and not fully recovered after three months in another. That is what a confirmation run is for.

Worked example: a flawless result you should not ship

Here is the uncomfortable part. Run these numbers in the calculator:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Arm	Visitors	Conversions	Rate
A, control	40,000	1,200	3.000 percent
B, variant	40,000	1,680	4.200 percent

The calculator returns plus 1.200 percentage points, plus 40.00 percent relative, z = 9.11, p-value below 0.0001, with a 95 percent confidence interval of plus 0.942 to plus 1.458 percentage points. There is no peeking, no sample ratio mismatch, no multiple comparisons problem. By every statistical check this experiment passes, and the interval is comfortably far from zero.

Now apply the base rate. The design’s minimum detectable effect at 40,000 per arm and a 3 percent baseline is plus 11.3 percent relative, so the observed effect is more than three times the smallest effect the test was built to see. A copy change producing a 40 percent relative lift in checkout conversion is not in the same category as the movement Kohavi and colleagues report as typical: at a site like Bing, where thousands of experiments run annually, most fail and those that succeed improve key metrics by 0.1 to 1.0 percent once diluted to overall impact. Put it in their Sessions per user frame: with a distribution of experiment effects centered on zero and a standard deviation of 0.25 percent, they point out that a 2.0 percent improvement sits eight standard deviations from the mean, with a probability around one in a thousand trillion before you account for anything else. Their conclusion is not to celebrate but to start working on finding the bug, which in their experience is usually an instrumentation error.

So the correct action on this result is not to ship and not to discard. It is to run the bug checklist, then run a confirmation run sized for the effect you would actually accept.

The bug checklist, before you spend traffic

Reruns cost calendar time. Defect checks cost an afternoon, so they come first.

Check	What a failure looks like	Where it is covered
Traffic split against the intended ratio	Observed split drifts from the configured one beyond chance	Sample ratio mismatch
Effect concentrated in one browser or device	The win exists in one segment and is flat everywhere else, a classic tracking-fidelity signature	Simpson’s paradox
Effect concentrated in one day	A release, an outage or a campaign landed inside the window	How to run an A/B test
Same tracking code firing in both arms	The variant fires an extra event, or the control loses one	This is the MSN double-search and the beacon cases above
Metric definition identical across arms	A denominator changed, or a filter applies to only one arm	Choosing the primary metric
Surrogate metric standing in for the real one	Clicks moved, purchases did not, and the ratio between them was never stable	The Office Online case above
Users carried over from a previous experiment	The buckets were not re-randomized before this run	A/A testing
The result appeared while you were watching	The decision was taken at the moment the line crossed	The peeking problem

If any row fails, you have your explanation and no rerun is needed. If every row passes, the result graduates from implausible to unexplained, and the confirmation run becomes the cheapest remaining way to learn anything.

Designing the confirmation run

Three rules, and the second is the one teams get wrong.

Re-randomize. The confirmation run must assign users with a fresh hash seed, not reuse the buckets from the first run. This is the direct lesson of the 2012 carryover finding: reusing the same user buckets means the second run inherits whatever the first one did to those users.

Size it for the effect you would accept, not the effect you observed. The first estimate is inflated precisely because it was selected for being large, so powering the rerun against it guarantees an underpowered rerun. Suppose the smallest lift worth shipping is plus 10 percent relative on a 3 percent baseline. At 95 percent confidence and 80 percent power, that requires 53,211 users per variant, which you can check in the sample size calculator. Powering instead for the observed plus 40 percent would call for a fraction of that and would fail to resolve the honest answer. The mechanics of choosing that threshold are in minimum detectable effect.

Decide in advance what each outcome means. Write down the three cases before the run starts: the effect replicates near the same magnitude and you ship; the effect replicates much smaller but still above your threshold and you ship the smaller number; the effect vanishes and you close the experiment and keep the bug hunt open. Deciding afterwards is how a failed confirmation run gets rationalized into a segment analysis.

Common mistakes

Mistake	What it produces
Treating a small p-value as proof the measurement is correct	A published win that was a beacon-tracking artifact
Powering the confirmation run against the first run’s estimate	A rerun too small to resolve the honest effect, read as “inconclusive, ship it”
Reusing the same user buckets for the rerun	Carryover from the first run baked into the second, per the 2012 findings
Applying the scepticism only to losses	The asymmetry Kohavi and colleagues name as the reason the law is needed
Trusting a surrogate metric when it moves a lot	The Office Online case: the surrogate moved because the relationship broke
Announcing the number before the checklist runs	An organizational commitment to a result you have not verified
Concluding “not a bug” because you looked and found nothing	Absence of a found bug is weak evidence; the confirmation run is the strong test
Calling the second run a failure when it returns a smaller effect	Smaller is the expected outcome, and often still worth shipping

Automate this with Donnu

Twyman’s law only works as a rule if the checks are already in the report, because nobody runs a manual checklist against a number they are excited about. Donnu A/B runs the sample ratio check on every experiment and blocks the reading when the split fails, breaks the result down by device and by day so a win concentrated in one browser is visible without a separate query, and lets you clone an experiment with a fresh randomization seed so a confirmation run never inherits the buckets of the run it is confirming. The result page reports the confidence interval next to the point estimate, which is what tells you whether a replicated smaller effect is still inside what you were willing to accept.

Start a free 14-day trial and put your next surprising win through the checks before it reaches a roadmap.

References

• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of Twyman’s law as stated (“any figure that looks interesting or different is usually wrong”), of the asymmetry in how teams treat positive and negative results, of the Bayes rule formulation and the published posterior figures of 89 percent for a prior of 1 in 3 and 3.1 percent for a prior of 1 in 500, of the one in 500 breakthrough rate at Bing and the 0.1 to 1.0 percent typical improvement to key metrics, of the eight standard deviations example on Sessions per user, and of the Office Online, slower-page, Bing Edge and MSN auto-suggest cases. exp-platform.com.
• Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T. and Xu, Y. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. KDD 2012. Source of the case where unrelated metrics moved with high significance and the effects largely disappeared on a larger rerun, and of the carryover measurements showing the effect still visible about three weeks later in one experiment and not fully recovered after three months in another. exp-platform.com.
• Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R. M. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 2009. Background on randomization, A/A validation and the practical checks that catch instrumentation defects. exp-platform.com.
• Gelman, A. and Carlin, J. Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 2014. Source of the result that statistically significant estimates from underpowered designs systematically overstate effect magnitude, which is why a spectacular first estimate is the wrong basis for sizing a rerun. stat.columbia.edu.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on trustworthiness, instrumentation and replication. Companion material at experimentguide.com.

Read also: A/A testing · Sample ratio mismatch · The peeking problem · Minimum detectable effect · Free A/B test significance calculator · Leia em português