The Pre-Registered Analysis Plan in A/B Testing

A pre-registered analysis plan is the list of decisions you lock down before the experiment starts, precisely because afterwards they could be chosen in light of the result. It does not exist to restrain bad actors: it exists because honest analysts looking at real data make reasonable choices that would have been different had the data come out differently, and that contingency alone breaks the interpretation of the p-value. This guide covers the mechanism behind the problem, a worked example where a test with no effect produces an apparently winning segment, the arithmetic of what each fork costs in false positives and in sample, and a copyable plan listing the fields that must be closed before the first visitor arrives. It is part of our complete guide to A/B testing and the design sibling of the peeking problem.

The garden of forking paths

The image belongs to Gelman and Loken, in a 2013 paper whose subtitle already carries the thesis: why multiple comparisons can be a problem even when there was no fishing expedition or p-hacking and the research hypothesis was posited ahead of time.

The authors explain that they came to regret the word “fishing”, because it evokes an image of someone trying comparison after comparison until something is snagged. They have no reason to think researchers regularly do that. The real story, they say, is that an analyst performs a reasonable analysis given their assumptions and their data, but had the data turned out differently they could have done other analyses that were just as reasonable in those circumstances.

The technical consequence is the heart of the matter: even if the chosen analysis is a deterministic function of the observed data, this does not eliminate the multiple comparisons problem, because p-values are based on what would have happened under other possible datasets. A p-value answers a question about hypothetical repetitions. If in different repetitions you would have run different analyses, the p-value you computed does not answer the question you think it answers.

Gelman and Loken give a concrete fork that shows up constantly in A/B testing. Often an analysis is performed on two groups or under two conditions. When a result is statistically significant overall, that can be reported. When it is significant in only one of the groups but not the other, that can be reported as evidence too. And if the difference between them, the interaction, is significant, that also works. All three seem fully consistent with the research hypothesis, but they are different analyses, different paths that could be taken.

Worked example: the segment that won alone

A checkout test runs for two weeks with 40,000 visitors per arm. Before any slicing, the overall result:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Overall result	Visitors	Conversions	Rate
Control	40,000	2,000	5.000 percent
Treatment	40,000	2,060	5.150 percent

On screen, the calculator shows rates of 5.00 and 5.15 percent, a lift of plus 3.0 percent, a p-value of 0.3338, an interval of minus 0.2 to plus 0.5 percentage points and a verdict of not significant yet. At full precision, and including the z score the interface does not display: plus 0.150 percentage points, plus 3.000 percent relative, z = 0.966, p-value 0.3338, with a 95 percent interval of minus 0.154 to plus 0.454 percentage points. Nothing. The interval covers zero comfortably.

Then somebody slices by device. Four segments, and the counts add up exactly to the totals above:

Segment	Control	Treatment	Relative lift	z	p-value
Mobile	12,000 / 540 (4.500 percent)	12,000 / 624 (5.200 percent)	plus 15.556 percent	2.524	0.0116
Desktop	20,000 / 1,100 (5.500 percent)	20,000 / 1,070 (5.350 percent)	minus 2.727 percent	minus 0.662	0.5078
Tablet	4,000 / 200 (5.000 percent)	4,000 / 206 (5.150 percent)	plus 3.000 percent	0.306	0.7599
App	4,000 / 160 (4.000 percent)	4,000 / 160 (4.000 percent)	0.000 percent	0.000	1.0000

Mobile shows up at plus 15.556 percent with a p-value of 0.0116. That is exactly the kind of number that turns into a roadmap decision on Monday morning.

Now the arithmetic that has to travel with that number. If all four segments had been declared before the test, the corrected threshold for four comparisons would be 0.05 divided by 4, that is 0.0125. Mobile would scrape through: 0.0116 is below 0.0125.

But four was never the number of paths available. If, beyond device, the dashboard also let you slice by secondary metric and five metrics sat on the screen, the space of comparisons that could have caught someone’s eye is 20. The corrected threshold drops to 0.0025, and mobile fails by a wide margin. The p-value of 0.0116 did not change; what changed was counting honestly how many doors were open.

The calculation is direct: with independent tests at the 5 percent level, the probability of at least one false positive is 1 minus 0.95 raised to the number of tests.

Comparisons available	Chance of at least one false positive	Corrected threshold	Corresponding critical z
1	5.0 percent	0.05000	1.960
2	9.8 percent	0.02500	2.241
4	18.5 percent	0.01250	2.498
5	22.6 percent	0.01000	2.576
10	40.1 percent	0.00500	2.807
20	64.2 percent	0.00250	3.023

Kohavi, Deng, Longbotham and Xu describe the same phenomenon at platform scale: because thousands of experiments run per year, a false positive rate of 0.05 implies hundreds of false positive results for a given metric, and it is exacerbated when multiple uncorrelated metrics are used.

The prior matters as much as the p-value

There is a second layer, and it is the one that explains why a post-hoc finding deserves far more suspicion than a pre-declared one carrying the same p-value.

The same authors formalize this with Bayes rule. Given the significance level, the Type II error level and the prior probability that the alternative hypothesis is true, the probability that a statistically significant result is genuinely true equals power times prior, divided by the sum of that product with significance level times the complement of the prior.

At a 5 percent level with 80 percent power, the table looks like this:

Prior probability that the idea works	Probability the significant result is true
33.33 percent	88.9 percent
20.00 percent	80.0 percent
10.00 percent	64.0 percent
5.00 percent	45.7 percent
2.00 percent	24.6 percent
0.20 percent	3.1 percent

Both ends of that table come from the authors themselves: they use a prior success rate of 1 in 3, which they report as the average across multiple experiments at Microsoft, and land on 89 percent; and they note that if breakthrough results are one in 500, the posterior probability drops to 3.1 percent.

The practical implication is blunt. A hypothesis declared upfront, derived from prior data and reasoning about the product, carries a decent prior. A hypothesis born from staring at a segment table and noticing mobile went up carries a low prior by construction, because it was selected precisely for being the biggest number on the screen. Same p-value, very different credibility.

The analysis plan: what must be closed before launch

An analysis plan is short. If it runs past one page it has probably become a process document rather than a decision lock. These are the fields that matter, and each one is on the list because it is choosable after the fact:

Plan field	What to fix	What happens if left open
Hypothesis	One sentence with a mechanism, not a wish	It becomes whatever the data supports
Primary metric	Exactly one, with its denominator unit	Whichever metric moved becomes primary
Randomization unit	User, session, account, region	See randomization unit
Minimum detectable effect	Absolute or relative, with the source of the number	The result accommodates any effect size
Sample size and duration	Derived from the MDE, with a target date	Stopping becomes a convenience call, see peeking
Stopping rule	Date or sample, exactly one	Every look becomes another chance to stop at a peak
Significance level	Already corrected for the number of planned comparisons	The correction never happens
Segments	Closed list, with correction applied	Mining becomes discovery
Secondary and guardrail metrics	Closed list, see guardrail metrics	Bad metrics quietly leave the report
Decision criterion	What ships, what gets dropped, what gets rerun	The decision gets justified afterwards

A useful sanity check for the plan: hand it to somebody who did not write it, have them imagine two opposite outcomes, and ask which decision follows in each case reading the plan alone. If either answer is “it depends”, there is still an open fork.

Pre-registration costs sample, and the cost is real

There is no point pretending rigor is free. If the plan honestly declares four comparisons, the significance level of each has to drop, and sample size climbs with it. With a 5 percent baseline, a 10 percent relative minimum detectable effect and 80 percent power:

Significance level	Context	Sample per variant
0.0500	one declared comparison	31,234
0.0125	four declared comparisons	44,375
0.0025	twenty declared comparisons	59,444

Twenty comparisons cost nearly twice the sample of one. That is uncomfortable, and it is exactly the point: the cost existed before the plan, it was just being paid in wrong decisions instead of in visitors. If the right-hand column is unaffordable for your traffic, the correct fix is to cut comparisons in the plan, not to leave them implicit.

What to do with what you found afterwards

None of this requires throwing away the mobile segment from the example. It requires changing its label.

Gelman and Loken are explicit that they do not want demands of statistical purity to straitjacket science, and that the most valuable analyses often arise only after an iterative process involving the data. Their proposed answer is the two-part structure: a first experiment that is exploratory but still theory-based, and a second that is purely confirmatory with its own pre-registered protocol.

They illustrate it with an instructive case from Nosek, Spies and Motyl reported in the paper: the researchers found a large and statistically significant relationship, but rather than stopping there and publishing, they gathered a large new sample and ran a replication with predetermined protocol and analysis. By their own estimates based on the initial significant result, the replication had over 99 percent power. The replication nonetheless failed, with a p-value of 0.59.

In online experimentation vocabulary, Kohavi, Deng, Longbotham and Xu say the same thing in a line: results with borderline statistical significance should be viewed as tentative and rerun to replicate. It is the same discipline as Twyman’s law and the confirmation run, and the same medicine against the winner’s curse.

In practice, a post-hoc finding has three legitimate destinations:

1. It becomes a new test. The mobile segment earns its own pre-registered experiment, with sample sized for it alone.
2. It becomes a note in the report, with no decision attached. Recorded as an exploratory observation, with no p-value presented as proof.
3. It disappears. If nobody thinks it deserves a dedicated test, it did not deserve a decision either.

One extra caution: the same authors point out that it is much easier to improve metrics for a small segment, and that the effect must be diluted by segment size. A 10 percent improvement in a segment representing 1 percent of the total has an overall impact of approximately 0.1 percent. Before celebrating mobile, check how much of the total it is, as discussed in triggered analysis and dilution.

Common mistakes

• Calling something a pre-registration when it was written after the first readout. A plan saved with the experiment already running locks nothing.
• Declaring three primary metrics. Three primary metrics are zero primary metrics, and whichever one moves becomes the chosen one.
• Correcting only for the comparisons that were run. What counts is the space of comparisons that were available, which is the central argument in Gelman and Loken.
• Pre-registering and then changing the plan silently. Plans do change, and sometimes they should; what cannot happen is the change going unrecorded.
• Assuming the plan replaces health checks. A closed plan offers no protection against an uneven traffic split, see sample ratio mismatch.
• Confusing a plan with paperwork. If the document does not change any possible decision, it is not a plan, it is a form.

Make this automatic with Donnu

The fragile part is not writing the plan, it is the plan and the report living in different places, so nobody notices when the second contradicts the first.

In Donnu, the primary metric, the minimum detectable effect, the target sample, the stopping rule and the segment list are part of creating the experiment, not of a parallel document. The report shows what was declared next to what was observed, and any slice outside the declared list is flagged as exploratory, with the corrected threshold spelled out instead of a bare p-value. To redo the arithmetic elsewhere, the p-value calculator, the sample size calculator and the A/B/n significance calculator all take raw counts.

References

• Gelman, A. and Loken, E. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Columbia University, 14 November 2013. Source of the thesis that researcher degrees of freedom create a multiple comparisons problem even when only one analysis is performed, of the caveat that the word fishing is unfortunate because the authors have no reason to think researchers test comparison after comparison, of the formulation that a reasonable analysis given the observed data would have been different had the data turned out differently, of the statement that even a deterministic analysis of the observed data does not eliminate the problem because p-values are based on what would have happened under other possible datasets, of the three-fork example (significant overall, significant in one group, significant interaction) all consistent with the same hypothesis, of the recommendation of a two-part structure with an exploratory study followed by a pre-registered confirmatory one, of the caveat that the authors do not want to straitjacket science and recognize the value of iterating with data, and of the Nosek, Spies and Motyl case in which a replication with over 99 percent estimated power failed with a p-value of 0.59. stat.columbia.edu.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the observation that running thousands of experiments a year at a 0.05 false positive rate implies hundreds of false positives for a given metric, exacerbated by uncorrelated metrics, of the recommendation to treat borderline significant results as tentative and rerun them to replicate, of the Bayes rule formalization of the probability that a significant result is true as a function of power, significance level and prior, of the two cited values (89 percent at a 1 in 3 prior, which the authors report as the average across multiple Microsoft experiments, and 3.1 percent at a 1 in 500 prior), and of the rule of diluting an improvement by segment size, with the example of 10 percent in a 1 percent segment being worth roughly 0.1 percent overall. exp-platform.com.
• Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Reference on randomization unit and analysis unit, used here to justify why the unit belongs in the plan before the experiment starts. arxiv.org/abs/1803.06336.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. General reference on experiment design, OEC selection and decision discipline. Companion material at experimentguide.com.

Read next: The peeking problem · Twyman’s law and the confirmation run · The winner’s curse · Choosing the primary metric · Testing multiple variants · P-value calculator · Leia em português