Group Sequential A/B Tests: Interim Analysis Rules
Interim analysis in A/B testing: how alpha spending buys 2 or 3 planned looks, what it costs in sample, and why 3 naive looks give 10.73% false positives.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Reading the result three times with a 5 percent cutoff at each reading pushes the real false positive rate to 10.73 percent, more than double what you agreed to. The fix is not to stop looking: it is to schedule each interim analysis before the test starts and give each one its own cutoff, so the pieces add up to 5 percent. In the plan we work through below, three looks with an O Brien-Fleming style boundary cost 2.21 percent extra maximum sample size and, when the effect was real, finished the test 14.9 percent sooner. This guide shows where the inflation comes from, how an alpha spending function splits the risk, what each boundary family charges, and gives you a three-look worked example you can paste straight into the calculator. It is part of our complete guide to A/B testing and it is the planned counterpart to the peeking problem.
Where the inflation comes from
A fixed horizon test promises exactly one thing: if you look once, at the agreed sample size, and call a winner when the p-value drops below 0.05, your chance of declaring victory over nothing is 5 percent. Every extra reading is another chance to cross the line by luck, and the chances pile up.
The pile is computable. The z statistics you read at different points in the same test are not independent; they form a Brownian path, because the second look contains all the data from the first. Integrating that dependence numerically, with equally spaced looks and a 5 percent cutoff at each one, the real false positive rate looks like this:
None of that is new or controversial: it is the classic result from sequential analysis, and our integrator reproduces it. The practical point is elsewhere. The obvious answer, “then stop looking”, is impossible to enforce in a team with a dashboard. Johari, Pekelis and Walsh open their paper on the always valid inference behind a large commercial testing platform with exactly this diagnosis: traditional p-values and confidence intervals are wholly unreliable when users choose sample sizes endogenously by continuously monitoring their tests. Their illustration is an A/A test whose “chance to beat baseline” climbs past the 95 percent threshold purely from being watched.
There are two honest ways out. One is always valid inference, which we covered in sequential testing explained and which accepts any number of looks. The other is older, and cheaper in sample size: schedule a few looks up front and split the alpha between them.
Alpha spending, the Lan-DeMets idea
Think of the 5 percent as a budget. A fixed horizon design spends the whole budget on one purchase, at the end. A group sequential design spends a slice at each planned reading, and the slices have to add up to 5 percent.
The alpha spending function is the rule that says how much of the budget may already have been spent by the time you reach a given fraction of the information. FDA guidance on adaptive designs describes the Lan-DeMets approach in exactly those terms: a function specifying how the Type I error probability is spent throughout the trial, while allowing flexibility in the number and timing of the interim analyses. Two functions dominate practice, and they disagree about when to spend.
Translated into the cutoff each reading demands, with three equally spaced looks:
| Look | Fraction of sample | Cumulative alpha (OBF) | Critical z (OBF) | p-value required (OBF) | Cumulative alpha (Pocock) | Critical z (Pocock) | p-value required (Pocock) |
|---|---|---|---|---|---|---|---|
| 1 | 33% | 0.00069 | 3.395 | 0.00069 | 0.02264 | 2.279 | 0.02264 |
| 2 | 67% | 0.01637 | 2.407 | 0.01610 | 0.03817 | 2.295 | 0.02174 |
| 3 | 100% | 0.05000 | 2.015 | 0.04388 | 0.05000 | 2.296 | 0.02168 |
Look at what the last row says. With the O Brien-Fleming boundary, a test that made it to the end still has almost the full traditional cutoff available (0.04388 against 0.05). With Pocock, the price of having been allowed to stop early is that the final reading is stricter than a normal test would be: 0.02168 instead of 0.05. There is no free lunch, only a choice about where to pay for it.
FDA guidance puts the difference between the families in a sentence worth copying: the O Brien-Fleming approach tends to require very persuasive early results to stop the trial for efficacy, while alternatives such as Pocock require less persuasive early results and have higher probabilities of early stopping.
What an interim analysis costs in sample size
A stricter boundary in the middle means a slightly looser one at the end, and to hold the same 80 percent power the design needs a larger maximum N than the fixed horizon test. We computed the exact factor for both shapes, with three looks:
| Design | Maximum N (factor over fixed) | Expected N when the effect is real | Expected N when it is not | Chance of stopping at look 1 |
|---|---|---|---|---|
| Fixed horizon | 1.0000 | 100% | 100% | not applicable |
| 3 looks, O Brien-Fleming | 1.0221 | 85.1% of fixed N | 101.6% of fixed N | 3.93% |
| 3 looks, Pocock | 1.1703 | 81.9% of fixed N | 114.7% of fixed N | 29.82% |
The reading is direct. The O Brien-Fleming boundary is nearly free: 2.21 percent more maximum sample, and if the effect does not exist the test costs only 1.6 percent more traffic than the fixed design. In exchange, it almost never stops at the first look (3.93 percent of the time). Pocock is the mirror image: nearly a third of real-effect tests end at the first reading, but when there is nothing to find the test burns 14.7 percent more traffic. If your bottleneck is the experiment queue, Pocock buys back calendar time; if your bottleneck is traffic, the choice goes the other way.
The concrete plan: 5 percent baseline, 10 percent MDE, 40k a week
Let us land this on a real test. For a 5 percent baseline rate, a minimum detectable effect of 10 percent relative, 95 percent confidence and 80 percent power, the sample size calculator returns 31,234 visitors per variant, which is 11 days at 40,000 visitors a week. That is the fixed horizon design, and the sequential plan is built on top of it.
| Item | Fixed horizon | 3 looks, O Brien-Fleming | 3 looks, Pocock |
|---|---|---|---|
| Maximum N per variant | 31,234 | 31,926 | 36,555 |
| Maximum duration | 11 days | 12 days | 13 days |
| Looks (visitors per variant) | 31,234 | 10,642 / 21,284 / 31,926 | 12,185 / 24,370 / 36,555 |
| p-value required at each look | 0.05000 | 0.00069 / 0.01610 / 0.04388 | 0.02264 / 0.02174 / 0.02168 |
| Expected N if the effect is real | 31,234 (11 days) | 26,575 (10 days) | 25,592 (9 days) |
| Expected N if it is not | 31,234 (11 days) | 31,744 (12 days) | 35,814 (13 days) |
That is the entire design in one table, and it fits in a paragraph of the analysis plan. If you do not write that document yet, the pre-registered analysis plan is where the three dates and the three cutoffs need to live.
A worked example, with the calculator
We simulated a test with a real 10 percent relative effect (5.0 against 5.5 percent) and read it at the three looks of the O Brien-Fleming plan. The numbers below are cumulative visitors and conversions; paste any row into the calculator and you get the same p-value.
| Look | Control | Variant | p-value | Relative lift | Below 0.05? | Crosses the boundary? |
|---|---|---|---|---|---|---|
| 1 (n = 10,642) | 549 of 10,642 (5.1588%) | 593 of 10,642 (5.5723%) | 0.180757 | 8.01% | no | no (z 1.338 against 3.395) |
| 2 (n = 21,284) | 1,078 of 21,284 (5.0648%) | 1,182 of 21,284 (5.5535%) | 0.024567 | 9.65% | yes | no (z 2.248 against 2.407) |
| 3 (n = 31,926) | 1,598 of 31,926 (5.0053%) | 1,768 of 31,926 (5.5378%) | 0.002608 | 10.64% | yes | yes (z 3.011 against 2.015) |
Row 2 is the reason this article exists. The p-value of 0.024567 is below 0.05, and a team without a plan would have stopped right there, banked a 9.65 percent lift and moved on. The plan says keep going, because 0.024567 is not yet the 0.01610 that this look demanded. The test runs on and finishes at 10.64 percent, closer to the true 10 percent effect, with a p-value an order of magnitude smaller.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Try the second row in the calculator above: 21,284 visitors and 1,078 conversions in control, 21,284 and 1,182 in the variant. The confidence interval comes back at 0.0627 to 0.9146 percentage points, which is to say it is still leaning on zero. It is a result that already counts as “significant” by the traditional rule and that, measured against the plan, is still not enough evidence to end the experiment at that moment.
Reproducing the FDA number
FDA guidance states that a group sequential design with a single interim analysis and a commonly used efficacy stopping boundary can reduce the expected sample size of the trial by roughly 15 percent relative to a comparable fixed sample trial. The footnote pins down the scenario: interim at half of the maximum sample, an O Brien-Fleming boundary, and the alternative hypothesis at which the design has 90 percent power.
We ran that exact scenario through our integrator. The boundaries came out at z of 2.7718 and 1.9793, the maximum N landed 0.79 percent above the fixed design, and the expected N under the alternative was 84.72 percent of the fixed N, a reduction of 15.28 percent. Under the null, the cost is 0.51 percent more than the fixed design. The agency’s number reproduces without adjustment, and now you know which scenario it came from, which matters because with three looks and 80 percent power the expected saving drops to 14.9 percent and the cost under the null rises to 1.6 percent.
As a check on the integrator itself, we used it to calibrate the classic boundaries: for three equally spaced looks and a two-sided 5 percent alpha it returned 3.4711, 2.4544 and 2.0040 for the classic O Brien-Fleming shape (published values: 3.471, 2.454 and 2.004) and 2.2895 for the Pocock constant (published value: 2.289).
Four things the boundary does not fix
1. The estimate from a test that stopped early runs hot. FDA guidance records that conventional estimates such as the sample mean tend to be biased toward greater effects than the true value under a group sequential design, and that confidence intervals lose their nominal coverage. Methods exist to correct this, and the plan has to name the one it will use. It is the same phenomenon we described in the winner’s curse, deliberately amplified by a design that only stops early when the result came in strong.
2. Moving the schedule after seeing the result breaks everything. The flexibility of alpha spending is about the number and timing of readings, not about reacting to what they showed. FDA guidance gives the example of what not to do: bringing the next interim analysis forward because the current one came close to the boundary. The recommendation is to hold a target number of analyses, an approximate schedule, and a written decision framework for changing them that depends on information independent of the estimated effect, such as the rate of data collection.
3. Futility stopping is a separate conversation. An efficacy boundary answers “can we already say it works”. A futility boundary answers “can we already say it is not worth continuing”. FDA guidance notes that adding nonbinding futility guidelines to a fixed sample trial, or to one with appropriate efficacy boundaries, does not increase the Type I error probability and is often appropriate. Binding rules let you relax the efficacy threshold, but they only control the error if they are actually obeyed. We covered the practical side in futility stopping.
4. None of it replaces the sanity checks. A test with an immaculate boundary and a broken traffic split is still a wrong test, just a more formal looking one.
Group sequential or always valid inference?
The two families solve the same problem and charge in different currencies.
| Criterion | Group sequential (alpha spending) | Always valid inference |
|---|---|---|
| Number of readings | finite, scheduled in advance | any number, including continuous |
| Must you fix dates up front? | yes, or at least the target count and the change rule | no |
| Cost in maximum sample | low with O Brien-Fleming (2.21% in our plan) | higher, because it reserves risk for infinitely many future looks |
| Whoever reads the dashboard daily | is not authorised to decide outside the looks | may decide at any moment |
| Best fit | a fixed decision ritual, few readings, scarce traffic | a dashboard open to the whole team, opportunistic decisions |
If your operation has a weekly results meeting and nobody decides outside it, group sequential is cheaper. If anyone can open the dashboard and act, always valid inference is the honest design, because it describes what actually happens at your company.
Make this automatic with Donnu
The failure this article describes is rarely statistical, it is operational: nobody wrote down when the test would be read and against what cutoff, so every reading becomes a fresh 5 percent decision. In Donnu the reading plan is part of creating the experiment: you pick the number of looks, the dashboard shows the cutoff each one requires, and it holds back the winner call when a reading has not crossed that look’s boundary, instead of showing a bare p-value that everyone reads as a green light. Planned readings are stored with their dates, so nobody can quietly pull one forward after seeing the result without it showing up in the history.
Frequently asked questions
The short answers live in the FAQ section of this page, built from the same calculations presented here.
References
- U.S. Food and Drug Administration. Adaptive Designs for Clinical Trials of Drugs and Biologics: Guidance for Industry, November 2019. Section V.A. Source of the definition of a group sequential design as prospectively planned interim analyses with prespecified stopping criteria, of the roughly 15 percent reduction in expected sample size with one interim at half and an O Brien-Fleming boundary at 90 percent power, of the contrast between O Brien-Fleming and Pocock in how persuasive early results must be, of the description of Lan-DeMets alpha spending, of the warning that moving an interim analysis earlier because of the current result inflates the Type I error, of the distinction between binding and nonbinding futility rules, and of the bias in estimates and loss of interval coverage under sequential designs.
- Ramesh Johari, Leo Pekelis and David Walsh. Always Valid Inference: Continuous Monitoring of A/B Tests, arXiv 1512.04922v3, 2019. Source of the diagnosis that traditional p-values and intervals become unreliable when sample size is chosen endogenously through continuous monitoring, of the A/A test example whose chance to beat baseline crosses the 95 percent threshold purely from being watched, and of the framing of always valid inference as the alternative that allows a decision at any moment.
- Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne. Controlled experiments on the web: survey and practical guide, Data Mining and Knowledge Discovery, 2009. Source of the online experimentation context used here, including the advice to determine the minimum sample size before starting and the warning about metrics whose power degrades the longer the experiment runs.
Read next
Frequently asked questions
- What is an interim analysis in A/B testing?
- It is a reading of the result taken before the test ends, at a point written into the plan before the test started, with its own stopping threshold for that point. The word doing the work is "planned": the number of readings and the p-value each one requires are fixed before the first user is randomized, and the total 5 percent alpha is split among them.
- How many times can I look without inflating the false positive rate?
- Zero times, if every look uses the 5 percent cutoff. In our numerical integration, two looks at 5 percent give an 8.31 percent false positive rate, three give 10.73 percent, five give 14.17 percent and twenty give 24.79 percent. With alpha spending boundaries the number of looks stops being the problem: two, three or five planned looks all land on 5 percent overall, because each one gets a stricter cutoff that pays for the repetition.
- What is the difference between O Brien-Fleming and Pocock boundaries?
- The shape of the boundary, which is really a decision about when to spend the alpha. The O Brien-Fleming style spending function spends almost nothing early and saves nearly everything for the end: with three looks it demands a p-value below 0.00069 at the first reading and allows 0.04388 at the last. The Pocock function spreads the budget almost evenly: 0.02264, 0.02174 and 0.02168. The first barely inflates the maximum sample size (2.21 percent in our plan) and rarely stops early; the second stops early often, and charges 17.03 percent more maximum sample size for the privilege.
- Is a group sequential design the same as sequential testing?
- No. Group sequential means a FINITE number of readings scheduled in advance, each with its own cutoff. Always valid inference, of the kind Johari, Pekelis and Walsh describe, lets you look at any moment, as often as you like, including continuously. Group sequential costs less sample size when two or three readings are genuinely all you need; always valid inference costs more, and in exchange removes the requirement to pick the dates up front.
- Can I move the interim analysis dates after the test has started?
- Only for reasons that do not look at the result. FDA guidance is explicit here: moving the next interim analysis earlier because the current one came close to the boundary is not appropriate and inflates the Type I error probability. Moving a date because enrollment ran slower than planned is acceptable, provided the decision framework was written in advance and depends on information that is statistically independent of the estimated treatment effect, such as the rate of data collection.
- Is the effect size trustworthy when a test stops early?
- The direction usually is, the magnitude is not. FDA guidance records that conventional fixed sample estimates such as the sample mean tend to be biased toward larger effects than the true value under a group sequential design, and that confidence intervals do not have the nominal coverage. That is why the plan has to name, from the start, which adjusted estimation method will be used to report a test that stopped early.