Futility Stopping: When to Shut Down an A/B Test

An A/B test that has burned half its planned traffic and is sitting on a tie usually will not turn around: in the worked example below, conditional power at the halfway point is 29.2 percent assuming the planned effect shows up over the remaining stretch, and only 1.1 percent if the observed trend continues. Shutting that test down returns 82,376 visitors and 14 calendar days to the next hypothesis. This guide covers futility stopping end to end: how to compute conditional power, where to set the threshold, how much power the decision costs, and why the evidence recommends caution. It is part of our complete guide to A/B testing and the companion to sequential testing, which handles the other half of the problem: stopping early when the result is good.

Two early stops, two different problems

Stopping early for a win is the subject of every article about peeking: looking at the result repeatedly and stopping the first time the p-value crosses 0.05 inflates false positives, which is why sequential methods exist.

Stopping early for futility is the mirror problem, and it is easier in one respect and harder in another. Easier because there is no false positive risk: nobody is declaring a winner. The FDA states this explicitly when discussing sequential designs: adding non-binding futility guidelines to a fixed sample trial, or to one with appropriate group sequential stopping rules for efficacy, does not increase the type I error probability and is often appropriate.

Harder because the cost turns up on the other side: every futility stop is one more chance to abandon a real effect. Type II error goes up, and the decision is only defensible when that increase is quantified and accepted before the test starts.

Conditional power: the number behind the decision

The quantity that answers “is it worth continuing?” is conditional power. Zhang and Clarke define it as the probability that the final analysis will reject the null hypothesis, given the data observed at the interim analysis and an assumption about the effect. The complement, 1 minus conditional power, has been called the futility index since Ware, Muller and Braunwald.

The calculation uses the B-value, the monitoring tool formalised by Lan and Wittes. With t as the information fraction (the share of the planned sample already collected) and z as the test statistic at that moment:

// Conditional power at information fraction t (Zhang and Clarke, 2009)
// zAlpha: winning threshold (1.9600 for a two-sided 5 percent test)
// drift: assumed effect over the remaining stretch, on the scale of the final z
function conditionalPower(z, t, drift, zAlpha = 1.959964) {
  const numerator = z * Math.sqrt(t) + drift * (1 - t) - zAlpha;
  return normCdf(numerator / Math.sqrt(1 - t));
}
// drift for the PLANNED effect = zAlpha + zBeta = 1.9600 + 0.8416 = 2.8016
// drift for the CURRENT TREND = z / Math.sqrt(t)

One adaptation needs to be stated: Zhang and Clarke develop the method for a one-sided test, with the 1 minus alpha percentile in place of zAlpha. Here zAlpha is the two-sided threshold of 1.9600, because that is the win threshold of the standard A/B test used across this blog. The formula is the same; what changes is the win line the conditional power is projected against.

Choosing the drift is the conceptual decision of the method, and it changes everything. Assuming the planned effect will materialise over the remaining stretch is the optimistic assumption, and it is the default in the clinical literature. Assuming the observed trend continues is the realistic one, and it produces far smaller numbers. Zhang and Clarke explain why the first route takes so long to flag futility: at early times, conditional power implicitly weights the assumed future data under the planned alternative more heavily than the data accumulated so far, so it drops slowly from the design value down past the threshold.

The worked test

The design: a 3 percent baseline conversion rate, a minimum detectable effect of plus 8 percent relative, 95 percent confidence and 80 percent power. That calls for 82,376 visitors per variant, 164,752 in total, which at 40,000 visitors per week means 29 days.

At the halfway point, with 41,188 visitors per variant, the scoreboard reads 1,236 conversions in control (3.0009 percent) against 1,248 in the variant (3.0300 percent).

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Pasting those numbers into the calculator: z of 0.2445 and a p-value of 0.8069. Conditional power at the halfway point comes out as follows:

• Assuming the planned effect over the remaining stretch: 0.2924. That is, even if the variant starts delivering exactly the promised 8 percent from here to the end, the chance the test finishes significant is 29 percent.
• Assuming the observed trend continues: 0.0112. If the coming weeks look like the ones already run, the chance of a win is roughly one in ninety.

The gap between those two numbers is the conversation the team needs to have. The first asks “what if the hypothesis is right and the first half was bad luck?”. The second asks “what if what we are seeing is reality?”. Neither is wrong; they describe different scenarios, which is why the method asks that the threshold be chosen in advance, alongside the drift assumption.

The decision tables

With the design fixed (3 percent baseline, plus 8 percent relative, 95 percent and 80 percent), conditional power depends on only two things: where the test is and what its z is at that moment. Assuming the planned effect over the remaining stretch:

z at the interim analysis	quarter of the way	halfway	three quarters of the way
-1.0	0.3393	0.0367	0.0000
-0.5	0.4500	0.0984	0.0004
0.0	0.5648	0.2145	0.0059
+0.5	0.6743	0.3856	0.0492
+1.0	0.7705	0.5829	0.2156
+1.5	0.8483	0.7609	0.5315

Assuming instead that the observed trend continues:

z at the interim analysis	quarter of the way	halfway	three quarters of the way
-1.0	0.0000	0.0000	0.0000
-0.5	0.0003	0.0001	0.0000
0.0	0.0118	0.0028	0.0000
+0.5	0.1338	0.0382	0.0028
+1.0	0.5184	0.2201	0.0536
+1.5	0.8851	0.5903	0.3243

Two readings jump out. First: in the first quarter, almost nothing counts as futility under the planned assumption. A z of minus 1.0 (variant losing) still returns 34 percent conditional power, because three quarters of the experiment are still ahead. That is the effect Zhang and Clarke describe, and it is why very early futility analyses rarely earn their keep.

Second: from three quarters of the way onward, the verdict is nearly binary. With a z below 1.0 at that stage, conditional power is already at 21 percent or less even under the optimistic assumption. But by then only 25 percent of the traffic remains, so the saving is small while the risk of being wrong is not.

The useful point is the middle. Inverting the formula lets you publish the boundary directly in z, which is the number the team actually sees on the dashboard:

conditional power threshold	stop if z is below (quarter)	stop if z is below (halfway)	stop if z is below (three quarters)
0.10	-2.5022	-0.4908	+0.7145
0.20	-1.7402	-0.0508	+0.9685
0.30	-1.1907	+0.2664	+1.1517

The worked case had a z of 0.2445 at the halfway point, just below the 0.2664 boundary that corresponds to a 30 percent threshold, and well above the minus 0.4908 boundary that corresponds to the conservative 10 percent threshold. In other words: the loose rule kills the test, the conservative rule keeps it running. The rule has to be chosen beforehand, not after seeing that number.

The price of stopping, in power

Futility is not free, and the price is known. Zhang and Clarke cite the result of Lan, Simon and Halperin: with continuous monitoring and a rule that stops when conditional power falls below gamma, the overall type II error is bounded by beta divided by 1 minus gamma.

threshold gamma	maximum type II error (starting from 20 percent)	minimum final power	relative inflation of type II error
0.10	22.22 percent	77.78 percent	11.11 percent
0.20	25.00 percent	75.00 percent	25.00 percent
0.30	28.57 percent	71.43 percent	42.86 percent
0.50	40.00 percent	60.00 percent	100.00 percent

That is why the literature calls the 0.1 threshold conservative: it costs at most 11.11 percent of type II error inflation, which in practice is close to nothing, but in exchange it almost never fires. Moving to 0.3 doubles the aggressiveness and charges 42.86 percent inflation. Zhang and Clarke recommend keeping gamma below 0.5 and cite numerical evidence that aggressive boundaries push both errors, type I and type II, substantially away from their nominal levels.

What stopping gives back

The other side of the ledger is what the team gains. In the worked design, the whole test consumes 164,752 visitors and 29 days:

stopping point	traffic already spent	traffic returned	days returned
quarter of the way	41,188	123,564	21
halfway	82,376	82,376	14
three quarters of the way	123,564	41,188	7

Two weeks of returned traffic is half of another complete experiment on the same flow. In a programme that runs one test at a time, that is the throughput ceiling moving: it is not a cost saving, it is one more hypothesis tested per quarter.

The evidence recommends caution

Here is the honest counterweight, and it comes from data rather than opinion. Jitlal, Khan, Lee and Hackshaw retrospectively re-analysed ten randomised cancer trials that reached their target sample size, applying futility analyses after 25, 50 and 75 percent of events observed. The result cuts both ways.

On the favourable side: after 50 percent of the target number of events, three of the five trials with no benefit could have stopped early, with conditional power of 15 percent or less. In two of them the duration would have fallen by between 4 and 24 months, saving between 44,000 and 231,000 pounds (the third had already stopped recruiting, so no savings would have been made).

On the unfavourable side, which matters more: two of the four trials with moderate treatment effects could also have been stopped at some point, although they eventually showed worthwhile benefit. The authors’ conclusion is the one that should shape programme policy: futility should be used more often, because it stops future participants receiving an ineffective treatment, but unless there is very good evidence for futility, it is often best to continue to the planned end.

The translation to product experimentation is direct. The cost of being wrong here is not ethical, it is portfolio-shaped: killing a test that would have won means filing away a good hypothesis with a “tested” stamp on it, and it rarely makes its way back into the queue.

Binding rules against guidelines

The FDA separates two kinds of futility rule, and the distinction matters outside the clinical context too:

• Non-binding (guideline). The criterion is a guide; the final decision considers the totality of the interim results. It does not increase type I error, and it is the format the agency describes as often appropriate.
• Binding. The trial must always stop when the criterion is met. It brings advantages in the efficacy analysis, such as a relaxed threshold for declaring an effect, but type I error control only holds if the rule is followed. A trial that continues after crossing a binding boundary is likely to be considered as having failed to provide evidence of efficacy, whatever the final result.

For a product experimentation programme the guideline form is almost always the right pick: it gives the team the right to continue when context justifies it (an unusual holiday in the middle of the test, a campaign that shifted the traffic mix) at no statistical cost. What it demands is honesty about the criterion having been written first.

It is worth recording the order of magnitude that interim analysis buys, also from the FDA: a group sequential design with a single interim analysis and a commonly used efficacy stopping boundary reduces the expected sample size by roughly 15 percent relative to a comparable fixed sample trial. That is a real and modest gain, not a revolution, and it is a good calibration point for expectations.

How to put this in the plan

Four lines in the pre-registered analysis plan settle it:

1. How many interim analyses and when, expressed as information fractions (a single one at the halfway point, for instance). Fixing them by fraction rather than by date keeps the calendar from deciding the statistics.
2. The drift assumption: planned effect or observed trend. Publish both in the report and decide on one.
3. The threshold gamma and the type II error inflation it implies, written as a number.
4. Whether the rule is a guideline or binding, and who has the authority not to follow it.

A word on discipline: an interim futility analysis is not a licence to check the p-value daily. The futility boundary is about conditional power, it is one-directional, and it is declared in advance. Peeking at the result hunting for a win is still peeking, with the usual false positive cost.

Common mistakes

• Stopping for futility with no written rule. Without a threshold declared beforehand, “this test is going nowhere” is an opinion dressed as a calculation.
• Running the futility analysis too early. In the first quarter, under the planned assumption, almost nothing crosses the boundary, and the analysis only spends a meeting.
• Using only the optimistic assumption. It answers “what if the hypothesis is right?”, not “what are the data saying?”. Both calculations fit in the same report.
• Confusing futility with a proven tie. Low conditional power says the test will not conclude; claiming the versions are equivalent has its own procedure, equivalence testing.
• Ignoring the cost in power. Every threshold carries an associated type II error inflation, and it is computable.
• Treating a hypothesis killed for futility as a refuted hypothesis. The test stopped for lack of traffic to decide, not because absence of effect was proved. If the hypothesis is still good, the route is a redesigned test, not the archive.

Make this automatic with Donnu

The decision to shut down a test that cannot conclude usually arrives late, and out of fatigue, because the dashboard only shows the current p-value and the p-value does not answer the question “is it worth continuing?”.

At Donnu, every experiment starts with its planned sample and its information fraction visible on the dashboard, so the interim analysis happens at the agreed point instead of whenever someone remembers, and the report shows the distance still to go rather than just the current verdict. To redo the calculations by hand, the significance calculator returns the z for any interim cut, the test duration calculator turns the remaining sample into calendar days, and the test velocity calculator shows what the returned traffic buys in hypotheses tested.

References

• Zhang, Y. and Clarke, W. R. An Adaptive Futility Monitoring Method with Time-Varying Conditional Power Boundary. Department of Biostatistics, University of Iowa, 9 March 2009. Source of the definition of conditional power as the probability that the final analysis rejects the null given interim data and an assumed effect; of the closed-form conditional power expression built on the Brownian motion B-value (Lan and Wittes, 1988); of the note that 1 minus conditional power is the futility index of Ware, Muller and Braunwald (1985); of the convention that the threshold gamma sits between 0 and 0.5 and of the common practice of stopping when conditional power falls below 0.1; of the explanation for why conditional power falls slowly at early times (the calculation weights assumed future data under the planned alternative more heavily); of the type II error bound of beta over 1 minus gamma from Lan, Simon and Halperin (1982), with the example of less than 11.1 percent inflation for gamma of 0.1; and of the numerical evidence from Chang and Chuang-Stein that aggressive boundaries push both errors substantially away from nominal levels. public-health.uiowa.edu.
• U.S. Food and Drug Administration. Adaptive Designs for Clinical Trials of Drugs and Biologics: Guidance for Industry. CDER and CBER, November 2019. Source of the distinction between non-binding and binding futility rules; of the statement that adding non-binding futility guidelines to a fixed sample trial, or to one with appropriate group sequential stopping rules for efficacy, does not increase the type I error probability and is often appropriate; of the caveat that type I error control under binding rules only holds if they are followed, and that a trial continuing after crossing a binding boundary is likely to be considered as having failed to provide evidence of efficacy; and of the estimate that a group sequential design with a single interim analysis and a commonly used efficacy boundary reduces the expected sample size by roughly 15 percent relative to a comparable fixed sample trial. fda.gov.
• Jitlal, M., Khan, I., Lee, S. M. and Hackshaw, A. Stopping clinical trials early for futility: retrospective analysis of several randomised clinical studies. British Journal of Cancer, volume 107, pages 910 to 917, 2012. Source of the retrospective re-analysis of ten randomised cancer trials that reached their target sample size, with futility analyses applied after 25, 50 and 75 percent of events or patients recruited; of the finding that after 50 percent of the target number of events, three of the five trials with no benefit could have stopped early with conditional power of 15 percent or less; of the duration reduction of 4 to 24 months in two trials, saving 44,000 to 231,000 pounds; of the concerning finding that two of the four trials with moderate effects could also have been stopped despite eventually showing benefit; and of the conclusion that unless there is very good evidence for futility, it is often best to continue to the planned end. nature.com.
• Walker, E. and Nowacki, A. S. Understanding Equivalence and Noninferiority Testing. Journal of General Internal Medicine, volume 26, number 2, pages 192 to 196, 2011. Source of the distinction between failing to demonstrate a difference and demonstrating equivalence, used here to separate futility stopping from concluding a tie. pmc.ncbi.nlm.nih.gov.

Read next: Sequential testing explained · The peeking problem · Equivalence testing · Pre-registered analysis plan · Test duration calculator · Leia em português