Long-Term Holdout: The Effect That Survives

A long-term holdout is a slice of users deliberately kept outside a change that already shipped, so that months later there is still somebody to compare against. It exists because the number a two-week test produces measures the effect on a user who has not yet adapted to the change, and that adaptation can eat half the gain without anything in the original report warning you. This guide covers why leaving the test running longer does not solve the problem, a worked example where a plus 5.007 percent gain becomes plus 2.222 percent six months later, the post-period design Google published for measuring learning, the exponential model that tells you how long a study must run, and the arithmetic of how much extra traffic a 5 percent holdout costs. It is part of our complete guide to A/B testing and the natural companion to the novelty effect in A/B testing.

The two-week number measures something else

Hohnhold, O’Brien and Tang, in a KDD 2015 paper on focusing on the long term, separate the two quantities precisely. The short-term effect is the impact observed during the experiment. The long-term effect is the final impact once the product has fully launched and users have changed their behavior in response.

Their central claim is blunt: in online experimentation it is straightforward to measure the short-term effect, but the short-term effect is not always predictive of the long-term effect. The challenge, they write, is determining the long-term user impact while still being able to make decisions in a timely manner.

The concrete case they study is ads blindness and sightedness, the phenomenon of users changing their inherent propensity to click on or interact with ads. That is an example of a mechanism, not a rule about your product: what matters here is that an entire class of effects only surfaces after someone has lived with the change for weeks.

The obvious workaround does not work. It might seem enough to leave the test running for three months and read the gap at the end. The authors describe exactly that naive setup and explain why it does not yield reliable user learning measurements: the long-term effect measured at the end of the experiment period may change for many reasons unrelated to user learning, such as system effects, seasonality and interactions with subsequent launches. Disentangling those to measure long-term learning within one cohort over time is, in their experience, very difficult if not impossible.

Worked example: the effect that shrank

A subscription product runs a two-week test on its plan screen, with 150,000 users per arm.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Original test, 2 weeks	Users	Conversions	Rate
Control	150,000	6,750	4.500 percent
Treatment	150,000	7,088	4.725 percent

On screen, the calculator shows rates of 4.50 and 4.73 percent, a lift of plus 5.0 percent, a p-value of 0.0033, an interval of plus 0.1 to plus 0.4 percentage points and B as the winner. At full precision, and including the z score the interface does not display: plus 0.225 percentage points, plus 5.007 percent relative, z = 2.942, p-value 0.0033, with a 95 percent interval of plus 0.075 to plus 0.375 percentage points. It passed, it shipped to 100 percent, it became a line in the quarterly report.

Except the team had reserved 5 percent of users outside the launch. Six months later, comparing the holdout against everyone else:

Holdout, 6 months later	Users	Conversions	Rate
Holdout, without the change	200,000	9,000	4.500 percent
Exposed base	3,800,000	174,800	4.600 percent

The calculator shows rates of 4.50 and 4.60 percent, a lift of plus 2.2 percent, a p-value of 0.0374 and an interval of plus 0.0 to plus 0.2 percentage points. At full precision: plus 0.100 percentage points, plus 2.222 percent relative, z = 2.082, p-value 0.0374, with an interval of plus 0.007 to plus 0.193 percentage points.

The effect still exists and is still positive. But it is now worth 44.4 percent of what it was worth in the two-week reading. If the financial plan was built on 5.007 percent, it is overstating the gain by more than a factor of two.

Notice the confidence interval on that second reading too: it nearly touches zero despite involving 4 million users. That is the signature of unbalanced allocation, and the next section puts a price on it.

The designs: post-period, holdout and reverse experiment

Three designs commonly get called “measuring the long term”, and they are not interchangeable.

Post-period. The authors’ key insight is that to measure user learning you need to compare the two cohorts while they receive the same treatment. To achieve that they sandwich the treatment period between two A/A test periods: a pre-period, where they ensure there are no statistically significant differences between cohorts when the study starts, and a post-period, where any behavioral differences due to user learning are measured. For the purpose of quantifying learning, they note, the behavior of the treated cohort while receiving treatment is not of interest: only the post-period measurement matters. Post-periods, the paper reports, have proven to give reliable and reproducible user learning measurements at Google.

Holdout. The reserved slice stays outside the change indefinitely, and the comparison happens whenever you want it. Kohavi reports that at Bing 90 percent of eligible users are in experiments while 10 percent form a global holdout, changed once a year.

Reverse experiment. Kohavi also describes assigning credit by running a reverse or holdback experiment, that is, shipping the change and then testing it in reverse. Statistically it is the same design as a holdout, except the slice is carved out after launch rather than before.

Two post-period caveats the authors themselves record are worth carrying along:

• Unlearning. Since both cohorts receive the same treatment in the post-period, their behavior becomes more similar over time. Ideally the measurement would be taken in a brief window right at the start of the post-period; measuring over a longer window buys statistical power at the price of bias from unlearning.
• Measurement environment. The treatment in force during the post-period may affect the magnitude or nature of the learning observed. They give the example of a great new interface feature that users learn to interact with more over time: that cannot be observed in a post-period where the feature is absent. They also recommend confirming that metrics which should be unaffected by behavior change are consistent across cohorts, as a validity check.

How long the study has to run

This is where the Google paper turns into a practical tool. They model learning as an exponential process and estimate the learning rate at roughly 0.012 per day. The fraction of learning captured by a study of duration t is then 1 minus the exponential of minus 0.012 times t.

Applying their model:

Study duration	Fraction of very-long-run learning captured
7 days	8.1 percent
14 days	15.5 percent
30 days	30.2 percent
60 days	51.3 percent
90 days	about 65 percent (the figure reported in the paper)
180 days	88.5 percent

Based on that rate, the authors report that they typically run long-term desktop experiments for 90 days, which strikes a reasonable trade-off between study runtime and captured learning. And they push the honesty further: applying the same model to unlearning bias, if the measurement is taken in the first two weeks of the post-period, unlearning reduces the observed effect to 92 percent. Combining both factors, a standard study measures approximately 60 percent of the effect that would be observed in a very long running study.

The authors carry that arithmetic through to the business consequence. They define a correction factor converting the study measurement into the effect actually realized in a launch, and note that because the presented methods underestimate learning, this factor is at least 1. From the exponential model, a 90-day study measures about 65 percent of the long-term effect from limited duration alone, which already implies a factor of at least 1.54 for desktop. In practice, they say, they often use values between 2 and 3 for desktop and laptop devices to also compensate for treatment inconsistency, and they record that this range is only a rough estimate.

What a small holdout costs in traffic

A 5 percent holdout sounds cheap, because only 5 percent of users miss out on the improvement. Statistically, it is the most expensive design there is.

The reason is that unbalanced allocation wastes power. For an allocation ratio k between the large arm and the small one, the total traffic inflation factor, relative to a 50/50 test of equal power, is the square of 1 plus k divided by 4 times k.

Starting from the reference case of 31,234 users per variant, which is what the engine returns for a 5 percent baseline and a 10 percent relative minimum detectable effect:

Allocation	Inflation factor	Smaller arm	Total traffic needed
50 / 50	1.00	31,234	62,468
80 / 20	1.56	19,522	97,607
90 / 10	2.78	17,353	173,523
95 / 5	5.26	16,439	328,779
99 / 1	25.25	15,775	1,577,475

Two readings come out of that, and the second is the less obvious one:

1. The total explodes. A 5 percent holdout needs 5.26 times more total traffic than the equivalent balanced test. At 200,000 users per week, the balanced design takes 3 days and the 95/5 takes 12.
2. The reserved arm is small, and that is the problem. The smaller arm in the 95/5 design holds 16,439 users, nearly half the 31,234 of the balanced design. It is not the expensive part. The expensive part is that everything else has to grow to compensate for that arm’s scarcity, because the precision of a difference is capped by whichever side has fewer people.

Since holdouts usually run for months, time generally solves the volume problem on its own. What it does not solve is expectations: a 1 percent holdout on a mid-traffic product will simply never reach power to detect a realistic effect, however long it stays live. Price it first, with the statistical power calculator and the test duration calculator.

Two legitimate schools

There is no single answer on how much to invest in long-term measurement, and it is worth saying so without rooting for either side.

Kohavi frames the tension explicitly when listing the open challenges of the field: there are long-term impacts not being seen in 1 to 2 week experiments, and Google and Bing act differently on this point, with Google running long-running experiments, for instance 90 days, and Bing focusing on agility with 1 to 2 week experiments and iterations.

Both choices are defensible and solve different problems:

Stance	Gains	Loses
Long-running experiments	Measures the effect that survives; avoids overstating gains in the financial plan	Pace. Every decision takes months
Short cycles with iteration	Volume of learning per quarter; mistakes get fixed fast	Exposure to effects that only appear later
Short cycles plus a global holdout	Day-to-day pace, plus a periodic read on the accumulated total	Costs a permanent slice of the base, and the discipline not to touch it

That third row is the middle ground a global holdout makes possible, and it is the design the Bing account describes: decide fast day to day, and once a year get a reading of what the whole set of launches actually delivered.

Holdout checklist

1. Was the slice reserved before the launch? Reserving it afterwards is a reverse experiment, and it does not cover the same period.
2. Was the slice size calculated rather than chosen by habit? Five percent is a convention, not the output of a computation.
3. Is there a clean pre-period? No statistically significant difference between cohorts before the start, which is an A/A test by another name.
4. Is the primary metric the same as the original test’s? Swapping metrics midstream turns a comparison into a narrative, see choosing the primary metric.
5. Is somebody accountable for not touching the slice? A holdout that receives “just one small launch” stops being a holdout.
6. Is the holdout’s randomization unit the same as the test’s? See randomization unit.
7. Will the long-term reading replace the number in the plan, or just sit next to it? If nobody corrects the projection, measuring changes nothing.

Common mistakes

• Reading the holdout as if it were a fresh test. It carries no fresh randomization at read time; it carries months of history.
• Leaving the test running and calling that long-term measurement. That is the naive design the authors reject for confounding learning with seasonality, system effects and later launches.
• Comparing the holdout against a historical average. Without a simultaneous group, anything that happened in the market becomes your result.
• Contaminating the slice with launches. Every exception reduces the holdout to a partially exposed group, and the measured effect shrinks with no explanation.
• Expecting the long-term effect to always be smaller. It is not always. The Google paper describes curves that climb after launch through positive user learning; shrinking is common, not compulsory.
• Using a 1 percent holdout and complaining about the wide interval. The inflation factor at that allocation is 25.25. The wide interval is the arithmetic, not a flaw in the tool.

Make this automatic with Donnu

What kills a holdout is almost never the statistics. It is forgetting it exists: six months on, nobody remembers the slice is reserved, somebody ships into it, and the measurement quietly disappears.

In Donnu, the reserved slice is a property of the experiment rather than a parallel setting: it stays declared after the launch, and any attempt to expose that slice to a new change surfaces as a conflict instead of happening silently. The long-term reading uses the same math and the same unit as the original reading, side by side, so the decay is visible instead of having to be reconstructed by hand. To redo the arithmetic elsewhere, the p-value calculator and the sample size calculator take raw counts from both arms, unbalanced included.

References

• Hohnhold, H., O’Brien, D. and Tang, D. Focusing on the Long-term: It’s Good for Users and Business. KDD 2015, Google. Source of the distinction between the short-term effect as the impact observed during the experiment and the long-term effect as the final impact once the product has fully launched and users have changed behavior, of the claim that the short term is not always predictive of the long term, of the framing of deciding in a timely manner, of the definition of ads blindness and sightedness as users changing their inherent propensity to click on or interact with ads, of the explanation of why the naive setup fails due to system effects, seasonality and interactions with subsequent launches, of the post-period method sandwiching treatment between two A/A periods and the insight that cohorts must be compared while receiving the same treatment, of the report that post-periods gave reliable and reproducible measurements at Google, of the unlearning and measurement-environment caveats including the absent interface feature example and the check on metrics unaffected by behavior, of the learning rate of approximately 0.012 per day, of the practice of running 90-day desktop studies capturing about 65 percent of learning, of the 92 percent unlearning factor for a 14-day measurement and the resulting approximately 60 percent, and of the correction factor at least equal to 1 with a minimum of 1.54 and practical use between 2 and 3 on desktop and laptop. research.google.
• Kohavi, R. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. KDD 2015 keynote. Source of the report that 90 percent of eligible Bing users are in experiments while 10 percent form a global holdout changed once a year, of the practice of assigning credit by running a reverse or holdback experiment, of the warning about carryover effects from prior experiments, and of the note that Google and Bing act differently on long-term impacts, with Google running long experiments of for instance 90 days and Bing focusing on agility with 1 to 2 week experiments and iterations. exp-platform.com.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Reference on organizational metrics rather than feature metrics, and on treating borderline significant results as tentative, which applies directly to reading a holdout whose interval sits close to zero. exp-platform.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. General reference on long-term effects, holdouts and unbalanced allocation. Companion material at experimentguide.com.

Read next: The novelty effect in A/B testing · Randomization unit · A/A testing: validate the setup · The winner’s curse · Guardrail metrics · Statistical power calculator · Leia em português