Carryover Effects Between A/B Tests: The Residue Problem
Carryover effects between A/B tests contaminate the next experiment on the same buckets. We measured 93% false positives in an A/A, plus the fix.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
When the second test reuses the same buckets as the first, the carryover effect of the first enters the read of the second as if it were a real effect. In the simulation in this article, an A/A test run right after an experiment that left the treatment group 0.60 percentage point down flagged a significant result in 93.30 percent of replicates, against the nominal 5 percent. With users rerandomized per experiment, the same measurement returned to 4.47 percent. The problem is not in the statistics or in traffic volume: it is in who was assigned where, and when. This guide shows why the bucket system creates the problem, measures how long the residue survives, proves that an SRM check will not catch it, and describes the fix. It is part of our complete guide to A/B testing and complements A/A tests for validating your setup and the novelty effect.
The bucket system and the price of convenience
Large platforms do not randomize users straight into experiments. They randomize a user once into a bucket, and then allocate buckets to experiments. Kohavi, Deng, Frasca, Longbotham, Walker and Xu describe the design and its advantage: it is flexible and allows easy reuse of users in subsequent experiments. The same group describes, in the paper on Bing’s platform at scale, the isolation mechanism running on top: a number line, equivalent to Google’s layers, guarantees that a user is in only one experiment per line, with assignment done through a pseudo random hash of an anonymous user id.
The convenience is real. So is the cost. If the randomization does not repeat per experiment, whoever was in treatment yesterday tends to be in treatment again today, carrying along everything the previous test did to them: the habit it created, the frustration it caused, the signup it completed, the cart it abandoned.
The classic result is in the paper by Kohavi, Deng, Frasca, Longbotham, Walker and Xu: an experiment ran and metrics unrelated to the change moved in unexpected directions, with high statistical significance. When it was rerun on a larger sample, many of those effects simply disappeared. The explanation was carryover.
How long the carryover effect survives
The authors report two measurements. In the first, the experiment ran in three stages: 7 days of A/A on the buckets, 47 days with the A/B turned on, then more than three weeks monitoring the same buckets with the experiment already off. The carryover effect on the main criterion, sessions per user, was visible and seems to die out at about the third week after the experiment ends. In the second, a bug exposed treatment users to a really bad experience: even after three months, the buckets had still not fully recovered to their pre-experiment levels.
We simulated that dynamic in order to measure its effect on decisions. The design: 1,000 buckets of 60 users each, 60,000 users per read, a 5 percent baseline rate, a test 1 that left the treatment group 0.60 percentage point down, and a residue decaying with a 14 day half-life. After that, test 2 runs as a pure A/A, that is, with no real effect at all, and we measure how often it flags significance. That is 3,000 replicates per row, with a Monte Carlo standard error of 0.40 percentage point around 5 percent.
| rest between tests | residue on that day | reusing buckets: false positive | mean measured lift | with rerandomization: false positive |
|---|---|---|---|---|
| 0 days | -0.6000 pp | 93.30% | -0.6032 pp | 4.47% |
| 7 days | -0.4243 pp | 67.93% | -0.4196 pp | 4.87% |
| 14 days | -0.3000 pp | 39.87% | -0.2982 pp | 4.90% |
| 21 days | -0.2121 pp | 23.10% | -0.2155 pp | 5.13% |
| 28 days | -0.1500 pp | 13.90% | -0.1496 pp | 6.00% |
| 42 days | -0.0750 pp | 7.03% | -0.0743 pp | 4.97% |
| 60 days | -0.0308 pp | 5.07% | -0.0292 pp | 4.47% |
| 90 days | -0.0070 pp | 4.77% | -0.0055 pp | 5.43% |
Two readings come out of that.
First: resting works, but it is far too expensive. Three weeks of waiting still left the false positive rate at 23.10 percent, nearly five times nominal. Only at 60 days of idle buckets did the read return to where it belongs. Few teams have the traffic capacity to give half their base a two month holiday between tests.
Second: rerandomization fixes it regardless of rest. The right-hand column sits between 4.47 and 6.00 percent on every row, including zero rest. That is the difference between waiting for the problem to fade and dismantling it.
A worked example, with the calculator
This is a single A/A test run immediately after the previous experiment, with no rest at all, on the same buckets:
- A (buckets that were in test 1 control): 30,000 visitors, 1,464 conversions, a rate of 4.8800 percent.
- B (buckets that were in test 1 treatment): 30,000 visitors, 1,300 conversions, a rate of 4.3333 percent.
Pasting those numbers into the significance calculator, the verdict is:
- z statistic of -3.1939
- p-value of 0.001404
- absolute difference of -0.5467 percentage point
- relative difference of -11.20 percent
- 95 percent confidence interval from -0.8821 to -0.2112 percentage point
- significant result, winner: A
A p-value of 0.001404 in a test where the two variants are identical. Nothing was changed. No code was shipped. The conclusion coming out of that dashboard, if nobody knows what ran before, is that the variant dropped conversion by 11 percent.
Run the numbers yourself in the calculator below:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
SRM does not catch this
The sample ratio mismatch check is the standard defence against broken randomization, and it is a good one: see SRM and unequal traffic split. It simply is not the defence for this problem.
Running the chi-square test on the split from the example above:
| check | result |
|---|---|
| visitors in A | 30,000 |
| visitors in B | 30,000 |
| expected split | 50/50 |
| chi-square statistic | 0.0000 |
| SRM p-value | 1.0000 |
| mismatch detected | no |
The randomization is perfect. The counts are perfect. The problem is not how many users landed in each arm, it is which ones. No proportion checker sees that, because the information that exposes carryover is not in the current test counts, it is in the history of who was where in the previous test.
This is the same trap as too good to be true and Twyman’s law: a result that is too strong on metrics that should not have moved at all is a sign of a measurement problem, not a discovery.
What happens to a test that has a real effect
The A/A case is instructive, but the real damage shows up when there is a genuine effect to measure and carryover distorts it. We ran the same design with a real effect of +4 percent relative, that is, +0.20 percentage point, 3,000 replicates per row:
| rest | reusing: significant result rate | mean measured lift | rerandomizing: significant rate | mean measured lift |
|---|---|---|---|---|
| 0 days | 63.33% | -0.4051 pp | 19.07% | +0.1931 pp |
| 7 days | 25.53% | -0.2274 pp | 21.33% | +0.2027 pp |
| 14 days | 8.57% | -0.0977 pp | 19.00% | +0.1950 pp |
| 21 days | 4.53% | -0.0167 pp | 20.53% | +0.1981 pp |
| 28 days | 6.10% | +0.0516 pp | 20.83% | +0.2015 pp |
| 60 days | 16.27% | +0.1698 pp | 19.23% | +0.1946 pp |
Read the measured lift column carefully. The real effect is +0.20 percentage point. With zero rest and reused buckets, the test measures -0.4051, that is, the wrong sign. And those 63.33 percent significant results are not wins: they are convictions. In nearly two thirds of replicates, a variant that genuinely improves conversion is declared a loser with statistical significance.
Notice the 21 day row too, where the significance rate falls to 4.53 percent. That is not good news: it is the point where the negative residue almost exactly cancels the real positive effect, and the test concludes a tie on an improvement that exists. Carryover does not always push results in one direction, it pushes them toward the previous test.
The rerandomization column, by contrast, measures between +0.1931 and +0.2027 percentage point at every interval, with the real effect of +0.2000 sitting in the middle of the range.
The fix: localized rerandomization
The root cause, in the words of the Bing authors, is direct: the bucket system does not rerandomize per experiment. It relies on an infrequent bucket reassignment, and between one reassignment and the next the allocation stays constant for a relatively long period.
Changing the hash function fixes it, but at a heavy operational cost: the bucket system couples all experiments in the same line, so changing the function requires stopping every experiment in that line, hurting capacity and agility.
The alternative they describe is a two-level bucket system:
- The top level defines which units enter the experiment.
- Treatment assignment happens at the second level, with a different hash seed for each experiment.
That guarantees per-experiment randomization, so treatment assignment becomes independent of any historical event, including carryover from previous experiments.
The cost they declare is a single one, and it matters: you cannot use a shared control. Each experiment needs its own control, so that any carryover from a prior experiment is mixed into control and treatment rather than concentrated on one side.
In exchange comes a benefit that pays for it well. With localized rerandomization you can run a retrospective A/A without spending calendar time: you change the hash function and re-evaluate the last few days before the experiment as if they were an A/A. By the independence property of localized rerandomization, retrospectively comparing users who would have been assigned to control and treatment for any period before the experiment is a legitimate A/A. If that A/A shows an effect on key metrics, say a p-value below 0.2, due to an unlucky split, you change the hashing key and retry, before spending a day of traffic.
What to do if you do not control the platform
Not every team can rewrite the assignment system of the tool it uses. What is left, in order of preference:
- Check whether your tool already rerandomizes per experiment. Many modern tools hash a combination of user id and experiment id, which is exactly localized rerandomization. If that is your case, the problem does not exist for you, and it is worth confirming that in writing rather than by assumption.
- Run a retrospective A/A before every test. Take the preceding days, apply the split the new test will use, and check that the key metrics tie. That is the test that would have caught the 93.30 percent case on the first attempt.
- Treat a p-value below 0.2 in the A/A as a yellow flag, not as a pass. The criterion comes from the Bing authors themselves and is deliberately loose, because the cost of a false alarm here is swapping a hash key, and the cost of letting one through is a quarter of wrong decisions.
- Record in the report which experiment ran before on those buckets. Without that record, nobody can even raise the carryover hypothesis when a result comes out strange.
- Be suspicious of metrics that moved for no reason. If the change was in the footer and the search metric moved with high significance, the leading hypothesis is not a discovery.
- Resting is the last resort, and you have to size it. In our simulation, 21 days still left a 23.10 percent false positive rate. If you are going to wait, wait properly and measure with an A/A before testing again.
Make it automatic with Donnu
Carryover is a randomization architecture problem, not an analysis problem. In Donnu, each experiment assignment comes from a hash combining the user id with that experiment id, so two consecutive tests never inherit the same split: whoever was treated yesterday is spread across control and treatment today. The dashboard also stores which experiments ran previously over the same population, so that the carryover hypothesis is the first thing anyone can check when an unrelated metric moves. If your current tool randomizes the user once and reuses the split, that is the item to change before any statistical refinement is worth anything.
Frequently asked questions
Short answers live in the FAQ section of this page, generated from the same measurements presented here.
References
- Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, KDD 2012. Carries the carryover section: the bucket system description, the case where unrelated metrics moved with high significance and vanished on a larger sample, the residue dying out around the third week in one case and not recovering in three months in the other, localized rerandomization through a two-level bucket system, the cost of the non-shared control, and the retrospective A/A with its p-value below 0.2 criterion.
- Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu and Nils Pohlmann. Online Controlled Experiments at Large Scale, KDD 2013. Describes Bing’s experimentation architecture, including the number line of orthogonal assignment (equivalent to Google’s layers), the isolation that guarantees one experiment per line, and the pseudo random hash of an anonymous user id applied as soon as the request arrives.
- Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu. Seven Rules of Thumb for Web Site Experimenters, KDD 2014. Context on why the sample size floor in online experimentation is higher than textbooks suggest, with the rule of 355 times the square of the skewness coefficient per variant, recommended when absolute skewness exceeds 1.
Read next
Frequently asked questions
- What is a carryover effect between A/B tests?
- It is when the same users who were affected by the first experiment get reused in the next one, and the residue of the first contaminates the read of the second. The term comes from the bucket system used by large platforms: users are randomized once into buckets, and it is the buckets that get allocated from experiment to experiment. Because the randomization does not repeat per test, whoever was in treatment yesterday tends to be in treatment again today, carrying along everything the previous test did to them.
- How long does a carryover effect last?
- Longer than intuition suggests. Kohavi and coauthors record two Bing cases: in one, a 47 day experiment left a measurable residue on the same buckets for about three weeks after it was turned off; in the other, a bug that exposed users to a really bad experience left the buckets still not fully recovered even after three months. In this article simulation, with a 14 day half-life, the false positive rate of an A/A test only returned to the nominal 5 percent after 60 days of rest.
- Does an A/A test catch carryover?
- It does, and it is the main defence available when you cannot rerandomize. In our simulation, an A/A run immediately after an experiment that left treatment 0.60 percentage point down flagged a significant result in 93.30 percent of replicates, against the expected 5 percent. One such A/A returned 4.8800 percent against 4.3333 percent, with a p-value of 0.001404 and a confidence interval from minus 0.8821 to minus 0.2112 percentage point: a strong and entirely false result.
- Will an SRM check detect this problem?
- No, and that is the dangerous part. SRM looks for deviation in the PROPORTION of users between arms, and under carryover the split is perfect. In our worked example both arms received exactly 30,000 users and the chi-square test returned a statistic of 0.0000 and a p-value of 1.0000. The randomization is right, the counts are right, and the read is still wrong. No proportion checker will raise its hand here.
- What is the definitive fix for carryover?
- Rerandomizing users per experiment, what Kohavi and coauthors call localized rerandomization. The implementation they describe is a two-level bucket system, where the first level defines who enters the experiment and the second performs treatment assignment using a different hash seed per experiment. In our simulation, with rerandomization the false positive rate stayed between 4.47 and 6.00 percent across every rest interval tested, including zero rest.
- What is the cost of rerandomizing per experiment?
- You lose the shared control. Kohavi and coauthors record that disadvantage explicitly: with per-experiment rerandomization, each experiment needs its own control, so that any carryover from a previous experiment is mixed equally into control and treatment. In exchange comes a gain that pays for it: you can run a retrospective A/A without spending calendar time, re-evaluating the last few days before the experiment as if they were an A/A. If that A/A shows an effect on key metrics, say a p-value below 0.2, you change the hashing key and retry.