Interference Between Variants: When the Arms Talk
Interference: when treatment affects control, your A/B test measures the wrong difference. The leakage channels, the bias they create, how to reduce it.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Every A/B test silently assumes that treating one user does not change any other user’s outcome. When that assumption breaks, and it breaks whenever both arms compete for something finite, control stops being a clean baseline: it becomes a group the experiment itself has already harmed. The measured difference now adds the treatment’s gain to the control’s loss, and the result can easily come out at double the real effect. This guide covers the three channels interference travels through, a worked example where a plus 22 percent winner is really a plus 10 percent winner, what eBay measured in a real experiment, and what you can do about it without abandoning experimentation. It is part of our complete guide to A/B testing and pairs with guardrail metrics.
The silent assumption interference breaks
The two-proportion test compares arm A’s rate with arm B’s rate and calls the difference the treatment effect. That is only valid if each user’s outcome depends solely on the arm they landed in, and not on what happened to anyone else. The causal inference literature names this condition SUTVA, the stable unit treatment value assumption.
It is reasonable when you change a button colour. It is false whenever something finite sits in the path.
The dangerous part is that interference leaves no statistical fingerprint. A contaminated experiment produces a small p-value, a tight confidence interval and a green bar on the dashboard, exactly like a clean one. There is no p-value correction that fixes it, because the problem is not in the variance, it is in what control represents. Running longer makes things worse, since more sample only narrows the interval around the biased number.
Three channels the leakage travels through
| Channel | What it looks like | Typical signal |
|---|---|---|
| Finite resource | Inventory, seats, ad supply, budget, support capacity | Per user metric moves, aggregate total does not |
| Shared infrastructure | Cache, queue, connection pool, rate limit, a model trained on both arms | Unevenly split A/A test fails |
| Connections between people | Invites, messages, feed, referrals, public content visible to both arms | Effect grows with the percentage exposed |
The first channel is the best known and the easiest to explain to people outside the data team: if there is a fixed amount of something, what treatment takes extra is what control does not get. The second is the sneakiest, because no user interacts with another user at all. The third is the classic social product case, and it is the only one of the three where the leakage can inflate control rather than depress it, when treatment produces content or invites that control then consumes.
The second channel deserves one more paragraph, because it catches teams who are certain they do not run a marketplace. In his paper on unexpected results in SIGKDD Explorations, Kohavi reports a 90 and 10 percent A/A test that failed consistently. The cause was a bounded resource: a least recently used cache in which control and treatment entries were disjoint. Because the experiment ran at 90 against 10, control had significantly more entries in the cache, which produced a higher cache hit ratio, better performance and therefore better metrics. No user spoke to any other user. The infrastructure leaked on its own.
Worked example: the plus 22 percent winner
A store with limited inventory tests a storefront change. It runs 100,000 users per variant, and the period has 5,400 units available, a real ceiling that does not move during the test. Paste the numbers into the calculator:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Without an inventory ceiling, control would convert at its historical 3.000 percent, or 3,000 orders, and the variant would convert at 3.300 percent, or 3,300 orders. But the shelf cannot carry both: the variant pulls 3,300 units, and what is left for control closes at 2,700 orders instead of 3,000. The 300 orders control lost did not vanish from the world, they went to treatment.
That is what the dashboard showed:
| Naive comparison | Users | Orders | Rate |
|---|---|---|---|
| A, control | 100,000 | 2,700 | 2.700 percent |
| B, variant | 100,000 | 3,300 | 3.300 percent |
The calculator returns plus 0.600 percentage points, plus 22.22 percent relative, z = 7.865, p-value below 0.00001, with a 95 percent interval of plus 0.450 to plus 0.750 percentage points. A crushing winner by any team’s standards.
Now the comparison that matters, between the variant and what control would have done in a world without cannibalisation:
| True effect | Users | Orders | Rate |
|---|---|---|---|
| A, control without interference | 100,000 | 3,000 | 3.000 percent |
| B, variant | 100,000 | 3,300 | 3.300 percent |
The calculator returns plus 0.300 percentage points, plus 10.00 percent relative, z = 3.841, p-value 0.00012, with an interval of plus 0.147 to plus 0.453 percentage points.
The change is real and worth shipping. The honest number is half of what the dashboard showed. The measured absolute lift, 0.600 percentage points, is exactly double the true one, 0.300, because every displaced order appears twice in the gap between arms: once adding to treatment and once subtracting from control. That doubling is the arithmetic signature of finite resource interference, and it is why a factor of two shows up so often.
What eBay measured
The example above is arithmetic built to teach the mechanism. The real measurement exists, and it is larger than intuition suggests.
Thomas Blake and Dominic Coey, of eBay Research Labs, published a study of an eBay email marketing campaign at the ACM Conference on Economics and Computation in 2014. Their starting point is the same one: classical statistical inference assumes the treatment affects the test group but not the control group, and that assumption is violated when experimenting in marketplaces because of general equilibrium effects, since changing test demand affects the supply available to the control group.
The result is blunt. In the paper’s words, ignoring test-control interference leads to estimates of the campaign’s effectiveness which are too large by a factor of around two. The user level estimate came to 0.74 percent. Redone at the auction level, with an average of 1.40 test bidders and 1.39 control bidders per auction, the calculation gave roughly 0.35 percent, over a sample of more than 10.4 million auctions. They also record the economics of the bias: it is larger in magnitude where supply is more inelastic, and it is positive if demand is elastic, meaning the naive comparison overstates the treatment’s benefit.
The recommendation closing the paper is the same one that applies to any team today: a better strategy may be to compare units at a higher level of aggregation, such as auctions instead of individuals, across which interference is less pronounced.
How to reduce the bias
There is no statistical correction for a contaminated control. What there is, is choosing a randomization unit that contains the interaction instead of cutting across it.
- Randomize by cluster, not by user. If interference happens within a city, randomize cities. If it happens within a corporate account, randomize accounts. If it happens within an auction, compare auctions. The arm becomes a closed set and the leakage stays inside it.
- Randomize by time window. Switching the whole site between control and treatment in blocks of hours or days removes leakage between simultaneous arms, at the cost of exposure to seasonality and trend. It works best where the shared resource replenishes quickly.
- Hold treatment at a small fraction. The smaller the exposure, the less pressure on the shared resource and the smaller the bias. This does not remove the problem, and uneven splits bring their own risks, but it works as a sensitivity read.
- Measure the aggregate, not just the user. If the per user metric rises and the total market metric does not move, you have not found growth, you have found displacement. That comparison is cheap and should be standard on any test touching a finite resource.
The cost of these alternatives is always the same: fewer independent units, therefore less power. Randomizing 40 cities instead of 200,000 users collapses the power of the design, and it has to be sized honestly, which usually means running longer. It is a trade of precision for the absence of bias, and it is worth it because a biased estimate does not improve with sample.
How to detect it before it hurts
- Unevenly split A/A test. Run 90 and 10 and see whether it passes. This is the detector that caught the cache problem in Kohavi’s account, and it is cheap. An evenly split A/A test can pass even with interference present, because the effect is symmetric; the uneven split is what breaks the symmetry. Worth reading alongside A/A tests and platform validation.
- Exposure ramp. Measure the effect at 5, 20 and 50 percent exposure. In a clean experiment the estimated effect should not depend on arm size. If it shrinks as you widen, interference is present.
- Aggregate panel. Put the total metric next to the per user metric on every test that touches inventory, budget or a queue.
- Ratio check. Unintended uneven splits have several causes, and interference is one of them. The sample ratio mismatch procedure is the first stop.
Common mistakes
- Assuming it is a marketplace problem. Anything finite qualifies: inventory, media budget, human support, cache, course seats, a promotional batch.
- Running longer to fix it. More time narrows the interval around the biased estimate and raises confidence in the wrong conclusion.
- Confusing interference with interaction between experiments. They are different things. Interaction is two distinct experiments whose effects do not add. Interference is a single experiment where arm A changes arm B’s outcome. We cover the former in concurrent experiments and interaction effects.
- Randomizing by cluster and analysing by user. If the randomization unit is the city, the analysis unit has to be too, or the confidence interval comes out too narrow. That is the same mechanism described in ratio metrics.
Make this automatic with Donnu
Interference is a design problem, so the moment to solve it is before running, which is exactly when nobody is thinking about it.
In Donnu, the randomization unit is an explicit choice at experiment creation rather than a silent default: you can randomize by user, by account or by geographic cluster, and the analysis automatically uses the same unit, avoiding the mismatch that squeezes the confidence interval. Any experiment flagged as sensitive to a finite resource carries the aggregate metric next to the per user metric in the same report, so displacement disguised as growth is visible at decision time. To size the power cost of cluster randomization, the sample size calculator accepts the number of independent units you actually have.
References
- Blake, T. and Coey, D. Why Marketplace Experimentation Is Harder than it Seems: The Role of Test-Control Interference. ACM Conference on Economics and Computation, 2014. Source of the premise that classical inference assumes treatment affects test but not control, the violation via general equilibrium effects, the conclusion that ignoring interference produces estimates too large by a factor of around two, the 0.74 percent user level estimate against roughly 0.35 percent at auction level with averages of 1.40 and 1.39 bidders per auction over more than 10.4 million auctions, the finding that bias is larger under inelastic supply and positive under elastic demand, and the recommendation to compare units at a higher level of aggregation. dominiccoey.github.io.
- Kohavi, R. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations, 12(2), 2010. Source of the 90 and 10 percent A/A test that failed consistently because of a least recently used cache with disjoint entries across arms, with control accumulating more entries, a higher hit ratio and better metrics, and of the recommendation to start with A/A tests and stay vigilant about performance. kdd.org.
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the 0.1 to 1.0 percent range for successful experiments once diluted and of the principle that results should be read as overall impact rather than segment impact. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on leakage and interference between variants and on choosing the randomization unit. Companion material at experimentguide.com.
Read also: A/A test validation · Sample ratio mismatch · Ratio metrics · Guardrail metrics · Statistical power calculator · Leia em português
Frequently asked questions
- What is interference between variants in A/B testing?
- Interference is when what happens to a treatment user changes the outcome of a control user. A standard A/B test assumes this does not happen, an assumption the causal inference literature calls SUTVA. When it breaks, control stops being a clean baseline and becomes a group the experiment itself has already affected, so the measured difference between arms is no longer the treatment effect. This is a design problem, not a statistical one: no p-value adjustment repairs a contaminated control.
- How large is the error interference causes?
- It depends on the channel and it can be very large. Blake and Coey of eBay Research Labs studied an eBay email campaign and concluded, in their paper at the ACM Conference on Economics and Computation in 2014, that ignoring test-control interference leads to estimates of the campaign effectiveness which are too large by a factor of around two. The user level estimate was 0.74 percent, while the auction level analysis pointed to roughly 0.35 percent.
- Does interference only happen in marketplaces?
- No. Marketplaces are just the most visible case, because supply is obviously finite. The same mechanism appears with limited inventory, shared media budget, shared human support capacity and shared infrastructure. Kohavi reports a 90 and 10 percent A/A test that failed consistently because both arms used a least recently used cache with disjoint entries, so the larger arm ended up with a better hit ratio. No user interacted with another user in that case, and the arms contaminated each other anyway.
- How do you reduce interference bias?
- By randomizing on a unit that contains the interaction rather than cutting across it. Blake and Coey suggest exactly this, concluding that a better strategy may be to compare units at a higher level of aggregation, such as auctions instead of individuals, across which interference is less pronounced. In practice this becomes randomizing by geographic region, by market, by account or by time window. You trade statistical power for the absence of bias, and that trade is almost always worth making.
- Can you detect interference before making the wrong call?
- Partly. Three signals help: an unevenly split A/A test that fails, an aggregate metric that does not move while the per user metric moves a lot, and an effect that shrinks as you raise the percentage exposed to treatment. That third signal is the most informative, because under interference the measured effect depends on arm size, and in a clean experiment it should not.