Concurrent Experiments: When Interaction Effects Matter
Why concurrent experiments are usually safe to run, what a real interaction effect looks like, and the assignment bug that is far more likely than one.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Running concurrent experiments on the same users is the normal state of a working experimentation program, not a compromise. The premise that makes it valid is that each experiment can be analyzed in isolation, which holds as long as assignments are independent and the treatments do not interact, and the far more common way that premise fails is a randomization defect rather than a genuine interaction. This guide covers what an interaction effect actually is, why the published experience from large platforms is that they are rare, the assignment bugs that masquerade as them, and a worked 2x2 example showing why “no significant interaction” usually means the test could never have found one. It is part of our complete guide to A/B testing and pairs with testing multiple variants.
Overlapping is the default, not the exception
At any meaningful volume, isolation stops being affordable. Kohavi and colleagues, describing the Bing experimentation system for KDD 2013, report over 200 concurrent experiments running on any given day, and put the combinatorics starkly: with five variants in each of fifteen concurrent experimentation areas, by their own description conservative numbers, users end up in one of five to the power of fifteen, roughly 30 billion possible variants of Bing. Nobody validates 30 billion configurations. The system has to be built so that validating them is unnecessary.
Google’s answer, described by Tang, Agarwal, O’Brien and Meyer for KDD 2010, is a layered infrastructure with three concepts: a domain is a segmentation of traffic, a layer corresponds to a subset of the system parameters, and an experiment is a segmentation of traffic where parameters get alternate values. Parameters that cannot be varied independently of each other go into the same layer, experiments within a layer are mutually exclusive, and traffic diversion into experiments in different layers is orthogonal. A request passes through one experiment per layer.
Their motivating example is the clearest one-sentence case for why layers exist at all: with one parameter for the background colour of a page and another for the text colour, blue is a valid value for both, and if both are blue at the same time the page is unreadable. That is a real interaction, and no amount of statistics fixes it. It is prevented by construction instead.
What an interaction effect actually is
Kohavi and colleagues define it precisely: a statistical interaction between two treatments A and B exists if their combined effect is not the same as the sum of the two individual treatment effects. They then name why it matters, and the sentence is the whole reason this topic exists: the existence of interaction violates the basic premise used to scale experimentation, that each experiment can be analyzed in isolation.
They list three harms. Interactions can harm users, because particular combinations trigger unexpected bugs. They skew results for all experiments involved, which matters most when the real treatment effect is small, since a small interaction can then produce completely misleading results on a key metric. And they cannot be fully prevented through testing and offline checks, because different teams do not know what other teams are shipping.
That is the case for taking interactions seriously. Here is the case for not panicking about them, from the same paper: in our experience, interactions are relatively rare and more often represent bugs than true statistical interactions. They continue to prefer orthogonal assignment, equivalent to a full factorial, and run small multivariate tests only when an interaction is suspected or detected. The 2009 survey from the same group makes a related argument, that in the online world the availability of users and the agility of continuous testing make concurrent uni-variate tests preferable to traditional multivariate designs, whose fractional and Plackett-Burman layouts confound two-factor interactions with main effects anyway.
The operational reading is a priority order: when the combined cell looks wrong, look for a bug before you look for an interaction.
The failure that is far more likely than an interaction
Independence of assignment is an assumption, and it is one that hashing schemes break quietly. Kohavi, Longbotham, Sommerfield and Henne tested this directly in 2009. Their setup hashes the concatenation of a user identifier and an experiment identifier, then partitions the range. They ran five simulated experiments against one million sequential user ids and applied chi-square tests looking for correlations between experiments. The results:
| Hash function | Result of the correlation tests |
|---|---|
| MD5 | Generated no correlations between experiments |
| SHA256 | Came close, requiring a five-way interaction to produce a correlation |
| The string hashing algorithm built into .NET | Failed to pass even a two-way interaction test |
And then the failure worth memorizing, because it is an optimization any competent engineer might reach for. MD5 is expensive, so one system tried to cache: hash the experiment name once, hash the user id once, store both, and XOR them at assignment time. The paper reports the consequence exactly. With two experiments at 50/50, if the most significant bit of the hashes of the two experiment names matched, users would always get the same assignment in both experiments; if it did not match, users would get exactly the opposite assignment. Either way the assignments are perfectly correlated and the results of both experiments are confounded.
Nothing about that shows up as a strange number in either experiment read on its own. Both look normal. Fabijan and colleagues (KDD 2019) list the same property as the third requirement of an assignment service, alongside equal probability and consistency across repeat visits: when multiple experiments are run, there must be no correlation between experiments. The way you find a violation is by looking for it, which is the argument for a permanently running validation experiment, covered in A/A testing, and for the split checks in sample ratio mismatch.
Worked example: a 2x2 factorial, and why the interaction says nothing
Two teams ship at once on the same checkout flow. Experiment A changes the shipping-cost banner. Experiment B changes the payment button. Assignment is independent, so users land in four cells of 50,000 each, at a 4 percent baseline conversion rate. Read the cells in the calculator as you go:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
| Cell | Users | Conversions | Rate |
|---|---|---|---|
| Neither change | 50,000 | 2,000 | 4.000 percent |
| A only | 50,000 | 2,120 | 4.240 percent |
| B only | 50,000 | 2,090 | 4.180 percent |
| Both A and B | 50,000 | 2,215 | 4.430 percent |
Main effect of A. Collapse over B: 4,090 conversions in 100,000 without A against 4,335 in 100,000 with A. That is 4.090 percent against 4.335 percent, +0.245 percentage points, +5.99 percent relative, z = 2.73, p-value 0.0064, interval +0.069 to +0.421 percentage points.
Main effect of B. Collapse over A: 4,120 against 4,305 out of 100,000 each, so 4.120 percent against 4.305 percent, +0.185 percentage points, +4.49 percent relative, z = 2.06, p-value 0.0395, interval +0.009 to +0.361 percentage points.
The interaction. The effect of A when B is off is 4.240 minus 4.000, or +0.240 percentage points. The effect of A when B is on is 4.430 minus 4.180, or +0.250 percentage points. The interaction is the difference of those differences: +0.010 percentage points. Its standard error is the square root of the four cell variances summed, which is 0.1797 percentage points, giving z = 0.06, p-value 0.9556, interval -0.342 to +0.362 percentage points.
Read literally, that says no interaction. Read honestly, it says something weaker and much more useful. The standard error of the interaction is exactly twice the standard error of a main effect, which is the general result for a difference of differences and the reason an interaction of a given size needs about four times the sample of a main effect of that size. At this design, the smallest interaction detectable at 80 percent power is 0.491 percentage points, about 12.3 percent relative, while the main effects resolve down to about 0.246 percentage points.
So the correct sentence in the write-up is not “there is no interaction between A and B”. It is “no interaction larger than about half a percentage point was detectable at this sample, and the observed gap is 0.01”. Given that each main effect is smaller than the interaction the test could have seen, an interaction big enough to change either decision would have shown up. That is the argument that actually licenses shipping both, and it is a power argument, not a p-value argument. The same reading discipline applies to any inconclusive result, as covered in minimum detectable effect.
Testing every pair does not scale, and Bing says so
If you decide to check all pairs of running experiments for interactions, the pair count grows quadratically and the false positives grow with it:
| Concurrent experiments | Pairs to test | Chance at least one pair looks interactive by chance, at 5 percent |
|---|---|---|
| 2 | 1 | 5.0 percent |
| 4 | 6 | 26.5 percent |
| 6 | 15 | 53.7 percent |
| 10 | 45 | 90.1 percent |
| 15 | 105 | 99.5 percent |
| 20 | 190 | 99.99 percent |
Kohavi and colleagues state the problem in exactly these terms: if we are running N experiments at a time, the complexity of detecting pairwise interactions is quadratic in N, and because of the scale their system must sometimes run hundreds of thousands of hypothesis tests. Their answer is not to stop testing but to control the error rate, using an empirical Bayesian false discovery rate algorithm to identify the cases most likely to be true positives, after which the tool runs a deeper diagnosis and alerts the experiment owners.
For a team running a handful of experiments rather than hundreds, the practical version is simpler: do not scan for pairwise interactions by default. Check a specific pair when there is a reason to, and if you do check many, apply a correction, the same arithmetic covered in testing multiple variants.
Deciding what to isolate
| Situation | Overlap or isolate | Why |
|---|---|---|
| Two changes on unrelated surfaces, for example checkout and onboarding | Overlap | The premise of independence is plausible and the traffic is worth more elsewhere |
| Two changes to the same element or the same configuration parameter | Isolate | This is the background-colour and text-colour case; the combination can be broken by construction |
| Two changes on the same page, different elements | Overlap, but declare the pair as one to check afterwards | Interactions are rare but this is where they concentrate |
| A change that alters who is eligible for the other experiment | Isolate | Eligibility becomes a post-treatment variable and both readings are compromised |
| Any two experiments where a combined state was never built or reviewed | Isolate until reviewed | The most common real interaction is a bug in an untested combination |
| A pricing or billing change alongside anything | Isolate | Combined states here are expensive to get wrong and hard to reverse |
Prevention beats detection, and the mechanisms used at scale are worth copying at small scale. Bing, per the 2013 paper, has each experiment declare a set of constraints so the system refuses to run conflicting experiments together, for example ensuring a user is never in two ad visual experiments at once; uses configuration management to detect experiments trying to change the same parameter before launch; and, when interactions cannot be avoided, uses mappings to exclude users of one experiment from another. Google’s launch layers and nested domains, per the 2010 paper, are the same idea expressed as infrastructure.
Common mistakes with concurrent experiments
| Mistake | What it produces |
|---|---|
| Reusing one hash of the user id across all experiments | Every experiment gets the same split of users, and all of them are confounded |
| Caching hashes and combining them with XOR | Perfectly correlated or perfectly opposite assignment, per the 2009 survey |
| Reading “no significant interaction” as “no interaction” | A conclusion the design had roughly a coin flip of supporting |
| Scanning all pairs for interactions without multiplicity control | At ten experiments, about a 90 percent chance of a false interaction to chase |
| Serializing every experiment to avoid interactions | The rarest risk avoided at the cost of the program’s throughput |
| Running two experiments on the same element and reading both as clean | The one configuration where interactions actually concentrate |
| Treating a suspected interaction as statistics before checking for a bug | Contradicts the published experience that interactions more often represent bugs |
| Letting one experiment change eligibility for another | Post-treatment selection, which no analysis recovers from |
Automate this with Donnu
Concurrent experiments are safe when the platform makes the assignments independent and tells you when they are not. Donnu A/B salts the assignment hash with the experiment identifier so two experiments never inherit each other’s split, runs the sample ratio check on every experiment rather than only on the one you are watching, and lets you declare an experiment mutually exclusive with another when the combination should not exist. Overlap stays the default, because it is what lets a small team run more than one test at a time, and the isolation is available for the specific pairs that need it.
Start a free 14-day trial and run your next two experiments at the same time without guessing whether they collide.
References
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the over 200 concurrent experiments at Bing and the five to the power of fifteen figure, of the definition of a statistical interaction and the three harms it causes, of the statement that interactions are relatively rare and more often represent bugs than true statistical interactions, of the constraint, configuration-management and mapping prevention mechanisms, and of the quadratic pairwise detection problem handled with an empirical Bayesian false discovery rate algorithm. exp-platform.com.
- Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R. M. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 2009. Source of the hash-and-partition method, of the chi-square tests over five simulated experiments and one million sequential user ids in which only MD5 produced no correlations while SHA256 required a five-way interaction and the .NET string hash failed a two-way test, of the XOR caching shortcut that produces identical or exactly opposite assignments, and of the argument for concurrent uni-variate tests over traditional multivariate designs. exp-platform.com.
- Tang, D., Agarwal, A., O’Brien, D. and Meyer, M. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. KDD 2010. Source of the domain, layer and experiment definitions, of the rule that parameters which cannot vary independently share a layer while diversion across layers is orthogonal, of launch layers, and of the background-colour and text-colour example of a combination that breaks by construction. static.googleusercontent.com.
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. and Dmitriev, P. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. KDD 2019. Source of the three requirements of an assignment service, including that there must be no correlation between experiments when several run at once. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on experimentation platforms, concurrent experiments and randomization. Companion material at experimentguide.com.
Read also: A/A testing · Testing multiple variants · Sample ratio mismatch · Minimum detectable effect · Free A/B/n significance calculator
Frequently asked questions
- Can I run several A/B tests at the same time on the same users?
- Yes, and at any real scale you have no choice. Kohavi and colleagues (KDD 2013) describe over 200 concurrent experiments running at Bing on any given day, with users landing in one of roughly 30 billion possible variants of the site. The requirement is not isolation, it is independent randomization: a user assignment in one experiment must not change the probability of their assignment in another. Get that right and each experiment can be analyzed on its own.
- What is an interaction effect between two experiments?
- A statistical interaction between treatments A and B exists when their combined effect is not the same as the sum of the two individual effects, as Kohavi and colleagues (KDD 2013) define it. The plain version: if A gives plus 0.24 points on its own and B gives plus 0.18 on its own, additivity predicts plus 0.42 when both run. An interaction is the gap between that prediction and what the combined cell actually shows. The problem is that this gap violates the premise that lets you analyze each experiment in isolation.
- How common are interaction effects in practice?
- Rare. Kohavi and colleagues (KDD 2013), writing from a platform running hundreds of experiments a day, state that in their experience interactions are relatively rare and more often represent bugs than true statistical interactions. That is why they continue to prefer orthogonal assignment, equivalent to a full factorial, over multivariate testing. The practical consequence is that a suspected interaction should be investigated as a defect first and as statistics second.
- How much sample do I need to detect an interaction?
- About four times what you need for a main effect of the same size, because the interaction is a difference of differences and its standard error is roughly twice as large. In a 2x2 design with 50,000 users per cell at a 4 percent baseline, a main effect resolves down to about 0.25 percentage points at 80 percent power while the interaction only resolves down to about 0.50. So most tests that report no significant interaction were never powered to find one, and the honest phrasing is that no interaction larger than X was detected.
- What actually breaks when experiments overlap?
- Assignment correlation, far more often than a genuine interaction. Kohavi and colleagues (2009) tested five simulated experiments against a million sequential user ids and found that only MD5 produced no correlation between experiments, while the string hash built into .NET failed even a two-way test. They also document a caching shortcut that XORed separately hashed experiment and user ids, which made users land in identical or exactly opposite assignments across two experiments depending on one bit. Both experiments are confounded and neither is readable.
- When should experiments be made mutually exclusive?
- When the combination can produce a broken or harmful experience, when two changes touch the same surface or the same configuration parameter, or when a specific interaction is genuinely suspected. Tang and colleagues (KDD 2010) give the clean illustration: with one parameter for page background colour and another for text colour, blue is a valid value for each and a disaster for both at once. Their answer is to group parameters that cannot vary independently into a layer, so experiments inside a layer are exclusive while different layers overlap freely.
- Should I test for interactions between every running pair of experiments?
- Only with multiplicity control, because the pair count grows quadratically. Ten concurrent experiments make 45 pairs, which at a 5 percent threshold gives about a 90 percent chance that at least one pair looks interactive by chance alone. Kohavi and colleagues (KDD 2013) describe monitoring all running experiments for pairwise interactions, sometimes running hundreds of thousands of hypothesis tests, and using an empirical Bayesian false discovery rate algorithm to keep the false positives down.