Switchback Experiments: Testing Under Network Effects

In a marketplace, both arms of an A/B test draw from the same pool of supply, so assigning one user changes another user outcome and the experiment measures a diluted effect. Switchback experiments fix this by changing the randomization unit: instead of drawing users, you draw time windows within each region, and every time and region pair becomes one experimental unit. This guide covers why user level tests shrink the effect in a system with shared supply, a worked example where the same true 2 point lift shows up as 1 point in the wrong design and costs 4.1 times the sample, how to choose window length between bias and margin of error, the optimal design established in the literature, and why the p-value your calculator returns needs a correction. It is part of our complete guide to A/B testing and it is the design that solves the problem described in interference between variants.

The problem: both arms share one fleet

Standard A/B testing rests on an assumption that rarely gets said out loud, namely that one user outcome does not depend on which variant everyone else received. On a content site that is almost always true. In a marketplace it almost never is.

The DoorDash Dispatch team published the clearest example of this. They use surge pricing when there are too few couriers for incoming order volume: the delivery fee rises, some demand shifts to later, and more couriers decide to go out and work. Suppose you want to test a change to that algorithm as a consumer level A/B test, with half seeing surge pricing and half not. Their account of what happens is blunt: the first group gets only half the benefit of supply equilibration, and the second group gets part of the benefit without any extra pay. Both groups still share the same courier fleet, so independence between them does not exist.

The practical effect is always in the same direction, and it is downward. Control is contaminated by treatment, the gap between arms shrinks, and the test concludes that the change is worth less than it is.

What switchback experiments actually are

In a switchback you flip an entire region between control and treatment over time, drawing at random which variant applies in each window. DoorDash describes the mechanism in one sentence: you switch back and forth between control and treatment algorithms in a certain region at alternating time periods, then compare metrics from the control time buckets against those from the treatment time buckets.

Two implementation details separate a correct switchback from naive alternation:

1. The variant for each window is randomized, not alternated. Their write up is explicit here: they randomize the variant used for each time window rather than randomizing the initial variant and flipping back and forth deterministically. Fixed alternation is vulnerable to any business cycle whose period resembles the flip rate.
2. Regions are randomized independently of each other. Region A can run the current algorithm in one window and the new one in the next, while region B does the reverse. That is what creates what they call time-region units.

Worked example: same effect, two designs

A delivery marketplace wants to test a new dispatch algorithm. Its true effect on the share of deliveries arriving within the promised window is 2.000 percentage points, from 85.000 percent to 87.000 percent, when the algorithm governs the whole market.

Design 1, consumer level A/B test. Half the consumers are dispatched by the new algorithm, half by the old one, and both halves compete for the same fleet. The benefit spreads, because a more efficient fleet is more efficient for everyone. Assume half the gain leaks, a conservative scenario. Paste the numbers into the calculator:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Consumer level test	Deliveries	On time	Measured rate
A, current algorithm, contaminated by leakage	120,000	102,000	85.000 percent
B, new algorithm	120,000	103,200	86.000 percent

The calculator returns plus 1.000 percentage point, plus 1.176 percent relative, z = 6.957, p-value below 0.00001, and a 95 percent interval of plus 0.718 to plus 1.282 percentage points. The result is significant and wrong by half.

Design 2, switchback. The same 240,000 deliveries, now organized into randomized 30 minute windows per region. No window splits the fleet:

Switchback by time and region	Deliveries	On time	Measured rate
Windows in control	120,000	102,000	85.000 percent
Windows in treatment	120,000	104,400	87.000 percent

Now the calculator returns plus 2.000 percentage points, plus 2.353 percent relative, z = 14.119, p-value below 0.00001, and an interval of plus 1.722 to plus 2.278 percentage points.

The point is not that the first design came out non significant, because with 120,000 deliveries per arm it did come out significant. The point is the sample price. Detecting 2.000 percentage points on an 85 percent baseline at 95 percent confidence and 80 percent power takes 4,724 per arm. Detecting the diluted 1.000 point takes 19,461 per arm. Leakage charges you 4.1 times the sample to see the same product change, and you can redo that arithmetic in the sample size calculator.

The analysis unit has to match the randomization unit

This is the most expensive switchback mistake, and it is subtle because it looks like throwing data away. You randomized windows, so the analysis runs on windows, not on deliveries.

DoorDash says it plainly: they run their statistical tests on the average values of control and treatment time-region units, rather than on the individual values of control and treatment deliveries. In practice that means each arm average is a simple average across units, not a delivery average weighted by how busy each window was.

The gap between those two averages is not cosmetic, and they record a useful diagnostic about it: the simple and weighted averages usually converge over time, and when they do not, that is a sign the intervention affects units with many deliveries differently from units with few. Compute both, precisely for that signal.

Question	User level A/B test	Switchback
What gets randomized	user or device	time window paired with region
What enters the statistical test	user	time-region unit
Typical sample size	hundreds of thousands	thousands of windows
Independence assumption	plausible	violated by construction
Handles supply side interference	no	yes
Cost	low	loses power, needs more calendar time

Choosing the window: bias on one side, margin of error on the other

Window length and region size are the design decision that actually matters, and DoorDash frames the choice as a tug of war between two problems.

Bias. It appears when unit randomization is compromised and the kinds of deliveries landing in each arm stop being similar on average. Their example is precise: an algorithm reluctant to accept deliveries with long store to customer distances may be observed as faster without being better, because it cherry picked the short runs and left the long ones for control to clean up. That bias becomes more likely when regions get too small or you switch back too frequently, since the next window inherits the work the previous one pushed forward.

Margin of error. It runs the other way. The more granular the unit, the higher the natural variation of a metric inside it, but also the more units you have. Since margin of error is proportional to natural variation divided by the square root of the number of units, there is a balance point, and it is found empirically. They found theirs by running a series of long A/A tests and measuring how margin of error changed as they switched faster or slower. The reported outcome: 30 minute windows on geographic divisions roughly at the city level.

The optimal design, according to the theory

The empirical side of that choice got a formal treatment in Design and Analysis of Switchback Experiments by Iavor Bojinov, David Simchi-Levi and Jinglong Zhao. They cast the design as a minimax optimization problem and reach two results that matter directly to anyone about to run one.

First: the coin has to be fair. Theorem 1 in the paper establishes that any optimal design must use assignment probability 1 over 2 at every randomization point. Splitting 70 against 30 is not a conservative choice here, it is a worse one.

Second: the period should track the carryover effect. The carryover effect is how long a treatment keeps influencing outcomes after it is switched off. Writing m for the order of that effect and T for the total horizon, when T is a multiple of m with at least four blocks, the optimal randomization points are time 1 and then 2m+1, 3m+1, and so on up to (n-2)m+1. The practical consequence is that the first and last blocks are twice as long as the middle ones.

Horizon T = 12 periods, carryover of order m = 2	Periods
Optimal randomization points	1, 5, 7, 9
Opening block	periods 1 to 4, length 2m
Middle blocks	5 to 6 and 7 to 8, length m
Closing block	periods 9 to 12, length 2m

The authors draw a practical implication from this that deserves a slow read: the optimal randomization frequency depends on the physical duration of the carryover effect, regardless of the granularity of any single experimental period. In other words, the right question is not how many minutes you want, it is how long your system takes to forget the treatment. If the market needs 25 minutes to rebalance supply after a pricing change, that is what sets the period, not reporting convenience.

They also note a useful robustness result: when there is no carryover (m equal to zero) or very little (m equal to 1), the optimal designs are almost the same. Erring toward assuming a short carryover is cheap.

The p-value your calculator returns is optimistic

Here is the part no calculator warns you about, ours included. The two proportion test you just ran assumes independent units. In a switchback that assumption is false by construction.

The reason is physical. DoorDash puts it in concrete terms: average delivery completion time in one area during a ten minute chunk is related to and highly correlated with the same area in the next ten minute chunk, far more so than one delivery is with the next. Neighboring windows share weather, traffic, promotions and the same courier shift.

Positive correlation between units in the same arm means underestimated variance, which means a standard error that is too small, which means a p-value that is too small. Their correction is a variance estimator robust to the lack of independence, of the sandwich type, and they report that across the A/A tests they ran, the effect of that estimator on their variance calculations stayed below 10 percent.

It is worth sizing that correction on our own example. Inflating variance by 10 percent multiplies the standard error by the square root of 1.10, that is by 1.0488. The switchback z of 14.119 falls to 13.462, and the result stays miles from any significance threshold. That is the good news buried in their number: when correlation between windows is moderate, the correction changes the arithmetic and rarely changes the decision. The bad news is that this is not a law, it is a measurement they took on their own system. Measure it on yours, with A/A tests, before assuming your correlation is small too. The procedure is the one described in A/A tests and validation.

Bojinov, Simchi-Levi and Zhao offer two more rigorous routes for anyone who wants them: exact randomization based p-values, and conservative tests built on a finite population central limit theorem. Both drop the normality assumption that the two proportion approximation carries.

Checklist before running a switchback

1. Does the interference actually exist? If the arms compete for nothing, a switchback only costs you power. A regular A/B test is better whenever it is valid.
2. What is the order of the carryover effect? How long your system takes to return to baseline after the treatment is turned off. That number, not convenience, sets the window.
3. Is each window genuinely randomized? A fresh draw per window, fifty fifty, independent across regions.
4. Does the analysis run on windows? Simple average across time-region units. Compute the weighted one too, but only to compare them.
5. Was the variance corrected for dependence between neighboring windows? Without it the p-value is optimistic.
6. Did randomization pass a health check? DoorDash reports that simply checking whether the expected proportion of deliveries landed in each arm catches most bias cases, and that when they do notice bias they shut the experiment down and restart with coarser time or geographic units. The same discipline as sample ratio mismatch.
7. Does the result match reality after launch? They verify by watching the metric time series before and after the full rollout, looking for directional confirmation rather than exact agreement.

Common mistakes

• Alternating instead of randomizing. Deterministic alternation aligns with any business cycle of similar period, and then you do not have an experiment, you have a comparison of times of day.
• Analyzing per delivery after randomizing per window. It inflates the sample artificially and produces confidence intervals that describe nothing. The same randomization unit confusion that shows up in ratio metrics.
• Shortening the window to buy power. It works in the spreadsheet and creates bias on the street, because treatment hands unfinished work to control.
• Using a switchback where it is not needed. The design costs power and calendar time. Outside systems with a shared resource, that is a price with no product.
• Trusting the raw p-value. Without a variance correction it describes an experiment of independent units that you did not run.

Make this automatic with Donnu

What makes a switchback hard is not the statistics, it is the discipline of keeping the randomization unit and the analysis unit matched all the way through, with a good looking result already on screen.

In Donnu, an experiment report always declares which unit was randomized and runs the test on that unit, with no option to swap the unit at analysis time because the number looks better that way. When the design has clustered units, the dependence warning sits above the result, not in a footnote. And if you want to redo any arithmetic by hand, the p-value calculator accepts raw counts from any slice.

References

• Kastelman, D. and Ramesh, R. Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash. DoorDash, February 13, 2018. Source for the surge pricing example and the explanation that both groups share the same fleet, the definition of time-region units, the decision to randomize the variant for each window rather than alternate deterministically, the rule of analyzing on simple averages of units rather than individual deliveries, the sandwich variance estimator with an effect below 10 percent in their A/A tests, the tug of war between bias and margin of error, the example of the algorithm reluctant to take long deliveries, and the operational choice of 30 minute windows at roughly city level divisions. careersatdoordash.com.
• Bojinov, I., Simchi-Levi, D. and Zhao, J. Design and Analysis of Switchback Experiments. arXiv 2009.00148, version of September 17, 2025. Source for Theorem 1 on the optimality of fair coin flipping, Theorem 2 with the optimal randomization points and the worked case of T equal to 12 with m equal to 2, the observation that optimal frequency depends on the physical duration of the carryover effect and the recommendation that each period be almost as long as the order of that effect, the robustness result when m is zero or 1, and the two inference approaches proposed. arxiv.org.
• Chamandy, N. Experimentation in a Ridesharing Marketplace. Lyft Engineering, 2016. Three part series cited by the DoorDash team itself as prior work on experimentation under network effects in mobility marketplaces. eng.lyft.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. General reference on violations of the independent units assumption and alternative randomization designs. Companion material at experimentguide.com.

Read next: Interference between variants · Concurrent experiments and interaction effects · Ratio metrics and the delta method · A/A tests and validation · P-value calculator · Leia em português