Simpson's Paradox in A/B Testing: Segments vs Total

Simpson’s paradox is when your A/B test reports a clear winner in the total while every segment inside it reports the opposite. The arithmetic is not broken: the aggregate is a weighted average, and when the two arms carry different mixes of users, the total can point somewhere no segment does. This guide covers a worked example with real numbers, the causes that produce it in online experiments, the one check that catches it before it fools anyone, and how to fix a test that already fell into it. It is part of our complete guide to A/B testing and depends on the diagnostic in sample ratio mismatch.

A worked example where every segment loses and the total wins

A pricing page test runs for two weeks and ends with 30,000 visitors in each arm, a perfectly even top-level split. The dashboard says the variation won by a lot. Here is the whole test, split by device.

Segment	Control visitors	Control conversions	Control rate	Variation visitors	Variation conversions	Variation rate	Winner
Desktop	10,000	600	6.00 percent	20,000	1,160	5.80 percent	Control, by 3.33 percent relative
Mobile	20,000	400	2.00 percent	10,000	190	1.90 percent	Control, by 5.00 percent relative
Total	30,000	1,000	3.333 percent	30,000	1,350	4.500 percent	Variation, by 35.0 percent relative

Every number in that table is correct. Check them yourself:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

The aggregate comparison gives z = 7.37 and a p-value of about 0.0000000000002, a lift of +35.0 percent, confidence interval +0.86 to +1.48 percentage points. If a result like that landed on your dashboard you would ship it without hesitating.

Now the segments. Desktop: 6.00 percent against 5.80 percent, z = -0.69, p-value 0.4871, control ahead. Mobile: 2.00 percent against 1.90 percent, z = -0.59, p-value 0.5565, control ahead again. Neither is significant on its own, but both point the same way, and the direction is the opposite of the total.

Why Simpson’s paradox happens: the total is a weighted average

Look at the segment mix inside each arm rather than at the conversion rates.

• Control: 10,000 desktop and 20,000 mobile, so 33.3 percent desktop and 66.7 percent mobile.
• Variation: 20,000 desktop and 10,000 mobile, so 66.7 percent desktop and 33.3 percent mobile.

Desktop converts three times better than mobile. The variation was handed twice as much of the good segment, and that is worth far more than the small penalty it carries inside each segment. Crook and colleagues at Microsoft, writing about experimentation pitfalls in 2009, put the algebra plainly: it is entirely possible for one fraction to be smaller than another, and a second fraction also to be smaller than its counterpart, while the sum of the numerators over the sum of the denominators reverses the comparison.

Their own example is a ramp-up, and it is worth reproducing because it is the case most teams will actually meet. A site with one million visitors a day runs an experiment at a 99 to 1 split on Friday and raises the treatment to 50 percent on Saturday. The treatment converts better on Friday (2.30 percent against 2.02 percent) and better on Saturday (1.20 percent against 1.00 percent), yet combining the two days makes the treatment look worse: 1.20 percent against 1.68 percent. The direction of the paradox flipped compared with our device example, but the mechanism is identical. Saturday was a worse day overall, and it made up half of the treatment’s data and only a third of the control’s.

The one check that catches it: segment-level split verification

Simpson’s paradox needs unequal allocation to exist. A sample ratio mismatch check measures unequal allocation directly, so running it inside each segment is the diagnostic.

In our example the numbers are stark. The experiment total is 30,000 against 30,000, giving a chi-square of 0.00 and a p-value of 1.00: a flawless pass at the top level. Inside desktop the split is 10,000 against 20,000, which gives a chi-square of 3,333.33 on one degree of freedom and a p-value indistinguishable from zero. Mobile is the mirror image, chi-square 3,333.33 in the other direction. A test can pass its top-level split check and still be built from two completely unbalanced halves.

Dmitriev and colleagues make this an explicit rule, and add the case that most teams do not see coming: the segment definition itself must not be affected by the treatment. They describe a Bing ranking experiment where users who saw a certain extra link and users who did not both showed a significant increase in sessions per user, while the combined population showed no change at all. The experiment had not moved sessions per user. It had moved which users landed in which segment, and the less active users who dropped out of the first group raised its average while also raising the average of the group they joined. Their recommendation is direct: test each segment group for sample ratio mismatch, and when the ratio differs significantly, the results for that segment group, and usually for every group in that segment, are invalid and should be ignored.

There is a second discipline in the same section. Recursively slicing the population until something reaches significance is a separate mistake that also invites the paradox, and the arithmetic is unkind: with independent segment groups read at a 5 percent threshold, about 1 in 20 will look significant by chance alone. Declare the segments you intend to report before launch, or apply a correction such as Bonferroni when you analyze them afterwards. The mechanics of that correction are in testing multiple variants.

How to fix a test that already fell into it

Crook and colleagues list three remedies, and are candid about which one they actually use.

Remedy	How it works	When it fits
Discard the ramp-up data	Analyze only the period where the allocation was stable	Their preferred fix, because ramp-up is normally short compared with the whole test
Pair within stable periods	Compare control and treatment inside each window where the proportions did not change, then combine	When allocation changed more than once and the stable stretches are long enough to be useful
Weighted combination	Reweight each segment by its true share of traffic before averaging	When the imbalance is across segments rather than across time

The third one applied to our example is instructive. Desktop and mobile each account for 30,000 of the 60,000 visitors, so weighting them equally gives a corrected estimate of 4.000 percent for the control (half of 6.00 plus half of 2.00) and 3.850 percent for the variation (half of 5.80 plus half of 1.90). The apparent +35.0 percent win becomes a 3.75 percent loss, which is the direction both segments were pointing all along.

The cleanest way to see what the test would have said is to rerun it with the mix balanced. Give each arm 15,000 desktop and 15,000 mobile visitors at exactly the same segment conversion rates, and the control finishes with 1,200 conversions on 30,000 visitors (4.000 percent) against 1,155 for the variation (3.850 percent). That comparison gives z = -0.95, p-value 0.3441, confidence interval -0.46 to +0.16 percentage points. Not a 35 percent winner. Not a proven loser either. Inconclusive, leaning slightly negative, which is a very different sentence from the one on the original dashboard.

Where the imbalance behind Simpson’s paradox comes from

Cause	What it looks like	Prevention
Ramp-up across the analysis window	Allocation starts at 5 or 10 percent and is raised mid-test, then everything is analyzed together	Discard the ramp-up period, or fix the allocation before the measurement window opens
Allocation that differs by region or platform	One country or app version runs a different split because a local team sized it independently	Analyze per region and never pool arms with different splits
A capped segment	High-value customers deliberately limited to a small share of the treatment	Report that segment separately and never fold it into a headline number
Non-uniform sampling	Some browsers or device classes sampled at a higher rate for coverage reasons	Reweight by the true population share before combining
A segment the treatment itself moves	Membership in the segment depends on something the change makes more or less likely	Run the segment-level split check; a failure means every group in that segment is unreadable
Redirect or load failures on one arm	One variation quietly loses a slice of a segment because it is slower or breaks somewhere	This is a plain sample ratio mismatch, and the fix is root cause plus rerun

Notice that only the last row is a defect. The others are ordinary, sensible operational choices that become dangerous only at the moment someone pools the arms into a single number without checking whether the pooling was legitimate.

Common mistakes with segments and totals

Mistake	What it produces
Reading only the total	A weighted-average artifact ships as a 35 percent win
Reading only the segments	The opposite artifact, plus a multiple comparison problem nobody corrected
Running the split check only at the top level	The check that would have caught it passes perfectly, as it does in the example above
Slicing until something turns significant	About 1 in 20 independent slices looks significant by chance, and the paradox becomes likelier the deeper you go
Segmenting on something the treatment changes	Every group in that segment is unreadable, however clean the numbers look
Calling it a heterogeneous effect	Leads to shipping the change to one device class when the real problem was an allocation bug
Reweighting and declaring victory	Recovers the direction but not the precision; a badly contaminated test should be rerun

If the segments genuinely disagree with each other on a test whose allocation was clean inside every segment, that is a different and much more interesting situation: a real heterogeneous treatment effect, which our guide to common A/B testing mistakes covers alongside the validity threats it is often confused with.

Automate this with Donnu

Simpson’s paradox survives on a dashboard that shows one aggregate number and hides the split behind it. Donnu A/B runs the sample ratio check inside each reported segment as well as on the experiment total, flags a segment whose allocation does not match the configured split before its conversion numbers are shown, and keeps the segment mix of each arm visible next to the headline result. When the total and the segments point in opposite directions, that contradiction is surfaced as a warning rather than left for someone to notice in a meeting.

Start a free 14-day trial and check the split inside your own segments before reading a single conversion rate.

References

• Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source of the ramp-up example with its Friday and Saturday splits, of the four situations that produce the paradox in online experiments, and of the three remedies including discarding the ramp-up period. It also traces the name to Simpson (1951). exp-platform.com.
• Dmitriev, P., Gupta, S., Kim, D. W. and Vaz, G. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments. KDD 2017. Source of the Bing ranking case where both segments moved and the total did not, of the rule that segment definitions must not be affected by the treatment, of the recommendation to run an SRM test per segment group, and of the 1 in 20 arithmetic for recursive segmentation. exp-platform.com.
• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. On ramping allocation and on automated data quality monitoring in a large experimentation platform. exp-platform.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on experiment trustworthiness, segmentation and ramp-up. Companion material at experimentguide.com.

Read also: Sample ratio mismatch · A/B testing statistical significance · Testing multiple variants · Common A/B testing mistakes · Free SRM checker · Leia em português