Heterogeneous Treatment Effects in A/B Testing

A significant effect on mobile and a null one on desktop does not prove the effect differs by device. Claiming that requires testing the difference between the two effects directly, and that test is always noisier than either one: in this article’s worked example, the mobile effect has a p-value of 0.006 while the difference between mobile and desktop has a p-value of 0.219. This guide covers how to claim heterogeneous treatment effects rigorously: the correct difference-in-differences arithmetic, why it needs roughly 5 times more sample in the worked case, Gelman’s theoretical ceiling of 16 times, and how to hunt for segments in the data without invalidating the confidence interval. It is part of our complete A/B testing guide and is the operational companion to the pre-registered analysis plan.

The average is a decision, not a description

Every A/B test reports one number per metric, and that number is the average treatment effect over the population that entered the experiment. It is a legitimate statistic and it is the one that supports the launch decision. What it is not: a description of what happened to each user.

The same average effect of half a percentage point can come from three completely different worlds:

• Everybody rose half a point.
• Half rose a full point and half stayed flat.
• A third rose two points and two thirds fell half a point.

The launch decision may be the same in all three (the aggregate is positive), but the decision about what to do next is radically different. And that is where the segment panel enters, almost always through the wrong door.

Worked example: the test wins and somebody wants to ship mobile only

A store runs a test with 60,000 users per arm spread across four devices. Paste any row into the calculator to reproduce the numbers.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

segment	A (conv./visitors)	B (conv./visitors)	rate A	rate B	effect	z	p-value
Mobile	1,500 / 30,000	1,650 / 30,000	5.00%	5.50%	+0.500pp	2.7457	0.006039
Desktop	1,260 / 18,000	1,278 / 18,000	7.00%	7.10%	+0.100pp	0.3706	0.710938
Tablet	300 / 6,000	306 / 6,000	5.00%	5.10%	+0.100pp	0.2501	0.802486
App	420 / 6,000	414 / 6,000	7.00%	6.90%	-0.100pp	-0.2154	0.829469
Total	3,480 / 60,000	3,648 / 60,000	5.80%	6.08%	+0.280pp	2.0517	0.040195

(The four segments sum to exactly 60,000 visitors and 3,480 against 3,648 conversions, so the total is arithmetically consistent with its parts.)

The total is significant: the test won. And then comes the conclusion that looks obvious and is not: “the gain is all mobile, desktop did not react, so let us ship mobile only and save the desktop work”.

That conclusion is not supported by the data above, and the reason is precise.

The difference between significant and not significant is not significant

Gelman and Stern treated exactly this in a paper whose title is the thesis: the difference between “significant” and “not significant” is not itself statistically significant. They are explicit that they are not making the commonplace observation that any threshold is arbitrary; the claim is stronger: even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities.

Their theoretical example is three lines long. Two independent studies with effect estimates and standard errors of 25 plus or minus 10 and 10 plus or minus 10. The first is statistically significant at the 1 percent level; the second is not at all significant, sitting only one standard error away from zero. It would be tempting to conclude there is a large difference between the two studies. But the estimated difference is 15, with a standard error equal to the square root of 10 squared plus 10 squared, that is the square root of 200, which the authors round to about 14. That gives a z of 1.0607 and a p-value of 0.2888. Not even close.

The authors add a second case that closes the argument from the other side: if a third study with a much larger sample estimates 2.5 with a standard error of 1.0, it attains the same significance level as the first, and yet the difference between those two is itself significant. Comparing significance labels tells you nothing about magnitudes.

The correct arithmetic: difference in differences

The question “does the effect differ between mobile and desktop” has a statistic of its own, and it is the difference in differences.

Step by step, with the example’s numbers:

1. Mobile effect: 5.50% minus 5.00% equals +0.500pp. Standard error of the difference: the square root of 0.05 times 0.95 over 30,000, plus 0.055 times 0.945 over 30,000, which gives 0.18209pp.
2. Desktop effect: 7.10% minus 7.00% equals +0.100pp. Standard error: the square root of 0.07 times 0.93 over 18,000, plus 0.071 times 0.929 over 18,000, which gives 0.26983pp.
3. Difference in differences: 0.500 minus 0.100 equals +0.400pp.
4. Standard error of the difference in differences: the square root of 0.18209 squared plus 0.26983 squared, which gives 0.32553pp.
5. Interaction statistic: 0.400 divided by 0.32553 equals a z of 1.2288, with a two-sided p-value of 0.219158.
6. 95 percent confidence interval of the difference in differences: from -0.238pp to +1.038pp.

The interval comfortably contains zero, and it contains negative values too: the data are compatible with the effect being larger on desktop than on mobile. The claim “the effect belongs to mobile” does not survive the very arithmetic that would support it.

Why the interaction costs so much sample

The root of the problem is the arithmetic of standard errors, and it is unforgiving.

The standard error of the difference in differences combines the uncertainty of both segments: in the example, 0.32553 against 0.18209 for the mobile effect alone, that is 1.7877 times wider. Since the required sample scales with the square of the ratio between standard error and effect, that alone multiplies the sample by 3.20.

And the interaction effect is usually smaller than the main effect. In the example, the difference in differences (0.400pp) is 0.8 of the mobile effect (0.500pp). Combining the two factors:

The sample required for the interaction, divided by the sample required for the main effect, equals the square of (1.7877 divided by 0.8), which gives 4.99.

Checking against the absolute numbers: detecting the mobile effect (baseline 5.00% rising to 5.50%, that is +10% relative) at 80 percent power requires 31,234 users per variant (that is the sample size calculator’s answer for a 5% baseline, 10% relative MDE, 95% two-sided confidence). Detecting the observed difference in differences at the same power would require roughly 156 thousand per cell, practically 5 times more. The test had 30,000 per cell on mobile: enough to read the effect, not enough to read the difference.

Gelman’s factor of 16 and when it applies

The reference number in this discussion comes from Gelman: under the standard assumptions about main effects and interactions, you need 16 times the sample size to estimate an interaction than to estimate a main effect.

The exact assumption is worth reading, because it is where the whole number lives. With predictors coded as minus one half and plus one half, the standard error of the main effect is 2 sigma over the square root of N, and the interaction term’s is 4 sigma over the square root of N, that is twice as large. If on top of that the interaction is half the main effect, the ratio between effect and standard error falls by a factor of 4, and the required sample rises by 4 squared, that is 16.

And the author himself clarifies what “half” means there, a point routinely quoted wrong: if the main effect of the treatment is 0.6 and the interaction with sex is 0.3, then the effect is 0.45 for one group and 0.75 for the other. If instead you had in mind a main effect of 0.6 that is 0.3 in one group and 0.9 in the other, then the required factor drops to 4 times, not 16.

This article’s example is in that second regime, which is why it gave 4.99 and not 16: the 0.400pp difference is large relative to the average effect. In a more typical case, where heterogeneity is subtle, the factor climbs fast.

relation between interaction and main effect	sample factor (with a 2 times larger standard error)
interaction equals the main effect	4 times
interaction is 0.8 of the main effect	6.25 times
interaction is half the main effect	16 times
interaction is a third of the main effect	36 times
interaction is a quarter of the main effect	64 times

Note that the table assumes the standard error is exactly 2 times larger, which is Gelman’s assumption. This article’s worked case came out at 4.99 rather than 6.25 because there the interaction standard error was 1.7877 times the mobile effect’s, not 2 times: the four segments are not the same size, so the split is not the four equal cells Gelman’s formula assumes. Use the table for orders of magnitude and the calculation in the previous section for your own case.

The practical warning Gelman derives from this is the most important part of his piece: the usual plan of designing the study focused on the main effect, maybe even preregistering, and then looking to see what shows up in the interactions is a major problem. And the worse case is designing the study, not finding the anticipated main effect, and then using the interactions to bail you out. The problem is not merely that this analysis is exploratory: it is that these data are far noisier than you realize, so what looks like an interesting exploratory finding could be just noise.

How to hunt for segments without invalidating the interval

If the test was not sized for interaction (it almost never is) and you still need to investigate heterogeneity, there are three legitimate paths, in increasing order of cost.

1. Pre-declare a few segments, with correction

Pick two or three slices that the test hypothesis genuinely predicts should respond differently, write which ones down in the analysis plan before running, and apply a multiplicity correction over them. Four segments read at the 5 percent level already give an 18.55 percent chance of at least one false positive, and that is what multiple metrics in one test covers in detail.

Kohavi and coauthors report that in Bing’s operation, for slice-and-dice analysis, they educate experimenters about false positives and encourage them to adjust the probability threshold, focusing on strong signals and smaller p-values, on the order of 0.0001. That is a far harsher ruler than 0.05, and it exists because the segment axis is the one that grows most without control.

2. Honest estimation: discover in one sample, estimate in another

Athey and Imbens propose a formal method for the case where you want the data to discover the subgroups, without that invalidating the inference. The central idea is honest estimation: one sample is used to construct the partition and another, independent one, to estimate the treatment effects within each part.

They frame the method as a complement to pre-analysis plans, where the researcher must commit in advance to the subgroups that will be considered: their approach lets researchers have the data discover relevant subgroups without falling prey to the multiple hypothesis testing concerns that would invalidate p-values.

The empirical result is what matters here. In a simulation study they show that adaptive methods (which use the same data for model selection and for estimation) have confidence interval coverage rates substantially below nominal, while the honest methods attain nominal coverage in every design tested. They are explicit about the price: honest estimation sacrifices some goodness of fit (the half of the sample used to discover does not enter the estimate) in exchange for valid confidence intervals.

Translated to an A/B test: if you have traffic to spare, set aside half the users to explore slices and the other half to estimate the effect in the slices the first half found. It is the practical, cheap version of the same principle.

3. Run a test designed for the interaction

The most expensive and most definitive alternative: run it again, sized for the difference you want to detect. That means multiplying the sample by the factor in the table above, or restricting the experiment to the segment of interest, which cuts the cost but answers a smaller question (“does it work on mobile”, not “does it work more on mobile”).

Heterogeneous treatment effects are not personalization

Conflating the two is common and expensive. Heterogeneity is a statistical claim about the experiment: the effect varied across subgroups, with declared uncertainty. Personalization is a product decision: serving different experiences to different groups.

Going from one to the other requires three things to be true at once, and the first is almost never checked:

1. The heterogeneity is real, meaning the interaction test supports the claim, not just a comparison of significance labels.
2. The segment is actionable at decision time, meaning you know which group the user belongs to before serving the page, not only afterward.
3. The gain from personalizing outweighs the permanent cost of maintaining two product paths, two sets of future tests and two bug surfaces.

The full treatment of that trade-off is in personalization versus A/B testing. And the pathological case where segment aggregation flips the sign of the result is in Simpson’s paradox in A/B testing.

Checklist before claiming the effect differs

1. Did you compute the difference in differences, or just compare two p-values? Comparing labels is not a test.
2. Does the interval of the difference exclude zero? If it includes it, the claim is “I do not know”, not “they do not differ” and not “they differ”.
3. Were the segments declared beforehand? If not, the result is exploratory and ships labelled as such.
4. How many slices were tested in total? The multiplicity arithmetic uses the number of slices attempted, not the number reported.
5. Is the segment known before the variant is served? If not, it cannot support a personalization decision.
6. Does the sample per cell match the factor in the table? If the test had power for the main effect, it almost certainly did not have power for the interaction.
7. Is a confirmation round planned? Heterogeneity found in the data is a hypothesis, and a hypothesis becomes a result in a second round.

Common mistakes

• Concluding the effect differs because one segment came out significant and another did not. It is exactly the error Gelman and Stern describe, and it is the most common of all.
• Slicing until something appears. Testing device, browser, country, channel, new versus returning and spend band is more than 20 undeclared comparisons.
• Using the same sample to discover the slice and to estimate the effect within it. That is the adaptive method which, in Athey and Imbens’s simulation, produces coverage well below nominal.
• Shipping to the winning segment only, without testing the interaction. Deciding not to ship on desktop is a claim about desktop, and it needs its own evidence.
• Mistaking a small segment for a null effect. Tablet and app in the example have very wide intervals: they do not say the effect is zero, they say it is unknown.
• Reporting the best segment’s effect as if it were the test’s effect. The best segment is biased upward by selection, which is the winner’s curse applied to slices.

Make this automatic with Donnu

The expensive error here is not looking at segments, it is a segment panel that looks like an answer when it is a question. Four rows, four p-values, one green and three grey, and nobody in the room computes the one statistic that would support the decision being made.

At Donnu, segment slices are declared alongside the hypothesis, and when two segments are compared the report shows the difference in differences with its own confidence interval, not just the two effects side by side. Undeclared segments appear in a separate tab, marked exploratory, with the number of slices already tested visible on screen. If you want to redo any row by hand, the significance calculator takes the raw counts and the sample size calculator shows what each cell would require.

References

• Gelman, A. and Stern, H. The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. The American Statistician, volume 60, number 4, November 2006. Source of the central thesis that changes in statistical significance are often not themselves statistically significant, of the explicit caveat that the observation goes beyond the commonplace point that any threshold is arbitrary, of the theoretical example of two studies with estimates of 25 plus or minus 10 and 10 plus or minus 10 (the first significant at the 1 percent level, the second one standard error from zero, with an estimated difference of 15 and a standard error equal to the square root of 10 squared plus 10 squared, that is 14), and of the complementary case of a third study estimating 2.5 with a standard error of 1.0, which attains the same significance level as the first even though the difference between the two is itself significant. stat.columbia.edu.
• Gelman, A. You need 16 times the sample size to estimate an interaction than to estimate a main effect. Statistical Modeling, Causal Inference, and Social Science, 15 March 2018. Source of the factor of 16 under the standard assumptions, of the standard error mechanics with predictors coded as minus one half and plus one half (2 sigma over the square root of N for the main effect and 4 sigma over the square root of N for the interaction), of the clarification that “the interaction is half the main effect” means a main effect of 0.6 with an interaction of 0.3, producing effects of 0.45 and 0.75, and that under the alternative reading (0.3 in one group and 0.9 in the other) the required factor drops to 4, and of the warning against designing the study focused on the main effect and then looking in the interactions, especially when the main effect fails to appear. statmodeling.stat.columbia.edu.
• Athey, S. and Imbens, G. Recursive Partitioning for Heterogeneous Causal Effects. December 2015. Source of the honest estimation proposal (one sample to construct the partition and another to estimate treatment effects in each subpopulation), of the framing of the method as a complement to pre-analysis plans that require committing in advance to the subgroups, of the explanation that adaptive methods use the training data for model selection so spurious correlations affect the selected model, and of the simulation result in which honest methods attain nominal confidence interval coverage while adaptive ones fall substantially below, at the cost of some goodness of fit. arxiv.org/abs/1504.01132.
• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the report that slice-and-dice analysis by segment violates the single-analysis assumption, and that the operational answer was to educate experimenters about false positives and encourage them to adjust the probability threshold, focusing on strong signals and smaller p-values, on the order of 0.0001. exp-platform.com.

Read also: Multiple metrics in one A/B test · Pre-registered analysis plan · Simpson’s paradox in A/B testing · Personalization versus A/B testing · The winner’s curse · Significance calculator · Leia em português