Missing Data in A/B Tests: MCAR, MAR and Imputation

The row that never reached your database is not neutral. In the worked example below, a variant that lost more tracking on older devices made the complete-case read return plus 14.71 percent when the real effect was plus 10.48 percent, a 40 percent overstatement; imputing the arm mean kept the same error and narrowed the interval on top of it; and standardizing on device type recovered exactly plus 0.6500 points and plus 10.48 percent. Missing data is not a cleanup detail, it is a statistical decision with predictable bias. This guide covers the classification that organizes the problem, the arithmetic that separates what can be fixed from what cannot, and how to fence off the rest. It is part of our complete guide to A/B testing and complements tracking loss from consent and adblockers, which covers the technical causes.

Missing data: loss is almost never symmetric

Every A/B test loses data. The causes are familiar: ad blockers, consent refusals, tabs closed before the event fires, slow devices, scripts failing in old browsers, users switching devices midway.

The problem is not the loss. It is loss that is different between arms, and it usually is, for a structural reason: the new variant almost always changes something about page weight, event ordering or firing timing. One extra component is enough for old devices to fail a little more often, and just like that arm B is being measured on a newer, faster sample that converts better.

Notice what this does to the comparison. The two arms are still identical among those randomized. They stop being identical among those measured. Randomization protects the first set and does not protect the second, and it is the second one that goes into the arithmetic.

Rubin’s three questions

The classification that organizes the subject comes from Donald Rubin in 1976, and van Buuren puts it in three plain sentences. If the probability of being missing is the same for all cases, the data are missing completely at random. If it is the same only within groups defined by the observed data, the data are missing at random. If neither holds, the data are missing not at random.

mechanism	loss depends on	typical example in a test	fixable?
completely at random	nothing	a brief collection outage hitting both arms equally	yes, complete cases suffice
at random	variables you measured	old devices drop more events, and you have device type	yes, if the variable was measured pre-treatment
not at random	the missing value itself	users who bounce without buying close the tab before the event fires	not with data, only with a declared assumption

Two consequences that matter more than the definitions:

• you cannot tell “at random” from “not at random” by looking at the data. It is a claim about the relationship between variables whose values you do not have. The National Research Council report on missing data is blunt about it: the choice of statistical method requires unverifiable assumptions concerning the mechanism that causes the missing data, and so always involves some degree of subjectivity.
• “at random” is a property of your database, not of the world. The same loss is not at random if you never stored device type and at random if you did. Better instrumentation changes the category of the problem.

Worked example: the variant that lost more tracking on old devices

A test with 20,000 users per arm, a base split 80 percent modern devices and 20 percent old ones by randomization, identical across arms. Variant B loads one extra component that fails more often on old hardware.

Here is the complete data, which in real life nobody ever sees:

arm	device	users	conversions	rate
A	modern	16,000	1,120	7.000%
A	old	4,000	120	3.000%
A	total	20,000	1,240	6.200%
B	modern	16,000	1,240	7.750%
B	old	4,000	130	3.250%
B	total	20,000	1,370	6.850%

The truth, therefore, is plus 0.6500 points and plus 10.48 percent, with a p-value of 0.008490.

Now the loss. Inside each cell, whoever disappears converts at the same rate as whoever stays, meaning loss is at random given device and arm. What differs is how many disappear:

arm	device	loss	users measured	conversions measured	measured rate
A	modern	2.5%	15,600	1,092	7.000%
A	old	30.0%	2,800	84	3.000%
A	total	8.0%	18,400	1,176	6.391%
B	modern	2.5%	15,600	1,209	7.750%
B	old	60.0%	1,600	52	3.250%
B	total	14.0%	17,200	1,261	7.331%

Notice that inside every device row the measured rate is identical to the true rate. Not a single cell is biased. And the total still is.

Three reads, and what each one returns

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

read	control	variant	relative effect	p-value
truth (complete data)	1,240 of 20,000 = 6.2000%	1,370 of 20,000 = 6.8500%	plus 10.48%	0.008490
complete cases	1,176 of 18,400 = 6.3913%	1,261 of 17,200 = 7.3314%	plus 14.71%	0.000448
arm mean imputation	1,278 of 20,000 = 6.3900%	1,466 of 20,000 = 7.3300%	plus 14.71%	0.000200

Paste all three rows into the calculator above. The first two match token for token; the third differs only by rounding imputed conversions to whole users.

Three reads of the same experiment, and the only one that answers the question is the first, which in real life you do not have. The other two err in the same direction and for the same reason:

Why complete cases overstated. Arm B lost 60 percent of its old devices and arm A lost 30 percent. Old devices convert at half the rate of modern ones. So B’s measured sample ended up richer in modern devices than A’s: 90.698 percent against 84.783 percent. The comparison started mixing the effect of the variant with the composition difference between the two measured samples. This is not a sample size problem, it is a mixture problem, and more traffic only makes the wrong number more precise.

Why mean imputation is worse. Filling the holes with the arm’s own observed mean reproduces the observed rate exactly, so the point estimate does not move. What moves is the denominator: you are back to 20,000 per arm and the interval shrinks. The p-value falls from 0.000448 to 0.000200 with no new information entering. That is invented precision on top of a biased number, the worst possible pairing. The National Research Council report is explicit in Recommendation 10: single imputation methods should not be used as the primary approach to the treatment of missing data unless the assumptions that underlie them are scientifically justified.

Standardization recovers the number

If loss depends on device, and device was measured before treatment, the fix is direct: compute each arm’s rate within each device type and recombine using the true design weights, which randomization guarantees are equal across arms.

arm	measured rate, modern	measured rate, old	recombined at 80% and 20%
A	7.000%	3.000%	0.8 times 7.000 plus 0.2 times 3.000 = 6.2000%
B	7.750%	3.250%	0.8 times 7.750 plus 0.2 times 3.250 = 6.8500%

Difference: plus 0.6500 points. Relative effect: plus 10.48 percent. Exactly the truth, recovered without ever seeing the missing data.

The standard error of that estimate, summing the weighted variances of each stratum, is 0.26088 points, giving z equal to 2.4916, a p-value of 0.012717, and a 95 percent confidence interval between plus 0.1387 and plus 1.1613 points.

Three conditions for that fix to hold:

1. the variable has to be pre-treatment. Conditioning on something the treatment affected reintroduces bias, the same error as segmenting on downstream behavior that we cover in heterogeneous treatment effects.
2. the weights have to come from the design, not from the measured sample. Using each arm’s observed composition would return the biased number, because that composition is exactly what got contaminated.
3. every stratum needs users in both arms. A stratum that lost almost everything on one side cannot support a comparison, and the way out is to merge strata, not to pretend there is data.

How much loss is already a problem

There is no universal threshold, but there is a back-of-envelope calculation that answers in thirty seconds whether to worry. Complete-case bias is approximately the composition difference between the two measured samples multiplied by the distance between the rates of the groups making up that mixture.

In the example, B’s measured sample had 15,600 of 17,200 users on modern devices, or 90.698 percent, against 15,600 of 18,400, or 84.783 percent, in A. The composition difference is 5.915 points. The distance between the modern rate and the old rate is around 4.5 points. The product is roughly 0.266 points of expected bias, and the observed bias was 0.2901 points, the gap between the 0.9401 from complete cases and the true 0.6500. The approximation errs little and is good enough to decide on.

loss difference between arms	resulting composition difference	expected bias, with groups 4 points apart
1 point	under 0.5 points	under 0.02 points, irrelevant
3 points	around 1.5 points	around 0.06 points
6 points	around 3 points	around 0.12 points
12 points	around 6 points	around 0.24 points

Compare the last column against the effect you expected to detect. A 0.30 point effect living alongside a 0.24 point bias is not a result, it is noise with a sign. And note the second term of the product: when the groups the loss separates convert almost identically, even enormous loss does no harm. It is asymmetric loss combined with unequal groups that ruins the read.

Multiple imputation: why more than one fill

When there are many covariates, or when what is missing is a continuous metric rather than a conversion flag, manual standardization becomes impractical and the standard tool is multiple imputation. The idea is the one van Buuren summarizes: create more than one complete dataset, analyze each with your usual method, and pool the results with Rubin’s rules.

The point is in the pooling. The final variance adds the usual sampling variance, called within-imputation variance, to the extra variance caused by the missing data, the between-imputation variance, which is the disagreement across fills. That second term is exactly what single imputation throws away, and it is why single imputation produces standard errors that are too small.

method	point estimate	acknowledges imputation uncertainty?	when to use
complete cases	biased unless loss is completely at random	not applicable	low and clearly symmetric loss
mean imputation	same as complete cases	no	never as the primary analysis
stratum standardization	correct under loss at random in the variable used	yes, in the pooled standard error	few covariates, measured beforehand
multiple imputation	correct under loss at random in the model used	yes, via between-imputation variance	many covariates or a continuous outcome

What cannot be fixed: loss that depends on the outcome

If the people who disappear are precisely the ones who were not going to convert, no covariate saves you, because the information you need is in the missing value. What remains is to state the assumption and test how much weight it can bear.

The usual format is to assume the missing convert at some fraction of the observed rate in the same group, and sweep that fraction. Applying the discount only to arm B, the one that lost more, and starting from the standardized read:

rate among B’s missing, as a fraction of the observed	estimated B rate	relative effect
100% (loss at random within stratum)	6.8500%	plus 10.48%
80%	6.7410%	plus 8.73%
60%	6.6320%	plus 6.97%
40%	6.5230%	plus 5.21%
20%	6.4140%	plus 3.45%
0% (no missing user converted)	6.3050%	plus 1.69%

The output of that sweep is the most useful thing you can carry into a meeting: the direction of the conclusion never flips, not even in the absurd scenario where none of B’s 2,800 missing users would have converted. What flips is the size, which collapses from plus 10.48 to plus 1.69 percent.

Compare that against the same sweep run on the naive read, without strata: there the conclusion cancels out when the missing convert at 8.41 percent of the observed rate, that is, 0.617 percent. Standardizing did not just correct the point estimate, it made the conclusion far more robust.

This is why Recommendation 15 of the National Research Council report asks that sensitivity analyses be part of the primary reporting of findings, and that examining sensitivity to assumptions about the missing data mechanism be a mandatory component of reporting.

A six step routine

1. Measure the loss before looking at the metric. Two lines: fraction of randomized users with no recorded outcome, by arm. If the gap between arms exceeds one point, treat loss as the main issue of that test.
2. Run a sample ratio test on the measured population. An uneven split that appears only after the outcome filter is the signature of differential loss, not of a broken randomizer.
3. Find the variable that explains the loss. Device, browser, country, channel, app version. It must be pre-treatment.
4. Standardize and compare against complete cases. If the two reads agree, loss was not distorting anything. If they disagree, the standardized one is the valid one, and the gap between them is the size of the bias you just removed.
5. Sweep the assumption that remains. The fraction table above takes ten minutes and answers the only question leadership will ask.
6. Fix the cause before the next test. An instrumentation fix is worth more than any correction method, because it moves the problem into a different category.

Common mistakes

• Reporting loss only as an aggregate. Eight percent overall loss says nothing; 8 against 14 percent between arms says everything.
• Filtering to complete records and calling it cleanup. That filter is an analysis decision with predictable bias, and it deserves a line in the report.
• Filling with zero. That is single imputation under the strongest assumption available, that everyone who vanished failed to convert, without declaring it.
• Using observed composition as weights. Returns the biased number wearing the costume of a correction.
• Conditioning on a post-treatment variable. Reintroduces the bias you tried to remove, through another door.
• Treating sensitivity analysis as an appendix. It is the result, just as much as the point estimate.

Do this automatically with Donnu

Fixing missing data depends entirely on something decided before the test: storing the assignment separately from the outcome. When the table only holds people who converted, there is no way to know how many vanished, and complete-case analysis becomes the only option available without anyone noticing they chose it.

Donnu records the assignment at the moment it happens, with the user’s pre-treatment state stamped alongside it, which puts the loss rate per arm on the same screen as the result and turns stratum standardization into a query instead of a project. The significance calculator closes out the reads in the table above with the same numbers you would paste by hand.

References

• van Buuren, S. Flexible Imputation of Missing Data, second edition, chapter 1. Source for the three definitions attributed to Rubin (1976): if the probability of being missing is the same for all cases, the data are missing completely at random; if it is the same only within groups defined by the observed data, the data are missing at random; and if neither holds, the data are missing not at random. Also the source for the note that missing completely at random, while convenient, is often unrealistic for the data at hand, and that modern missing data methods generally start from the missing at random assumption. stefvanbuuren.name.
• van Buuren, S. Flexible Imputation of Missing Data, multiple imputation in a nutshell. Source for the three step description (create more than one complete dataset, analyze each with standard software, pool the results with Rubin’s rules) and for the decomposition of the final variance into within-imputation variance, the conventional sampling variance, and between-imputation variance, the extra uncertainty caused by the missing data that single imputation discards. stefvanbuuren.name.
• National Research Council. The Prevention and Treatment of Missing Data in Clinical Trials, conclusions and recommendations chapter, 2010. Source for Recommendation 10, that single imputation methods should not be used as the primary approach to the treatment of missing data unless the assumptions underlying them are scientifically justified; for Recommendation 15, that sensitivity analyses should be part of the primary reporting of findings and that examining sensitivity to assumptions about the missing data mechanism should be a mandatory component of reporting; and for the statement that the choice of statistical method requires unverifiable assumptions concerning the mechanism causing the missing data. ncbi.nlm.nih.gov.

Read also: Tracking loss from consent and adblockers · Differential attrition · Stratification in A/B testing · Sample ratio mismatch · Significance calculator · Leia em português