Stratification in A/B Testing: The Ceiling Nobody Computes
Stratification in an A/B test only removes the variance BETWEEN strata. We measured the ceiling: 2.68% in a real case, against 97.8% for continuous CUPED.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Stratifying an A/B test reduces variance by exactly the share of it that sits BETWEEN strata, and by nothing more. In the revenue scenario measured in this article, with new and returning users as strata, that share is 2.68 percent: the ceiling of the method is saving 2.68 percent of your traffic. In the same simulation, CUPED using the same information in continuous form cut measured variance by 97.82 percent. The gap is not about implementation quality, it is about how much information each method consumes. This guide walks through the arithmetic that sets the ceiling, proves numerically that post-stratification is CUPED with a categorical covariate, measures the two cases where stratifying does not pay, and translates variance reduction into days of testing. It is part of our complete guide to A/B testing and complements what CUPED is and sample size for revenue and continuous metrics.
Stratification: the arithmetic that sets the ceiling
The idea is old and comes from survey sampling. You split the population into internally similar groups, compute the mean inside each group, then combine everything using each group population weight. The stratified mean has the same expected value as the plain mean, but smaller variance.
Deng, Xu, Kohavi and Walker, in the paper introducing CUPED, write the reason in a single line. The variance of the mean decomposes into two terms:
- the within-strata variance, the weighted average of each group variance;
- the between-strata variance, the weighted average of the squared distances from each group mean to the overall mean.
Stratification removes the second term and leaves the first untouched. The authors use children heights as the example: the variance of heights in general is large, but if you stratify by age you get a much smaller variance within each age group. A good stratification, they write, is one that aligns well with the underlying clusters in the data.
That is the good news and the bad news at once. Good because you can compute the gain before writing a line of code, using historical data alone. Bad because in most product metrics the between-strata term is small.
Stratify at assignment, or after collection?
There are two possible moments to use strata, and they are not worth the same.
At assignment (stratified randomization). You build the groups up front and randomize inside each one, guaranteeing by construction that control and treatment are balanced in every bucket. This is the classic clinical trial design.
After collection (post-stratification). You randomize normally, let the test run, and only at analysis time split users by a characteristic that already existed before the test started.
The CUPED paper is direct about which one makes sense online: because data arrive over time, teams are usually unable to sample from strata formed ahead of time, but they can still use pre-experiment variables to construct strata after all the data have been collected.
We measured the difference. In the heavy-tailed revenue simulation described below, with 60,000 users per replicate and 2,000 replicates, stratified assignment returned a variance of 1.5930 against 1.5322 for simple randomization: 3.97 percent worse, that is, noise around zero. It makes sense. With tens of thousands of users per arm, plain randomization already balances the strata essentially on its own, so forcing the balance leaves nothing to gain.
| where the strata enter | stratum balance | measured variance gain | engineering cost |
|---|---|---|---|
| stratified assignment | perfect by construction | 3.97% worse than simple randomization (noise) | high: the stratum must be decided at randomization time |
| post-stratification at analysis | near perfect by accident, with large N | equals the theoretical ceiling of the decomposition | low: one more join in the pipeline |
| neither | random | none | none |
The practical reading: if you have enough traffic for a normal A/B test, you have enough traffic for simple randomization to balance your strata. Stratifying at assignment pays a complexity bill without delivering precision.
Post-stratification is CUPED with a categorical covariate
This part tends to surprise people. Appendix A of the CUPED paper shows that when the covariates are categorical, stratification and control variates produce identical results. Not similar: identical.
We reproduced the identity numerically on our simulation data. On one side, the post-stratified mean of each arm, computed as the sum of stratum means weighted by each stratum share of the combined sample. On the other, the control variate estimator using the stratum indicator variables as covariates, with each indicator coefficient equal to the difference between that stratum mean and the reference stratum mean.
arm A: post-stratified = 0.73465908140242364
arm A: control variate = 0.73465908140242353
absolute difference = 1.110e-16
arm B: post-stratified = 0.59895701571944082
arm B: control variate = 0.59895701571944071
absolute difference = 1.110e-16
post-stratified delta = -0.13570206568298282
indicator variable delta = -0.13570206568298282The 1.110e-16 gap is double precision floating point epsilon. The two calculations are the same calculation.
That changes the question worth asking. It is not “should I stratify or use CUPED?”. It is “does my covariate lose information when it becomes a category?”. If the covariate is already categorical (acquisition channel, platform, country, new against returning), stratifying and running CUPED on that variable are the same thing. If the covariate is continuous (pre-period spend, sessions in the last 30 days, account age), bucketing it throws information away, and CUPED on the continuous value dominates.
The measured ceiling: new against returning
We built a scenario deliberately favourable to stratification: revenue per user, with two segments that convert very differently.
- New users: 70 percent of the base, 2.5 percent conversion, lognormal ticket. Mean revenue per user of 1.3667 and variance of 99.23.
- Returning users: 30 percent of the base, 9 percent conversion, larger ticket. Mean revenue per user of 8.1117 and variance of 923.56.
The overall mean is 3.3902. The two segments are six times apart in mean revenue, which looks like an enormous separation. Running the decomposition analytically:
| term | value | share of total variance |
|---|---|---|
| total variance | 356.08 | 100.00% |
| within strata | 346.53 | 97.32% |
| between strata | 9.55 | 2.68% |
Then we measured it for real with a paired A/A test: 8,000 replicates, 80,000 users per replicate, a 50/50 split, both estimators reading exactly the same data.
| estimator | true variance of the delta | measured false positive rate | variance reduction |
|---|---|---|---|
| difference in means | 1.84925e-2 | 5.64% | reference |
| post-stratified (new against returning) | 1.79246e-2 | 5.42% | 3.07% |
The measured 3.07 percent matches the analytical ceiling of 2.68 percent within Monte Carlo noise. Recording what else showed up: both false positive rates landed above the nominal 5 percent (the Monte Carlo standard error at 5 percent with 8,000 replicates is 0.24 percentage point). That is the skew of the lognormal ticket stretching the tail of the z test, not a defect of the stratified estimator, and it applies equally to both.
When post-stratification makes things worse
The second scenario is revenue with a genuinely heavy tail, the kind that shows up on platforms with in-app purchases: 3 percent of spenders, spend per spender following a power law with exponent 1.15, plus a pre-period spend correlated with the test-period spend. The measured correlation between pre and post was 0.7643.
We stratified by pre-period spend percentile, with cuts at 97, 99.5 and 99.9 percent:
| stratum | weight | users | mean revenue | internal variance |
|---|---|---|---|---|
| base | 97.002% | 58,201 | 0.002 | 0.0 |
| mid | 2.500% | 1,500 | 8.604 | 54.3 |
| high | 0.400% | 240 | 37.503 | 815.7 |
| tail | 0.098% | 59 | 478.067 | 2,178,816.5 |
With 2,000 A/A replicates and 60,000 users per replicate, the result was this:
| estimator | variance of the delta | reduction |
|---|---|---|
| difference in means | 1.5322 | reference |
| post-stratification (4 strata) | 1.6352 | 6.72% WORSE |
| CUPED on the continuous pre-period | 0.033340 | 97.82% |
| post-stratification with CUPED inside the stratum | 0.031682 | 97.93% |
Post-stratification alone increased variance. The cause is the tail stratum: 59 users with an internal variance of 2.18 million. The mean of that handful is a terrible estimate, and the stratified estimator depends on it with a weight of its own, while the plain mean dilutes those 59 users across 60,000.
The ShareChat team, which published its use of post-stratification across more than 40 production experiments with over 1 million users each, arrives at a neighbouring recommendation by a different route: the method should not be used when tail user behaviour is the primary decision objective, because the post-stratified estimator down-weights precisely the population under study. Their positive result deserves reproducing too, because it is the other side of the same coin: post-stratification combined with CUPED inside the stratum delivered more than 99 percent variance reduction on real data, against 47.62 percent for CUPED alone, with the p-value computed on the adjusted metric. Our simulation shows the same stacking, with the two numbers essentially tied at the top (97.82 against 97.93 percent).
From variance reduction to days of testing
The translation is direct: sample size is proportional to variance. An X percent cut in variance removes X percent of the required sample, or equivalently shrinks the minimum detectable effect by the square root of 1 minus X.
Taking as a reference a test with a 5 percent baseline, a 10 percent relative minimum detectable effect, alpha of 5 percent and 80 percent power, which gives 31,234 users per variant and 11 days at 40,000 visitors per week:
| variance reduction | N per variant | days at 40,000/week | relative MDE reachable with the original N |
|---|---|---|---|
| none | 31,234 | 11 | 9.77% |
| 3% (measured post-stratification) | 30,297 | 11 | 9.62% |
| 10% | 28,111 | 10 | 9.27% |
| 30% | 21,864 | 8 | 8.18% |
| 50% (CUPED at Bing) | 15,617 | 6 | 6.91% |
| 98% (continuous CUPED measured here) | 625 | 1 | 1.38% |
The 3 percent row did not change a single day of testing. The 50 percent row cut the test in half. That is the difference between stratifying and using the whole covariate.
Run your own case in the calculator below. To see the effect of variance reduction, enter your usual baseline and MDE, note the N, then multiply the result by 1 minus the reduction you measured on your own history:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
How to pick strata in practice
A routine you can copy and run over your own history before touching any pipeline:
- Pick the metric that matters and pull at least 30 days of history for it, at the level of the randomization unit (usually the user).
- List the candidate covariates that exist BEFORE the test starts: pre-period spend, session count, acquisition channel, platform, account age, country. The hard rule is that the variable cannot be affected by the treatment, and the safe way to guarantee that is to freeze it in the period preceding the test.
- Compute the decomposition for each categorical candidate: total variance, weighted within-strata variance, and the difference between the two. The between-strata share is your ceiling.
- Discard anything with a low single-digit ceiling. If the between-strata share is 2 or 3 percent, post-stratification will not change a decision you make.
- For continuous candidates, compute the correlation with the final metric and compare its square against the ceiling of the bucketed version. In our simulation the continuous version won by two orders of magnitude.
- Check the size of the smallest stratum. If any stratum holds fewer than a few hundred users and has high variance, either merge it with its neighbour or the method will inflate your variance.
- Run an A/A with the new estimator before trusting it. Our measurement showed both estimators running above the nominal false positive rate because of metric skew, and that only surfaces by testing.
- Freeze the cuts before looking at outcomes. Stratum thresholds must come from the period preceding the test, exactly as ShareChat describes: computed from the pre-period population and frozen before any experiment outcome data is observed.
Three mistakes that show up every time
Stratifying on something the treatment affects. If the stratum is “user who visited the new page”, the treatment changed who lands in each group and the estimator stops being unbiased. The defense is rule 2 above: pre-period frozen variables only. This is the same care required in triggered analysis and dilution.
Confusing explaining with reducing. A stratum can be fascinating for understanding the business (whales spend 300 times more) and useless for reducing variance. The two questions are independent. If the goal is understanding where the effect concentrates, the right instrument is heterogeneous treatment effects, with all the multiple comparison discipline it demands.
Thinking stratification fixes the read. Stratification is variance reduction, not bias correction. It does not fix SRM, the sample ratio mismatch, it does not fix instrumentation bias and it does not fix bot traffic. A test with a broken randomization stays broken after stratification, only now with a tighter error bar around the wrong number, which is worse.
Frequently asked questions
Short answers live in the FAQ section of this page, generated from the same measurements presented here.
References
- Alex Deng, Ya Xu, Ron Kohavi and Toby Walker. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, WSDM 2013. Carries the decomposition of variance into within and between strata, the discussion of why online stratification has to happen at analysis rather than at assignment, and Appendix A with the proof that stratification and control variates coincide for categorical covariates. It is also the source of the roughly 50 percent variance reduction result at Bing.
- Neeti Pokharna, Olivier Jeunen, Yatharth Saraf and Aleksei Ustimenko. Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification, SIGIR 2026. Reports deployment across more than 40 production experiments at ShareChat and Moj, with the comparison between CUPED alone (47.62 percent) and post-stratification with CUPED (over 99 percent), the requirement to freeze the cuts before observing outcomes, and the record that k-means grouping over behavioural features increased variance while percentile cuts on revenue dominated.
- Winston Lin. Agnostic Notes on Regression Adjustments to Experimental Data: Reexamining Freedman’s Critique, The Annals of Applied Statistics, 2013. Shows that ordinary least squares adjustment cannot hurt asymptotic precision when a full set of treatment by covariate interactions is included, and that the Huber and White sandwich standard error estimator is consistent or asymptotically conservative. It is the theoretical basis for why adjusting is not cheating.
- Yu Zhang, Bokui Wan, Yongli Qin, Jinyong Ma and Yifan Guo. Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED, ByteDance, 2026. Compares post-CUPED adjustment specifications and shows that in multi-arm designs and two-stage sampling, naive reliance on standard variance estimators leads to severely misleading inferences.
Read next
Frequently asked questions
- What is stratification in an A/B test?
- It means splitting users into internally similar groups (the strata) using information that predates the test, computing the treatment effect inside each group, and then combining the group results using each group population weight. The goal is not to change the estimated effect, it is to reduce the variance of that estimate. The calculation gets more precise because the variation that exists BETWEEN groups stops counting as noise. The variation left inside each group still counts.
- What is the maximum gain stratification can deliver?
- Exactly the share of variance that sits between the strata. Deng, Xu, Kohavi and Walker write that decomposition in closed form in the CUPED paper: total variance is the sum of the within-strata variance and the between-strata variance, and stratification removes only the second term. In the scenario measured in this article, with new and returning users as strata, that term is 2.68 percent of total variance. So the ceiling of the method in that case is saving 2.68 percent of your traffic, not half of it.
- Is post-stratification the same thing as CUPED?
- When the covariate is categorical, yes, and that is proved in Appendix A of the original CUPED paper: stratification and control variates produce identical results. We reproduced the identity numerically and the two calculations matched to the 16th decimal place, with a difference of 1.110e-16, which is floating point rounding error. The practical difference shows up when the covariate is continuous: CUPED then uses the whole value while post-stratification uses only the bucket the user fell into.
- Should I stratify at assignment or after collecting the data?
- After. The CUPED paper itself records that in the online world data arrives over time, so you are usually unable to sample from strata formed ahead of time, but you can still use pre-experiment variables to build strata after all the data are collected. In this article simulation, stratified assignment did not reduce variance relative to simple randomization: the measurement came out 3.97 percent worse, meaning noise around zero, at a real engineering cost.
- Can post-stratification make things worse?
- It can, and this shows up both in our simulation and in production. When a stratum is very small and very volatile, its mean becomes a poor estimate and the stratified estimator gets noisier than the plain average. In our heavy-tailed revenue simulation, a top stratum with 59 users and enormous variance made post-stratification increase variance by 6.72 percent. The ShareChat team records an effect of the same nature when comparing stratum designs: grouping users by k-means over behavioural features increased variance, while cutting by pre-period revenue percentile was what worked. The lesson is the same: a badly designed stratum makes the estimator worse.
- So when is stratification actually worth it?
- When the strata genuinely separate the metric, that is, when the group means are far apart and the variation inside each group is small. Proportion metrics almost never satisfy that, because the variance of a proportion is pinned to p times 1 minus p. Heavy-tailed revenue metrics can satisfy it, but then the tail stratum needs enough users in it. Before investing in the method, measure the decomposition on your own history: if the between-strata share is a low single-digit percentage, the gain will be a low single-digit percentage.