What Is CUPED? Reducing A/B Test Variance
CUPED is a variance reduction technique that uses pre-experiment data to shorten A/B test duration without losing statistical rigor.
📚 This article is part of the guide A/B Testing Statistical Significance Guide.
CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance reduction technique that uses a metric from the period BEFORE a test, measured on the same user, to explain away part of the noise in the metric you are testing, which lowers residual variance and makes the test more sensitive without adding traffic. It came out of a 2013 Microsoft paper and is now the standard variance-reduction method across experimentation platforms at Netflix, Booking.com, and much of the A/B testing industry. If you have already read our guide to A/B testing statistical significance, CUPED answers the most common problem that guide leaves out: what to do when your real conversion lift is genuine, but the test stays inconclusive because your traffic is not large enough to prove it in time.
The problem: average-traffic sites pay a premium for statistical power
Every honest A/B test needs enough sample to separate a real effect from chance. That sample size depends, among other things, on the variance of the metric you are measuring: the more that metric swings from visitor to visitor for reasons that have nothing to do with your change, the more people you need to see the signal through the noise. Revenue-per-user, session counts, or time spent in a product tend to have much higher variance than a simple binary conversion rate (bought or did not buy), because a handful of users who spend far above average drag the whole variance upward.
In practice, that means months of testing to detect an effect that genuinely exists. A team with average traffic testing a change that lifts revenue per user by 5% might need a sample so large the test never finishes before the roadmap moves on. The obvious fix, wait longer, costs market timing; the risky fix, lower your rigor, costs trustworthiness. CUPED attacks the problem from a third angle: instead of touching rigor or waiting for more traffic, it reduces the noise the sample has to overcome.
The core idea: a covariate that already existed before the test
The intuition behind CUPED is easy to state: if you already know, before the test even starts, that a user historically spends a lot or a little, that prior information explains a slice of the value they will generate during the test, regardless of which variation they saw. A customer who spent $500 a month before the experiment tends to land in a similar range afterward, and that tendency has nothing to do with A or B. If you can “subtract out” that predictable slice of the metric, what is left over (the residual) has less variance, and it is inside that cleaner residual that the difference between A and B becomes easier to see.
Formally, that pre-experiment covariate is called X, and the metric the test measures is called Y. The adjusted metric, called Y-CUPED, is:
Where θ (theta) is the coefficient that captures how much of X explains Y, computed as:
Term by term: Y is the metric’s value during the test (revenue, sessions, whatever you are measuring); X is the value of that same metric (or a correlated one) for the same user, before the test started; E[X] is the average of X across the whole population; and θ is exactly the coefficient of a simple linear regression of Y on X, the value that minimizes the residual’s variance. Notice what that subtraction does: when X sits above average (the user historically spends more), the term θ·(X − E[X]) is positive and gets subtracted from Y, removing the slice of Y’s value that was already expected from the user’s history, not from the variation they saw.
Where the variance reduction comes from: the role of correlation
The most useful part of CUPED, in practice, is that its variance reduction has a closed-form formula that is easy to apply. Choosing θ optimally (the regression coefficient above), the variance of the adjusted metric is:
Where ρ (rho) is the Pearson correlation between X and Y, a number between −1 and 1 that measures how closely the two move together. This formula says the variance left after the adjustment is a fraction (1 − ρ²) of the original variance. Flip it around: the percentage reduction in variance equals ρ², the correlation squared. A correlation of 0.5 between the pre-experiment covariate and the in-test metric cuts variance by 0.5² = 25%; a correlation of 0.7 cuts it by 0.7² = 49%, nearly in half.
That relationship explains why CUPED is so sensitive to the choice of covariate: low correlations (below 0.3, say) yield a small reduction, under 10%, barely noticeable in practice; moderate to high correlations (0.5 to 0.7), common in revenue or engagement metrics for returning users, yield reductions of 25% to nearly 50%, enough to shorten a months-long test into weeks.
How much traffic does CUPED save? Start from your own sample size
Before running the worked example, it helps to see the baseline you are shrinking. Plug in your own baseline conversion rate, minimum detectable effect, weekly traffic, and confidence and power targets below to get the sample size and duration a standard test would need with no CUPED applied at all. That number is exactly what a correlated pre-experiment covariate then multiplies by (1 − ρ²):
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
A worked example, start to finish
To keep this concrete, we reuse the same scenario from our multi-armed bandits vs A/B testing guide: a page with 10,000 visitors per week, a 5% baseline conversion rate, and a 10% relative minimum detectable effect, at 95% confidence and 80% power. This blog’s sample size engine returns, with no CUPED, 31,234 visitors per variant (62,468 total), running for 44 days at that weekly traffic. That is the starting point.
Now suppose the test’s primary metric is revenue per user, and you have, for every visitor, the revenue they generated in the two weeks before the test started. A correlation of 0.5 between that pre-experiment revenue and in-test revenue is a plausible number for a product with a solid share of returning users (Kameleoon, after testing CUPED internally across more than 200 of its own experiments, found that having a meaningful share of returning visitors is one of the conditions that makes the technique work best). Running the numbers:
- Variance reduction: ρ² = 0.5² = 0.25, meaning 25% less variance in the adjusted metric.
- Remaining variance fraction: 1 − 0.25 = 0.75 (75% of the original variance).
- Adjusted sample per variant: 31,234 × 0.75 ≈ 23,426 visitors (a drop of roughly 7,808 per variant).
- Adjusted total sample: 23,426 × 2 ≈ 46,852 visitors, versus 62,468 without CUPED.
- Adjusted duration, at the same 10,000 weekly visitors: 46,852 ÷ 10,000 ≈ 4.69 weeks ≈ 33 days, versus 44 days without CUPED.
- Savings: 44 − 33 = 11 days, or roughly 25% less test time for the same confidence and power.
This is a back-of-envelope approximation: it assumes the rest of the design (significance, power, minimum effect) stays fixed and that the required sample size scales approximately linearly with the metric’s variance, a standard simplification used by tools like GrowthBook and Kameleoon to communicate CUPED’s payoff. The exact gain your own tool reports may vary a bit from this napkin math, but the direction and order of magnitude hold.
The table below extends the same math to other correlations, always starting from the same 31,234-per-variant, 44-day baseline:
| Correlation (ρ) | Variance reduction (ρ²) | Adjusted sample per variant | Adjusted duration | Days saved |
|---|---|---|---|---|
| 0.3 | 9% | ~28,423 | ~40 days | ~4 days |
| 0.4 | 16% | ~26,237 | ~37 days | ~7 days |
| 0.5 | 25% | ~23,426 | ~33 days | ~11 days |
| 0.6 | 36% | ~19,990 | ~28 days | ~16 days |
| 0.7 | 49% | ~15,930 | ~23 days | ~21 days |
Declare significance on the already-adjusted metric
CUPED does not replace the significance test, it only swaps out the metric that feeds it: instead of comparing A’s raw Y against B’s raw Y, you compare each side’s Y-CUPED, which has the same mean but less noise. After applying the subtraction per visitor (or using a tool that already does this natively), paste the adjusted conversions and visitors into the calculator below to see the significance gain in practice:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
If your metric is a rate (binary conversion), treat the numbers you paste here as the result after any covariate adjustment has already been applied behind the scenes by your experimentation tool; for continuous metrics (revenue per user, sessions), the same significance logic applies by swapping the proportion test for a t-test or equivalent over the adjusted mean, but the “less variance, same effect, easier to detect” logic is identical.
When CUPED helps a lot, and when it barely helps at all
CUPED is not universal. Its payoff depends entirely on the existence of a correlated pre-experiment covariate, and that cleanly separates the scenarios where it is worth applying from the ones where it is not:
| Scenario | Does CUPED help? | Why |
|---|---|---|
| Metric with the same user’s history (revenue per user, sessions, time in product) | Yes, substantially | Pre-experiment behavior tends to be a good predictor of in-test behavior, giving a high correlation |
| Product with a solid base of returning users | Yes | More people have a usable history as a covariate, more people benefit from the adjustment |
| Simple binary metric (clicks, conversion) on a product with recurring visits | Moderate help | Correlation tends to be lower than for continuous metrics, but some gain still exists |
| Acquisition landing page, mostly new visitors | Little to none | A new visitor has no pre-experiment value; with no covariate, there is nothing to subtract |
| Brand-new product launch, no historical data for the metric | None | There is no pre-experiment window to collect X from |
| Very short test, not enough time to compute a stable covariate | Reduced gain | The covariate needs to be calculated over a reasonably stable pre-experiment window to be useful |
The practical rule of thumb: ask whether you can answer “what did this user do before the test started, on the same metric or a related one” for most of the people who will enter the experiment. If the answer is yes for most of the audience, CUPED is usually worth the implementation effort. If the answer is no, because the audience is mostly new, the gain approaches zero and the technique does not pay for itself.
CUPED is not peeking, and it is not a bandit
It is worth separating CUPED from two other ideas on this blog it is easy to confuse it with. The peeking problem is about when you decide to stop a test: checking the result repeatedly and stopping too early inflates the false positive rate. CUPED has nothing to do with that, it changes which metric feeds the significance calculation, not when you look at it. You can, and should, keep respecting the planned sample size and duration even while using CUPED, the planned sample just ends up smaller.
Multi-armed bandits, on the other hand, solve a completely different problem: instead of reducing the variance of a fixed-horizon test, a bandit continuously reallocates traffic toward the variation that looks best, trading formal statistical significance for lower opportunity cost during the test. CUPED still produces a normal p-value and confidence interval, just on a lower-noise metric; a bandit gives up the p-value in exchange for something else entirely. The two techniques do not compete, and in fact combine well: nothing stops you from applying CUPED to the metric feeding a Bayesian bandit, reducing the variance the allocation algorithm itself sees.
Automate This in Donnu
Reducing a test’s variance with CUPED demands three things that take manual work: identifying a correlated pre-experiment covariate, computing the optimal θ by regression, and applying the adjustment before running any significance test, without disturbing the frozen math that underpins the rest of the experiment. Donnu already runs on a native Bayesian engine, the same statistical foundation that supports variance adjustments like CUPED; the honest significance read you saw in our main guide is the base that a technique like this builds on top of, with no separate regression spreadsheet required for every test.
Start a free 14-day trial and read next what statistical significance is and the formula behind it, the peeking problem and why stopping early ruins a test, and multi-armed bandits vs A/B testing, for when the goal is continuous optimization instead of a single verdict. Prefer Portuguese? Read this guide in Portuguese.
References
- Deng, A., Xu, Y., Kohavi, R. & Walker, T. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. WSDM 2013 (the original paper defining CUPED). dl.acm.org.
- Kameleoon. What we learned from running 200+ experiments on CUPED. kameleoon.com/blog/cuped.
- GrowthBook. CUPED (technical documentation, θ formula and implementation). docs.growthbook.io/statistics/cuped.
- Optimizely. CUPED (optimization glossary). optimizely.com/optimization-glossary/what-is-cuped.
Frequently asked questions
- What is CUPED, in plain terms?
- CUPED (Controlled-experiment Using Pre-Experiment Data) is a technique that uses a metric collected BEFORE a test starts, from the same user, to explain away part of the natural noise in the metric you are measuring during the test. Removing that predictable slice of noise shrinks the variance of the adjusted metric, which makes the test more sensitive to a real effect on the same traffic, or lets you reach the same confidence with less traffic and less time.
- Does CUPED change who wins the test?
- No, it does not change the direction of a real effect, only the noise around it. The adjusted metric has the same expected value as the original one; what changes is the standard error, which gets smaller. A genuine effect becomes easier to separate from chance, but CUPED cannot manufacture a winner that was not there, nor hide a real loser.
- Which covariate should I use for CUPED?
- The best covariate is almost always the same metric you are testing, measured on the same user over a period before the experiment began, for example, revenue per user over the two weeks prior to the test, if the primary metric is revenue per user during the test. The stronger the correlation between that pre-experiment value and the in-experiment value, the bigger the variance reduction. A weakly correlated covariate, or none at all, does not help much.
- Does CUPED work for new visitors with no history?
- Barely, if at all. The technique depends on a pre-experiment value from the same user to serve as a covariate; a visitor who has never been seen before has no such value, so there is nothing to subtract to reduce their variance. Tests dominated by new traffic, an acquisition landing page, for instance, capture a much smaller CUPED gain than tests inside a product with a returning user base.
- Do I need special software to use CUPED, or can I compute it by hand?
- The math fits in a spreadsheet: it is a simple linear regression between the pre-experiment metric and the in-test metric, followed by a subtraction. Platforms like GrowthBook and Statsig already build CUPED natively into their significance calculations. The manual work is organizing pre-experiment data per user and recalculating significance on the adjusted metric, not the raw one.