Ratio Metrics in A/B Testing: The Naive Interval Is Wrong

A ratio metric is one whose denominator is not the thing you randomized, and the standard two-proportion test is wrong for it. Clicks per pageview, items per order and revenue per session all break the same assumption, because rows from the same user are correlated, and the fix is to estimate the variance with the delta method instead of pretending the rows are independent. This guide covers which metrics are affected, why the naive interval comes out too narrow, the closed-form correction, and a worked example where the naive reading declares a significant winner at p = 0.016 and the corrected one lands at p = 0.12. It is part of our complete guide to A/B testing and builds on statistical significance in A/B testing.

Randomization unit against analysis unit

Deng, Knoblich and Lu, writing about metric analytics at Microsoft for KDD 2018, put the problem in two definitions. The randomization unit is the granularity at which sampling or randomization is performed. The analysis unit is the aggregation level at which the metric is computed. In their words, analysis is straightforward when the randomization and analysis units agree, for example when randomizing by user while also computing average revenue per user.

The trouble starts when they disagree, which they note happens for the same experiment across different metrics: you randomize once, by user, and then compute a scorecard where some metrics are per user and others are per pageview, per session or per order. The per-user ones are fine. The others are what the paper calls cluster randomized, where the randomization unit is a cluster of analysis units, and their variance cannot be estimated the usual way.

Metric	Randomization unit	Analysis unit	Standard test valid?
Conversion rate per user	user	user	yes
Revenue per user	user	user	yes
Clicks per pageview	user	pageview	no, ratio metric
Items per order	user	order	no, ratio metric
Revenue per session	user	session	no, ratio metric
Add-to-cart rate per product view	user	product view	no, ratio metric
Click-through rate per email sent	user	email	no, ratio metric
Conversion rate per session	user	session	no, ratio metric

The rule is mechanical and worth writing on the wall: if one randomized subject can contribute more than one row to the denominator, the metric is a ratio metric. Note that the last row catches a lot of teams by surprise. Conversion rate per session looks like an ordinary proportion, but if the same user can start three sessions, it is not one.

Why the naive interval comes out too narrow

Every standard significance formula for a proportion carries an assumption that each observation is an independent draw. When the rows are clustered inside users, that assumption fails in a specific direction: rows from the same user are positively correlated, because a user who clicks a lot clicks a lot on every pageview they generate. Positive within-cluster correlation means the sample carries less information than the row count implies, so the true variance is larger than the formula says.

Bakshy and Eckles, evaluating bootstrap methods on Facebook data for KDD 2013, state the consequence plainly: procedures that neglect this dependence structure are anti-conservative, meaning they have higher Type I error rates than expected and their nominal 95 percent confidence intervals include the true value less than 95 percent of the time. Their illustration is worth keeping in mind whenever a row count looks reassuring: an advertising experiment with a million impressions may contain only a thousand distinct ads and ten thousand distinct users, so the effective sample size is nowhere near a million.

The standard way to quantify it is the design effect. Deng, Knoblich and Lu give the closed form for the simplified case of equal cluster sizes: with K clusters of m observations each and an intra-cluster correlation rho, the variance of the mean is the independent-case variance multiplied by 1 + (m - 1) times rho. The standard error is inflated by the square root of that factor:

Observations per user	rho = 0.02	rho = 0.05	rho = 0.10	rho = 0.20
1	1.00x	1.00x	1.00x	1.00x
2	1.01x	1.02x	1.05x	1.10x
3	1.02x	1.05x	1.10x	1.18x
5	1.04x	1.10x	1.18x	1.34x
10	1.09x	1.20x	1.38x	1.67x
25	1.22x	1.48x	1.84x	2.41x

Two readings of that table matter. First, at one observation per user every column is 1.00, which is why per-user metrics are safe. Second, the inflation grows with both the clustering and the correlation, so a metric like pages per visit on a content site, where users generate many rows and behave very consistently, sits in the bottom right corner where the naive standard error is off by more than a factor of two.

The paper is explicit that this closed form has, in their words, only limited practical value, because in reality the cluster sizes and the distributions inside each cluster differ. Unequal cluster sizes inflate the variance beyond what the table shows, which is why the delta method, which uses the observed variances directly rather than assuming equal clusters, is the practical tool rather than the formula above. Their own simulation gives the size of the error concretely: the naive method reported an average standard error of 0.00522 against a true standard deviation of 0.00895, roughly 1.7 times too small, while the delta method reported 0.00908, which is right.

The delta method, in five numbers per arm

The correction needs nothing exotic. For a ratio with numerator N and denominator D measured per randomized user, over n users, the variance of the ratio is approximately:

Var(N̄/D̄) ≈ (1/n) × [ var(N)/mean(D)²
                      − 2 × mean(N)/mean(D)³ × cov(N,D)
                      + mean(N)²/mean(D)⁴ × var(D) ]

That is the estimator in Algorithm 1 of Deng, Knoblich and Lu, and the whole reason it is practical at scale is the input list: mean of N, mean of D, variance of N, variance of D, covariance of N with D. Five numbers per arm, all of them sums that aggregate trivially in a distributed pipeline. No resampling, no model fitting, no per-row storage.

Bootstrapping over users is a valid alternative and makes no distributional approximation, but it is expensive; the same paper frames the closed-form approach as a way to solve big data problems with distributed algorithms at a fraction of the cost of simulation-based procedures like bootstrap. Fitting a mixed effect model is a third option and the paper warns against it for this purpose: their simulation shows the mixed effect estimator severely biased for the metric as usually defined, estimating 0.547 against a ground truth of 0.667, because the model weights every cluster equally while the metric weights bigger clusters more.

Worked example: the verdict flips

A search results page test measures clicks per results pageview, randomized by user, over two weeks. Each arm has 20,000 users. Start with the reading a standard calculator gives you, by pasting in the totals:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

The naive reading. Control: 6,000 clicks over 60,000 pageviews, a rate of 10.0000 percent. Variation: 6,252 clicks over 60,000 pageviews, 10.4200 percent. That is +0.42 percentage points, +4.20 percent relative, z = 2.40, p-value 0.0163, confidence interval +0.077 to +0.763 percentage points. Significant, ship it.

Now the per-user summary statistics that the totals were built from, which is what the delta method needs:

Per-user statistic	Control	Variation
Users (n)	20,000	20,000
Mean pageviews per user	3.00	3.00
Variance of pageviews per user	12.00	12.00
Mean clicks per user	0.3000	0.3126
Variance of clicks per user	0.79	0.82
Covariance of clicks and pageviews	1.30	1.35
Ratio (clicks per pageview)	10.0000 percent	10.4200 percent

Note the point estimates are unchanged: 0.3000 divided by 3.00 is 10.0000 percent, and 6,000 over 60,000 is the same number. The delta method does not move the estimate, only the interval around it.

Control. The bracket evaluates to 0.79 minus 2 times 0.10 times 1.30 plus 0.10 squared times 12.00, which is 0.79 - 0.26 + 0.12 = 0.65. Divide by n times mean(D) squared, so 0.65 / (20,000 x 9) = 3.6111e-6, giving a standard error of 0.1900 percentage points. The naive standard error on the same arm, the square root of 0.10 times 0.90 over 60,000, is 0.1225 percentage points. The naive figure is 1.55 times too small, which corresponds to a design effect of about 2.41. Note that no single intra-cluster correlation reproduces that number through the equal-cluster formula above, because the pageview counts here have a variance of 12.00 around a mean of 3.00: part of the inflation is the within-user correlation and part of it is the spread in cluster sizes, which is exactly why the delta method uses the observed variances instead of a table.

Variation. The bracket is 0.82 - 0.28134 + 0.13029 = 0.66896, giving a variance of 3.7164e-6 and a standard error of 0.1928 percentage points.

The difference. The standard error of the difference is the square root of the sum, 0.2707 percentage points. Against an observed difference of 0.42 percentage points that gives z = 1.55, p-value 0.1208, confidence interval -0.11 to +0.95 percentage points. The interval contains zero.

So the same 40,000 users, the same 120,000 pageviews and the same 12,252 clicks support p = 0.0163 and ship or p = 0.1208 and keep running, depending entirely on whether the analysis remembers what was randomized. And the corrected reading is not merely more conservative, it is more useful: the interval says the effect is somewhere between a small loss and a gain of about 0.95 points, which is an honest description of a test that has not finished.

What to do about it

There are three legitimate responses, and one that is not.

Move the metric to the randomization unit. Clicks per user instead of clicks per pageview makes the analysis unit match the randomization unit, and the standard test becomes valid again. This is the cheapest fix and often the right one, but it is a genuine change of question: clicks per user goes up when the same people simply visit more often, while clicks per pageview asks how compelling a single page is. Only make the swap if the new metric still answers what you were asking, a distinction covered in choosing the primary metric.

Apply the delta method. Keep the metric and correct the variance. Five summary statistics per arm, computed once, applied to every ratio metric on the scorecard.

Bootstrap over users. Valid, more expensive, and it makes no approximation about the shape of the distribution. Resample users, never rows. Resampling rows reproduces the naive answer with more machinery around it.

What does not work is applying a correction factor pulled from a table like the one above without measuring your own intra-cluster correlation, or, worse, deciding that the naive reading is close enough because the p-value was small. In the example above the naive p-value was small precisely because the variance was wrong.

Two practical notes. Ratio metrics interact with sample sizing: the design effect multiplies the users you need, not the rows, so a test sized off pageview counts will be underpowered by exactly that factor, an arithmetic laid out in minimum detectable effect. And variance reduction techniques stack with this rather than replacing it: CUPED lowers the variance you are estimating, while the delta method makes sure you are estimating it correctly in the first place.

Common mistakes with ratio metrics

Mistake	What it produces
Pasting pageview totals into a per-user significance calculator	Intervals too narrow by the design effect, and false winners at the rate that implies
Assuming any percentage is a proportion	Conversion rate per session looks like a proportion and is not one, if users can start several
Sizing the test off denominator rows	A test that looks well powered on paper and resolves nothing
Bootstrapping rows instead of users	The naive answer, arrived at expensively
Fitting a mixed effect model to get the ATE	A biased estimate whenever cluster sizes differ, per Deng, Knoblich and Lu
Swapping to a per-user metric without saying so	The reported number no longer answers the question the test was designed around
Applying a design effect from a paper instead of measuring rho	A correction that is arbitrary in size and defensible only by accident
Correcting the primary metric and leaving the rest of the scorecard naive	Every secondary metric keeps its inflated false positive rate

Automate this with Donnu

The reason ratio metrics stay broken in most setups is that the correction has to be applied per metric, every time, and nobody does that by hand across a scorecard. Donnu A/B stores the randomization unit alongside each metric definition, applies the delta-method variance whenever the analysis unit is finer than the unit that was randomized, and labels the metric on the readout so the difference between a per-user rate and a per-event rate is visible rather than implied. When an interval widens because of clustering, the readout says that is why.

Start a free 14-day trial and check whether your denominators are the thing you actually randomized.

References

• Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Source of the randomization unit against analysis unit framing, of the delta-method variance estimator and its algorithm, of the equal-cluster design effect formula and the caveat about its limited practical value, of the simulated comparison where the naive standard error came out at 0.00522 against a true 0.00895 while the delta method gave 0.00908, of the mixed effect estimate of 0.547 against a ground truth of 0.667, and of the framing of distributed closed-form estimation as costing a fraction of simulation-based procedures like bootstrap. arxiv.org/abs/1803.06336.
• Bakshy, E. and Eckles, D. Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods. KDD 2013. Source of the statement that procedures neglecting dependence are anti-conservative, producing higher Type I error rates than nominal and 95 percent intervals that cover the true value less than 95 percent of the time, and of the advertising illustration where a million impressions span only a thousand ads and ten thousand users. arxiv.org/abs/1304.7406.
• Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R. M. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 2009. On statistical power, sample size and the assumptions behind the standard tests used in online experiments. exp-platform.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on metrics, variance and the analysis of online experiments. Companion material at experimentguide.com.

Read also: A/B testing statistical significance · Choosing the primary metric · What is CUPED · Minimum detectable effect · Free confidence interval calculator · Leia em português