Sample Size for Revenue and Continuous Metrics

Sample size for a continuous metric such as revenue per user does not come from the conversion rate: it comes from the coefficient of variation of the metric itself, which is the standard deviation divided by the mean. Feeding a conversion rate into a two-proportion calculator and using the answer to size a revenue test is the quietest mistake in experiment planning, because the number that comes out looks reasonable and is wrong by a factor you never see. This guide covers the continuous formula, a worked example where the same store needs 487,847 users per variant to read conversion and 761,217 to read revenue per user, the exact decomposition that explains the gap, and what capping and CUPED do to that number. It is part of our complete A/B testing guide and continues the reasoning of how many visitors an A/B test needs.

Two families of metric, two different calculations

Every experiment metric belongs to one of two families, and the family decides which sample size formula applies.

A binary metric gives each user one of two values: converted or did not, clicked or did not, came back or did not. Its mean is a proportion, and its variance is fully determined by that proportion: for a rate p, the per-user variance is p times 1 minus p. There is no free parameter here. If you know the rate, you know the variance.

A continuous metric gives each user a number that can take many values: revenue, sessions, minutes of use, items in cart, time to first action. Its mean is an ordinary average, and its variance is an independent parameter: two stores with identical average revenue per user can have completely different dispersion, and therefore need completely different sample sizes.

That is the root of the problem. The sample size calculator everyone uses, ours included, solves the binary case: you give it the baseline rate and the minimum effect you want to detect, and it returns N per variant. It cannot solve the continuous case, because the essential piece of information is missing, and that piece is dispersion. And because it returns a number without complaining, that number ends up justifying the duration of a test whose primary metric is revenue.

The continuous formula, in three shapes

The sample size calculation for comparing two means on a continuous metric, under the normal approximation and with equal allocation between arms, is the familiar one with the variance in the right place:

Shape 1, with the standard deviation and the absolute effect. The number of observations per variant equals 2 times the square of the sum of the two critical values, multiplied by the variance and divided by the square of the absolute effect you want to detect. For 95 percent two-sided confidence and 80 percent power, the critical values are 1.959964 and 0.841621, whose sum squared is 7.848880. Times 2, the constant in the formula is 15.697759.

Shape 2, with the coefficient of variation and the relative effect. Dividing top and bottom by the mean makes the formula far more useful day to day, because almost everyone thinks in relative effects (“I want to detect 3 percent more revenue”), not absolute ones:

N per variant is 15.697759 times the squared coefficient of variation, divided by the squared relative effect. The coefficient of variation is the standard deviation divided by the mean. That is the only shape feature of the distribution that enters the calculation.

Shape 3, the binary case as a special case. For a binary metric with rate p, the per-user standard deviation is the square root of p times 1 minus p, and the mean is p, so the coefficient of variation is the square root of 1 minus p, divided by p. Plug that into shape 2 and you recover the proportion calculation exactly. These are not two formulas, they are one. What changes is where the coefficient of variation comes from.

Worked example: the same store, two metrics

A store converts at 3.5 percent with an average order value of 1,200 dollars and a standard deviation of order value of 900 dollars. Average revenue per user is 0.035 times 1,200, that is 42.00 dollars. The team wants to detect a 3 percent relative gain and gets 120,000 visitors per week into the tested flow.

Step 1: the conversion metric, in the calculator

Start with the calculation the calculator does solve. Baseline rate 3.5 percent, minimum detectable effect 3 percent relative, 95 percent confidence, 80 percent power, two-sided.

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

The answer is 487,847 users per variant. That is the right number for the question “does this test move the conversion rate”.

Step 2: the coefficient of variation of revenue per user

Now the revenue metric. A single user’s revenue is the product of two things: whether they bought (a binary variable with rate p) and how much they spent when they did (a variable with its own mean and standard deviation). That decomposition yields an exact identity for the coefficient of variation of revenue per user:

The squared coefficient of variation of revenue per user equals 1 plus the squared coefficient of variation of order value, all divided by the conversion rate, minus 1.

With the store’s numbers: the coefficient of variation of order value is 900 divided by 1,200, that is 0.75. So the squared coefficient of variation of revenue is 1 plus 0.5625, divided by 0.035, minus 1, which gives 43.6429. The square root is 6.6063.

It is worth checking by another route, because the identity is easy to get wrong. Computing the variance directly: mean revenue per user is 42.00 and the variance is 76,986.0, giving a standard deviation of 277.4635 and a coefficient of variation of 277.4635 divided by 42, that is 6.6063. It closes.

Step 3: the sample for the revenue metric

Applying shape 2 with a coefficient of variation of 6.6063 and a relative effect of 0.03:

15.697759 times 43.6429, divided by 0.0009, gives 761,217 observations per variant.

metric	coefficient of variation	N per variant	days at 120k per week
conversion rate (calculator)	5.2509	487,847	57
revenue per user	6.6063	761,217	89
ratio between the two	1.2581	1.56 times	+32 days

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Number of variations (incl. control)23456Visitors per week (total)Confidence909599%Power8090%Testtwo-sidedone-sided

-Estimated duration

Total visitors-

Projected finish-

Two-proportion normal approximation, traffic split evenly across variations. The date uses your timezone and updates live.

Paste 761,217 into the sample per variant field, 2 variants and 120,000 weekly traffic to confirm the 89 days. Then switch to 487,847 and see the 57. In this store, the difference between planning the test on the wrong metric and the right one is a full month of calendar.

Step 4: why exactly 1.56 times

The ratio is not arbitrary, and it follows from the same identity. For the conversion metric, the squared coefficient of variation is 1 minus p, divided by p, which here is 27.5714 (coefficient of variation 5.2509). For revenue it is 43.6429. The ratio between the samples is the ratio between those two squares: 43.6429 divided by 27.5714, that is 1.5829.

Notice the limiting case, which is the best way to check whether the intuition holds: if everyone who buys spent exactly the same amount (order value coefficient of variation of zero), the squared coefficient of variation of revenue would drop to 1 divided by 0.035, minus 1, that is 27.5714, exactly the conversion value. Which makes sense: with no dispersion in order value, revenue per user is conversion multiplied by a constant, and multiplying by a constant changes no coefficient of variation. The entire sample surcharge comes from the dispersion of order value, and from nothing else.

standard deviation of order value	CV of order value	CV of revenue per user	N per variant	days
0 (everyone spends the same)	0.000	5.2509	480,900	57
300	0.250	5.4182	512,046	60
600	0.500	5.8919	605,486	71
900 (the store’s case)	0.750	6.6063	761,217	89
1,500	1.250	8.4979	1,259,559	147
2,400	2.000	11.9104	2,474,266	289

A note on the 1.4 percent gap

The zero row above gives 480,900, and the calculator gave 487,847 for the same scenario. The 1.44 percent difference is not an error: the continuous formula uses the variance under the alternative hypothesis in both arms, while the calculator uses the pooled form, which estimates the standard error under the null hypothesis from the average of the two rates. The pooled form is slightly more conservative and is what the two-proportion literature adopts. The gap grows with the effect: 0.96 percent at a 2 percent effect and 4.81 percent at a 10 percent effect. For planning purposes this changes no decision, but it is worth knowing where it comes from before accusing either of being wrong.

How many users the mean needs just to behave

There is a second lower bound, independent of the power calculation, and it gets skipped often: the formula above assumes the mean of the metric is approximately normal. For a heavily skewed metric, that takes far more users than the “n greater than 30” of introductory textbooks.

In the seventh rule of thumb of their KDD 2014 paper, Kohavi, Deng, Longbotham and Xu propose a minimum of 355 times the square of the skewness coefficient per variant, recommended when absolute skewness exceeds 1. They publish the table of real Bing metrics, and it reproduces the rule point by point:

Bing metric	absolute skewness	reported minimum sample	355 times the square	sensitivity at 80% power
Revenue per user	17.9	114k	113,746	4.4%
Revenue per user (capped)	5.2	9.7k	9,599	10.5%
Sessions per user	3.6	4.70k	4,601	5.4%
Time to success	2.1	1.55k	1,566	12.3%

They further report that at a commerce site, skewness for purchases per customer exceeded 10 and for revenue per customer exceeded 30. And they add an honest caveat that is almost never quoted alongside the rule: if control and treatment have the same distribution and the split is equally sized, the distribution of the delta is approximately symmetric (perfectly symmetric under the null hypothesis), and the rule then stops providing a useful lower bound, because it was built for absolute skewness above 1. In that case the power calculation from the earlier sections is what governs. The 355 rule is the alarm for the case where you read each arm separately, or where the arms are not equally sized.

Three ways to bring the number down (and what each costs)

If the number that came out is too large for your traffic, there are three legitimate levers. All of them move the coefficient of variation, because that is what governs.

1. Cap the tail

Cutting values above a ceiling drops the standard deviation far more than it drops the mean, and it is the ratio between the two that enters the calculation. Kohavi, Deng, Longbotham and Xu report that capping revenue per user at 10 dollars per user per week at Bing dropped skewness from 18 to 5.3, and that for the same sample size, capped revenue detects a 30 percent smaller change than raw revenue.

The cost is conceptual, not statistical: the capped metric answers a different question. You stop measuring revenue and start measuring revenue up to the ceiling. If the effect you are testing acts precisely in the tail (an expensive-item upsell, for instance), capping erases exactly what you wanted to see. The full treatment of that decision is in outliers and metric capping in A/B testing.

2. Reduce variance with pre-experiment data (CUPED)

Deng, Xu, Kohavi and Walker show that using a covariate measured before the experiment reduces variance by a factor of 1 minus the squared correlation between the metric and the covariate. They report roughly 50 percent variance reduction at Bing, which they describe as equivalent to doubling traffic or halving duration. They also report that the best single covariate is the same metric measured in the pre-experiment period (over 45 percent reduction in their case, against 9 to 10 percent using entry day alone, with only 2 to 3 percent extra gain from combining the two).

The most important caveat in this article comes from that same study, and it lands on exactly the metric we are dealing with. The authors report that CUPED worked well on clicks per user and visits per user, and that the notable exception was revenue per user, where variance reduction came in below 5 percent, because the correlation of revenue per user between the pre-experiment period and the experiment period is low. That makes sense: whoever spent a lot last month is not, in general, whoever will spend a lot this month. The table below shows what would happen to the store’s sample at each correlation level, and it is useful for sizing the potential gain. It is not a forecast: for revenue per user, the correlation observed in the Bing study would sit at the first row of the table or below it. Measure the correlation on your own data before counting on any of these rows.

Applied to the store example, starting from the 761,217:

correlation with the covariate	variance factor	N per variant	days
0.5	0.750	570,913	67
0.6	0.640	487,179	57
0.7	0.510	388,221	46
0.8	0.360	274,039	32

The cost is engineering: it requires storing per-user data from the prior period and applying the adjustment in the analysis. And it requires the user to exist before the experiment, which does not hold for new visitors. Details in what is CUPED.

3. Change the primary metric

The cheapest lever and the least used. If revenue per user takes 89 days and conversion rate takes 57, and if the test hypothesis is about checkout friction (which acts on conversion, not on basket size), the honest primary metric is conversion, with revenue as a monitored secondary. This is not picking the metric that gives you the answer you want: it is picking the metric on which the mechanism you are testing actually acts. How to make that choice without fooling yourself is covered in primary metric and OEC.

Ratio metrics are a third case

There is a family that looks continuous and is neither: metrics whose denominator is not the randomized unit. Clicks per page, items per session, revenue per order. In those, each user contributes several correlated observations, and a standard error computed as if they were independent comes out smaller than the true one. The sample calculation in this article still applies, but the coefficient of variation has to be the one of the metric aggregated by randomized unit, not of the individual observation. The correct treatment is the delta method, covered in ratio metrics in A/B testing, and the unit choice that creates the problem is in randomization unit.

How to estimate the coefficient of variation before the test

All of this depends on a number you need before you start: the coefficient of variation of the metric in your own data. It is not a market benchmark, it is a property of your base. Three ways to get it, in order of preference:

1. Compute it directly from history. Take the last 4 to 8 weeks, aggregate the metric per user over the same window the test will use, and divide the standard deviation by the mean. It is the right path and takes minutes in any data tool.
2. Reconstruct it from the decomposition. If all you have is a conversion rate and order value statistics, use the identity from step 2: 1 plus the squared coefficient of variation of order value, divided by the rate, minus 1. You need the standard deviation of order value, not just the mean.
3. Run an A/A test first. A two-week A/A test hands you the observed dispersion for free, along with confirmation that the randomization is sound.

What does not work is adopting a “typical” coefficient of variation. The revenue dispersion of a store selling 30-dollar items and that of a furniture store have no relationship, and the number enters the calculation squared.

Checklist before setting sample size

1. Is the primary metric binary or continuous? If it is continuous, a proportion calculator does not answer it.
2. Do you have its standard deviation in your own data? Without that, the number that comes out is a guess wearing the clothes of a calculation.
3. Does the aggregation window of the estimate match the test? Revenue coefficient of variation over 7 days and over 28 days are different numbers.
4. Does skewness exceed 1? If so, check the 355 times squared skewness rule, remembering the equal-split caveat.
5. Is the metric’s denominator the randomized unit? If not, it is a ratio metric and needs the delta method.
6. Did you compute both numbers, conversion and revenue? The test runs for the larger of the two, and it is better to know that beforehand.
7. Were the reduction levers considered in the right order? Changing the primary metric is cheaper than capping, which is cheaper than implementing CUPED.

Common mistakes

• Using the conversion rate to size a revenue test. This is the central mistake of this article. The number looks plausible and has no relationship to the question.
• Estimating the coefficient of variation over buyers instead of over all users. Revenue per user includes the zeros. Ignoring them artificially deflates dispersion and undersizes the test.
• Comparing coefficients of variation from different windows. Aggregating over 28 days instead of 7 changes both the mean and the standard deviation, and not in the same proportion.
• Applying capping after seeing the result. Picking the ceiling by looking at which value makes the p-value pretty is fishing, not variance reduction. The ceiling goes into the analysis plan, up front.
• Treating the 355 rule as a replacement for the power calculation. They are two different bounds and the test has to clear both.
• Assuming CUPED always delivers 50 percent. The factor depends on the correlation measured in your own base, and a correlation of 0.5 delivers 25 percent, not 50. In the original study revenue per user was the notable exception, with less than 5 percent reduction.

Make this automatic with Donnu

The expensive error here is not getting the formula wrong, it is never noticing there were two. A revenue metric lands in the report next to a conversion metric, with the same shape, the same interval and the same color, and nothing on screen says one of them was sized and the other was not.

At Donnu, every metric declared in an experiment carries its family, and the power calculation is done per metric: a continuous metric requires an estimated dispersion from your own base, and the report shows the effective power that metric actually reached, not the power the conversion metric reached. If you want to redo any calculation by hand, the sample size calculator solves the binary case and the duration calculator turns any N into calendar days.

References

• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the seventh rule (“Have Enough Users”), of the minimum of 355 times the squared skewness coefficient per variant, of the recommendation to apply it when absolute skewness exceeds 1, of the Bing metrics table (revenue per user at skewness 17.9 and 114k sample at 4.4 percent sensitivity; capped revenue 5.2 and 9.7k at 10.5 percent; sessions per user 3.6 and 4.70k at 5.4 percent; time to success 2.1 and 1.55k at 12.3 percent), of the report of skewness above 10 for purchases per customer and above 30 for revenue per customer at a commerce site, of the effect of the 10 dollars per user per week cap dropping skewness from 18 to 5.3 and allowing detection of a 30 percent smaller change at the same sample size, and of the caveat that with an equal split the distribution of the delta is approximately symmetric and the rule stops giving a useful lower bound. exp-platform.com.
• Deng, A., Xu, Y., Kohavi, R. and Walker, T. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. WSDM 2013. Source of the variance reduction factor of 1 minus the squared correlation, of the result of about 50 percent variance reduction at Bing described as equivalent to doubling traffic or halving duration, and of the empirical result that the metric itself in the pre-experiment period is the best single covariate (over 45 percent reduction, against 9 to 10 percent with entry day, with only 2 to 3 percent extra gain from combining the two), and of the notable exception they report: for revenue per user, variance reduction came in below 5 percent because of the low correlation of the metric between the pre-experiment period and the experiment period. exp-platform.com.
• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the quadratic relationship between sensitivity and traffic (increasing sensitivity by a factor of 10, from a 5 percent delta to 0.5 percent, requires 100 times more users) and of the general guidance of at least thousands of active users. exp-platform.com.
• Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Reference on metrics whose denominator is not the randomization unit and on the correct treatment of variance in that case. arxiv.org/abs/1803.06336.

Read also: How many visitors an A/B test needs · Outliers and metric capping · What is CUPED · Ratio metrics in A/B testing · Minimum detectable effect · Sample size calculator · Leia em português