Outliers in A/B Testing: When to Cap the Long Tail
Revenue outliers: one customer can invent a 50 percent win per user. How to use metric capping without trading one bias for another.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Revenue per user metrics are heavily skewed, and the mean is the most fragile summary you can pick for a distribution like that. A single customer, one revenue outlier, moves an entire arm average by tens of percent with nothing at all having happened to the product. This guide covers a worked example where one corporate buyer invents a 50 percent win single handedly, the rule of thumb published by the Bing team for knowing how many users your metric actually needs, why the power floor and the normality floor are two different floors, how to choose a cap without turning capping into manipulation, and what capping definitely does not fix. It is part of our complete guide to A/B testing and connects directly to ratio metrics and the delta method.
The mean cannot carry your revenue distribution
Conversion rate is a well behaved metric: each user contributes zero or one, variance is bounded, and the central limit theorem arrives quickly. Revenue per user has none of those properties. Most users contribute zero, most buyers contribute little, and a tiny fraction contributes hundreds of times the average.
The size of this is not an impression. Kohavi, Deng, Longbotham and Xu report that at a commerce site the skewness of purchases per customer exceeded 10 and that of revenue per customer exceeded 30. At Bing, revenue per user had a skewness coefficient of 17.9. For comparison, a symmetric distribution has skewness zero, and the authors only invoke their rule of thumb when absolute skewness passes 1.
Worked example: one purchase that invents 50 percent
A B2B commerce site runs a test with 60,000 users per arm. Real behavior is identical on both sides: the change under test moved no purchase at all. Each arm accumulates 240,000 in revenue, giving an honest average of 4.0000 per user.
Except one user in arm B is a corporate buyer who placed a single order of 120,000. Not fraud, not a robot, not a bug: a real sale that would have happened anyway.
| Arm | Users | Total revenue | Revenue per user |
|---|---|---|---|
| A, control | 60,000 | 240,000 | 4.0000 |
| B, treatment, with the corporate order | 60,000 | 360,000 | 6.0000 |
| Measured difference | 120,000 | plus 2.0000, plus 50.00 percent |
A 50 percent win in revenue per user, produced by 1 user out of 120,000. Had the order landed in arm A, the very same experiment would have produced a 33.3 percent loss. The test outcome was decided by a coin flip that has nothing to do with the product.
Now the same experiment with a cap of 200 per user, set in advance and applied equally to both arms:
| Arm | Capped total revenue | Capped revenue per user |
|---|---|---|
| A, control | 240,000 | 4.0000 |
| B, treatment | 240,200 | 4.0033 |
| Measured difference | plus 0.0033, plus 0.08 percent |
The correct reading appeared: no effect. And what happened to the company real revenue? Nothing. The 120,000 hit the bank. What the cap did was stop a sale that would have happened regardless from being credited to a product change.
The Kohavi rule: 355 times skewness squared
If the mean is slow to become normal, the confidence interval your calculator prints is not worth what it claims. The practical question is how many users are enough. Kohavi, Deng, Longbotham and Xu published a direct rule of thumb at KDD 2014, derived from work by Boos and Hughes-Oliver: the minimum number of independent and identically distributed observations for the mean to be approximately normally distributed is 355 times the square of the skewness coefficient, per variant. They recommend applying it whenever absolute skewness exceeds 1.
Here is the table they published with real Bing metrics, alongside what the formula returns when redone by hand:
| Bing metric | Skewness (absolute) | 355 times the square | Published sample | Sensitivity at 80 percent power |
|---|---|---|---|---|
| Revenue per user | 17.9 | 113,746 | 114k | 4.4 percent |
| Revenue per user, capped | 5.2 | 9,599 | 9.7k | 10.5 percent |
| Sessions per user | 3.6 | 4,601 | 4.70k | 5.4 percent |
| Time to success | 2.1 | 1,566 | 1.55k | 12.3 percent |
The recomputed arithmetic reproduces the published table within the rounding of the skewness coefficient, and the jump between the first two rows is the entire case for capping in a single number: the same metric, capped, needs twelve times fewer users for its mean to be trustworthy.
Two different floors, and the higher one wins
Here is the confusion that makes teams underestimate test duration. A sample size calculator answers a question about power: how many users to detect an effect of size X. The 355 rule answers a question about validity: how many users for the confidence interval arithmetic to mean anything. They are separate, independent floors, and you have to clear both.
Run the power floor for your binary conversion metric:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
With a 5 percent baseline and a minimum detectable effect of 10 percent relative, at 95 percent confidence and 80 percent power, it returns 31,234 users per variant. If the same test also tracks revenue per user with skewness near 18, the normality floor of that metric alone is 113,746 per variant. The test you thought would take two weeks takes seven, and the metric setting the pace is revenue, not conversion.
| Floor | What it answers | Example | Value |
|---|---|---|---|
| Power | can I detect the effect I care about | 5 percent conversion, 10 percent relative MDE | 31,234 per variant |
| Normality | does the confidence interval mean what it says | revenue per user, skewness 17.9 | 113,746 per variant |
| Decision | take the larger of the two | 113,746 per variant |
Kohavi and colleagues add a cheap recommendation that helps for free: make sure control and treatment are equally sized. With a 50 against 50 split and similar distributions, the distribution of the delta becomes approximately symmetric, and skewness stops being the problem it was. Unequal splits cost more than they look, and checking for them is the subject of sample ratio mismatch.
Choosing a cap without it becoming manipulation
The difference between method and manipulation lives entirely in the procedure, not in the value of the cap.
- Choose the cap before the experiment, from historical data, never by looking at the result. Kohavi and colleagues report capping Bing revenue per user at 10 dollars per user per week; skewness fell from about 18 to about 5, and at the same sample size the capped metric could detect a change 30 percent smaller than the uncapped one.
- Apply exactly the same cap to both arms. An asymmetric cap is not analysis, it is picking a winner.
- Fix a window along with the value. Ten dollars per user per week is not the same as ten dollars per user per experiment. Without a declared window, the same cap silently tightens on longer tests.
- Report both metrics. The capped one decides, the raw one accompanies. When the two disagree in a large and persistent way, that is information: it means the effect is concentrated in the tail, and the business decision may differ from the statistical one.
- Say which question you are answering. The capped metric speaks about the typical customer. It does not speak about total company revenue, and presenting it as if it did is the only real way to lie with capping.
One alternative to a hard cap, when the tail is the product rather than the noise: instead of limiting, reduce variance another way. CUPED uses pre-experiment data to strip out variation that has nothing to do with the treatment and usually buys sensitivity without touching a single observed value.
What capping does not fix
A cap is a tool against legitimate extreme values. It is not, and does not replace, cleaning dirty data.
A robot still has to be handled as a robot. Crook, Frasca, Kohavi and Longbotham draw the distinction carefully in their KDD 2009 paper: if robot traffic is distributed across variants in an unbiased way, it adds noise and reduces the power of the experiment but does not invalidate the results; a robot seen as multiple unique users because it resets cookies or runs from several machines introduces no bias either. The dangerous case is the robot acting as a single user that consistently generates traffic for one variant, because it can make that variant statistically better without any human having preferred anything. They report a case on the MSN portal where robots accepted cookies and executed JavaScript, firing click events at roughly 100 per minute for 2.5 hours.
A measurement defect stays a defect. If the difference comes from how much of the behavior got recorded rather than from the behavior, no cap will help. That is the subject of instrumentation bias, and the classic symptom is an unbalanced pre-trigger metric.
An effect concentrated in the tail does not vanish because you cut it. If your product change genuinely makes large customers buy more, the cap will hide exactly the effect that matters. The answer there is not to remove the cap mid test, it is to declare a second metric in advance, for instance revenue among large accounts, and size the test for it.
Outlier checklist before looking at the mean
- What is the skewness of my primary metric? Above 1, the 355 rule applies and probably changes your duration estimate.
- Have robots, internal tests and monitoring traffic already been removed? That comes before any capping decision.
- Was the cap set in advance, with a window, and does it apply to both arms?
- Do the raw and capped metrics tell the same story? A large divergence is information, not an annoyance.
- Is the split 50 against 50? It is the cheapest correction available for skewness.
- Was the higher of the two floors respected? Power and normality are separate requirements.
- Can a single user flip the decision? If so you do not have a result, you have a coin flip. Recompute each arm mean without its largest value and see whether the conclusion survives.
Common mistakes
- Removing outliers after seeing the result. It is the most common form of analysis bias, and it survives because it looks like hygiene.
- Confusing a large customer with dirty data. That is real revenue. Limiting its leverage over an average is legitimate; erasing it from the revenue report is not.
- Using the mean without looking at the median and the high percentiles. No skewed distribution can be described by a single number.
- Capping only the metric that is losing. That is picking the outcome with one extra step.
- Assuming a large sample fixes everything. It does, but the 355 rule shows the requirement grows with the square of skewness, and “large” can mean ten times what the team imagined.
Make this automatic with Donnu
The outlier problem is not mathematical, it is sequential: the capping decision is almost always made after someone has seen a strange number, and at that point it is no longer a neutral decision.
In Donnu, a revenue metric arrives with its skewness computed and with the 355 rule normality floor shown next to the power floor, so the duration estimate is born from the larger of the two. The cap is a field in the experiment design, filled in before the run and recorded in the history, and the report always shows the raw and capped metrics side by side, with a warning when they disagree. If you want to redo the duration arithmetic yourself, the sample size calculator and the test duration calculator take your own numbers.
References
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source for the rule of 355 times the square of the skewness coefficient as the minimum number of observations for the mean to be approximately normal, the recommendation to apply it when absolute skewness exceeds 1, the table with Bing revenue per user (skewness 17.9, 114 thousand users, 4.4 percent sensitivity), capped revenue (5.2, 9.7 thousand, 10.5 percent), sessions per user (3.6, 4.70 thousand, 5.4 percent) and time to success (2.1, 1.55 thousand, 12.3 percent), the account of capping revenue per user at 10 dollars per user per week with skewness dropping from about 18 to about 5 and a sensitivity gain equivalent to detecting a change 30 percent smaller at the same sample size, the skewness values above 10 and above 30 at the commerce site, the derivation from Boos and Hughes-Oliver, and the recommendation to keep control and treatment equally sized. exp-platform.com.
- Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source for the rule that unbiased robot traffic adds noise and reduces power without invalidating results, that robots resetting cookies or running from multiple machines introduce no bias, that a robot acting as a single user consistently assigned to one variant creates significant bias, and for the MSN portal case with robots firing click events at roughly 100 per minute for 2.5 hours. exp-platform.com.
- Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Reference on variance estimation for metrics whose analysis unit does not match the randomization unit, a case where the fragility of the mean to extreme values compounds the ratio problem. arxiv.org.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on metrics, variance and the treatment of extreme values in online experiments. Companion material at experimentguide.com.
Read next: Ratio metrics and the delta method · What is CUPED · Instrumentation bias · Guardrail metrics · Sample size calculator · Leia em português
Frequently asked questions
- What is metric capping in A/B testing?
- It is limiting the maximum value a single user can contribute to a metric before the average is computed. Kohavi, Deng, Longbotham and Xu report capping revenue per user at Bing to 10 dollars per user per week; the skewness of the metric fell from about 18 to about 5 and, at the same sample size, capped revenue per user could detect a change 30 percent smaller than the uncapped version. The cap has to be decided before the experiment and applied identically to both arms.
- How many users does a revenue metric need?
- More than the power calculation suggests. The rule of thumb published by Kohavi, Deng, Longbotham and Xu at KDD 2014 is 355 times the square of the skewness coefficient per variant, and they recommend using it whenever the absolute skewness exceeds 1. For Bing revenue per user, with skewness 17.9, that comes to roughly 114 thousand users per arm just for the mean to be approximately normal, before any consideration of statistical power.
- Is capping just massaging the result?
- It is massaging when the cap is chosen after seeing the data, or applied to one arm only, or adjusted until the number looks good. It is not massaging when the cap is set before the experiment from pre-experiment data, applied equally to both arms, documented, and reported alongside the uncapped metric. What genuinely changes is the question being answered: the capped metric speaks about the typical customer, not about total revenue, and that has to be stated.
- Does a robot ruin the revenue average of an experiment?
- It depends how it distributes. Crook, Frasca, Kohavi and Longbotham are precise about this: robot traffic spread across variants in an unbiased way adds noise and reduces the power of the experiment but does not invalidate results; a robot that resets cookies or runs from multiple machines shows up as multiple users and creates no bias either. The dangerous case is the robot that acts as a single user and consistently generates traffic for one variant, because then it can make that variant statistically better without any human having preferred anything.
- What is the difference between an outlier and a legitimate long tail?
- An outlier is a value that should not be in the dataset at all: a robot, an internal test, an instrumentation error, a duplicated order. A legitimate long tail is the real corporate customer who genuinely spent thirty thousand. The first should be removed, and removing it is the correct decision. The second should not be removed, because it is real company revenue; what you do instead is limit its leverage over the average, accepting that the capped metric answers a slightly different question.