Count Metrics in A/B Tests: Poisson and Overdispersion

Sessions per user, items per order and pages per session are count metrics, and their variance is almost always larger than their mean. That is called overdispersion, and it has a direct price in sample size: at a dispersion index of 4, a common value in product data, a test that a Poisson model would size at 2,011 users per variant actually needs 8,194. Ignore it and compute the standard error the Poisson way, and it comes out at half the correct value, so the same data that gives a z of 3.42 in the correct calculation gives 6.85 in the naive one. This guide covers how to measure dispersion, how to size correctly, and when to swap the count for a binary proxy. It is part of our complete guide to A/B testing and is the direct counterpart to sample size for revenue and continuous metrics, which covers means and revenue.

Count metrics are neither a proportion nor an ordinary mean

Most A/B testing tools know how to do one calculation: compare two proportions. That covers conversion rate, click rate, bounce rate, anything where each user contributes a zero or a one.

A count is a different animal. Each user contributes a non-negative integer, and the distribution of those integers has a tail:

count metric	unit	where it shows up
sessions per user	randomized user	engagement, weekly retention
pages per session	session	browsing depth, internal search
items per order	order	ecommerce, basket size
searches per visit	visit	site search, catalogue
messages sent per account	account	communication products
tickets opened per customer	customer	support guardrail metric

They all share the same behaviour: a mass of users with zero, one or two events, and a minority with dozens. It is the same long tail we discussed in outliers and metric capping, except that here it comes from the nature of counting rather than from an extreme monetary value.

Poisson assumes the variance equals the mean

The standard model for a count is Poisson. It has a single parameter, which is simultaneously the mean and the variance of the distribution. That is the entire assumption, and it essentially never holds in product data.

The reason is simple and not statistical: users are not alike. A Poisson with mean 3.2 would describe a population where everybody has the same propensity to generate sessions and the variation is just luck. Your real user base has one group that visits once a month and another that visits three times a day. That heterogeneity between users enters the total variance and does not enter the mean.

The model that accommodates this is the negative binomial, which has two parameters: the mean and a dispersion parameter, usually written kappa. The relationship between them is direct:

variance = mean + kappa times the square of the mean

With kappa at zero, the negative binomial becomes exactly the Poisson. The larger the kappa, the more spread out the distribution around the same mean.

Measure dispersion before sizing anything

You do not need any special tooling to know whether you have overdispersion. Take a week of historical data, compute the mean and variance of the count per user, and divide one by the other. That is the dispersion index:

dispersion index = variance divided by mean

Under Poisson it is 1. In product data it usually lands between 2 and 8. And it relates to kappa exactly: the dispersion index equals 1 plus kappa times the mean, which lets you move from one to the other without simulation.

With a mean of 3.2 sessions per user per week and a target of detecting plus 5 percent on the rate, the calculation moves like this:

observed variance	dispersion index	kappa	users per variant	days at 40,000 users per week
3.2	1.0 (Poisson)	0.0000	2,011	1
6.4	2.0	0.3125	4,072	2
9.6	3.0	0.6250	6,133	3
12.8	4.0	0.9375	8,194	3
19.2	6.0	1.5625	12,315	5

Note that the relationship is essentially linear in the dispersion index. Ignoring overdispersion is not a rounding error, it is an error by a whole factor. Anyone who sized a test with a dispersion index of 4 using Poisson is running with a quarter of the sample they need, and will read the result as “nothing happened” when the test never had the power in the first place.

The formula, and the proof that it is right

The reference for sizing the ratio of two count rates is Zhu and Lakkis, published in Statistics in Medicine in 2014. It is the formula implemented in sample size software and the one the PASS documentation describes as method 3, the version that estimates the null variance by maximum likelihood. Zhu and Lakkis record that their simulations indicate methods 2 and 3 are more accurate than method 1.

Here it is, in JavaScript, exactly as we used it to generate the tables in this article:

// Zhu and Lakkis (2014), method 3 (maximum likelihood under the null).
// lambda1 = control rate | RR = target rate ratio
// kappa   = dispersion parameter | mut = average exposure time
function nbSampleSize({ lambda1, RR, kappa, mut = 1, R = 1, alpha = 0.05, power = 0.8 }) {
  const lambda2 = lambda1 * RR;
  const VA = (1 / mut) * (1 / lambda1 + 1 / (R * lambda2)) + ((1 + R) * kappa) / R;
  const V0 = Math.pow(1 + R, 2) / (mut * R * (lambda1 + R * lambda2)) + ((1 + R) * kappa) / R;
  const za = zCritical(alpha, 2);   // 1.9600 two-sided at 5%
  const zb = normInv(power);        // 0.8416 for 80% power
  return Math.ceil(
    Math.pow(za * Math.sqrt(V0) + zb * Math.sqrt(VA), 2) / Math.pow(Math.log(RR), 2)
  );
}

A formula published without verification is just a claim. We ran our implementation against the reference scenario Zhu and Lakkis publish in Table I of the paper (control rate of 0.8 events per unit of time, rate ratio of 0.85, average exposure time 0.75, alpha 0.05, 80 percent power, balanced allocation) and compared it row by row:

dispersion parameter	our implementation	published value	difference
0.4	1,311	1,311	0.000%
0.7	1,490	1,490	0.000%
1.0	1,668	1,668	0.000%
1.5	1,965	1,965	0.000%

Exact reproduction on all four rows. It is the same block of code that generated the dispersion index table above and the comparison below.

How much Poisson lies, by effect size

With the dispersion parameter fixed at 0.9375 (which corresponds to variance 12.8 over mean 3.2, or a dispersion index of 4), the difference between sizing with Poisson and sizing correctly is constant and large:

target effect on the rate	negative binomial	days	Poisson (kappa zero)	days	ratio
plus 2 percent	49,915	18	12,387	5	4.0 times
plus 3 percent	22,376	8	5,532	2	4.0 times
plus 5 percent	8,194	3	2,011	1	4.1 times
plus 8 percent	3,282	2	797	1	4.1 times
plus 10 percent	2,135	1	515	1	4.1 times

Days computed at 40,000 users per week, two arms. The ratio between the two columns stays pinned around 4, which is exactly the dispersion index: the correction factor is the dispersion index, and it does not depend on the size of the effect you want to detect. That is the rule of thumb worth keeping: measure the dispersion index once, multiply the Poisson calculation by it, and you have a good approximation before you even open the formula.

The most expensive mistake happens before the formula

There is a worse mistake than sizing with Poisson, and it happens at the reading stage rather than the design stage: counting the event as the unit of observation.

The scenario: 30,000 randomized users, 96,000 sessions generated. The temptation is to analyse those 96,000 sessions as independent observations, or to compute the standard error of the mean assuming variance equals the mean, the way Poisson prescribes. Both mistakes land in the same place.

calculation	standard error of the mean per arm	z of the difference between the two arms, for an effect of plus 0.10 sessions per user
correct (variance 12.8 over 30,000 users)	0.020656	3.4233
naive (variance assumed equal to the mean, 3.2)	0.010328	6.8465

To redo the arithmetic: the standard error of the difference between two arms of the same size is the square root of 2 times the standard error of one arm, so the correct z is 0.10 divided by 0.020656 times the square root of 2, which gives 3.4233.

The standard error comes out exactly halved, the confidence interval comes out 50 percent narrower than it should be, and the z doubles. In this specific case both calculations agree on the verdict (both clear 1.96), but the one producing a z of 6.85 will be described as an “overwhelming result” in a meeting, and it is not. With a smaller effect, the same distortion becomes an outright false positive.

The root of the problem is the same one we described in randomization unit: the unit of analysis has to be the unit of randomization. If you randomized users, the observation is the user and their metric is the total count, with all the variance it carries. Sessions from the same user are correlated with each other, and treating them as independent invents information that does not exist.

Is it worth swapping the count for a binary proxy?

One pragmatic escape is to convert the count into an indicator: instead of “sessions per user”, measure “user with at least 4 sessions in the week”. That hands the problem back to the binary calculator every tool ships with, and to the two-proportion formula.

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

With the proxy baseline at 35 percent of users, the calculator above returns:

relative effect on the proxy	users per variant	days at 40,000 per week
plus 2 percent	73,214	26
plus 3 percent	32,612	12
plus 5 percent	11,791	5
plus 8 percent	4,635	2

Compare that with the negative binomial column: detecting plus 5 percent on the session rate asked for 8,194 users per variant, while detecting plus 5 percent relative on the proxy asks for 11,791. In this scenario the proxy came out more expensive, not cheaper.

That is not a general rule, and this is exactly where care is needed. The two lines do not measure the same thing: one asks whether the average count rose by 5 percent, the other asks whether the share of users above a threshold rose by 5 percent. There is no automatic conversion between them, and assuming that a plus 5 percent effect on the mean produces plus 5 percent on the proxy is your hypothesis, not a fact. The honest criteria for choosing the proxy are:

• Choose the proxy if the threshold is what the business cares about. “An active user is one who visits 4 times a week” is a legitimate product definition, and in that case the proxy is not an approximation, it is the metric.
• Choose the proxy if you cannot compute the variance of the count. Without the variance number you have no way to size the negative binomial, and a well-sized proxy beats a badly sized count.
• Do not choose the proxy just so you can use the calculator you already have. The cost is throwing away the information about how much the count moved, and the table above shows that this does not always come with a sample discount.
• Never compare the two numbers as if they were alternatives for the same effect. They answer different questions.

And the normality of the mean

One last check remains, independent of sizing for the effect: the mean of the count has to be approximately normal for the z test to hold. The ruler is the rule of thumb from Kohavi, Deng, Longbotham and Xu, which asks for at least 355 times the square of the skewness coefficient and is recommended when absolute skewness exceeds 1.

variance (mean 3.2)	skewness	minimum from the CLT rule	minimum from a plus 5 percent effect	which one binds
3.2	0.56	111	2,011	the effect
6.4	1.19	500	4,072	the effect
12.8	1.96	1,359	8,194	the effect
19.2	2.51	2,238	12,315	the effect

With typical counts, sizing for the effect binds comfortably, and the central limit theorem rule never becomes the bottleneck. That is why counts are a more comfortable case than revenue: in the Bing metrics Kohavi and coauthors publish, sessions per user has skewness 3.6 and asks for 4.70 thousand observations, while revenue per user has 17.9 and asks for 114 thousand. The tail of a count is long, but the tail of money is far worse.

Two honest caveats about the table above. First: the skewness values come from the closed-form formula of the negative binomial with those means and variances, not from your data. Second: if your count has a handful of users with hundreds of events, which is common when bots enter the calculation, the real skewness will run well past this table. Before sizing, clean the bot traffic and consider applying a cap.

Checklist

1. Measure the variance before anything else. One week of historical data, count per user, mean and variance. Divide one by the other.
2. If the dispersion index passes 1.5, forget Poisson. Use the negative binomial and the corresponding kappa.
3. As a quick approximation, multiply the Poisson calculation by the dispersion index. The factor stayed pinned at 4.0 to 4.1 across the whole range of effect sizes tested.
4. The unit of analysis is the unit of randomization. Never compute a standard error over the event count.
5. Clean bots and consider a cap before estimating the variance, otherwise you size for the wrong tail.
6. If you use a binary proxy, declare the translation hypothesis. And do not assume it is cheaper: in this scenario it was not.
7. Check the 355 times skewness squared rule. With counts it rarely binds, but confirming takes a minute.

Make this automatic with Donnu

Almost every A/B testing tool on the market only knows how to compare proportions. When a team wants to test “sessions per user” or “items per order”, the default escape is to invent a binary proxy and hope, or to export everything to a spreadsheet and do the arithmetic by hand, with no sizing beforehand.

Donnu stores the count per randomized user, so the dispersion index is something you read before starting the test, rather than discovering afterwards. That changes the planning question: instead of “how many visitors do I need”, which is the binary question, you can answer “how much does this metric vary across my users and what does that cost in test days”. If your case is still binary, start with the sample size calculator; if it is already a count, measure the variance first and use the formula published above.

References

• Zhu, H. and Lakkis, H. Sample size calculation for comparing two negative binomial rates. Statistics in Medicine, volume 33, number 3, pages 376 to 387, 2014. Source of the sample size formula for the ratio of two rates under the negative binomial model, of the three methods for estimating the variance under the null hypothesis, of the simulation result that methods 2 and 3 are more accurate than method 1, and of the reference table (control rate 0.8, rate ratio 0.85, average exposure 0.75, alpha 0.05, 80 percent power) that we reproduce exactly on all four dispersion rows tested. pubmed.ncbi.nlm.nih.gov.
• NCSS. PASS Sample Size Software, chapter 438: Tests for the Ratio of Two Negative Binomial Rates. Source of the operational transcription of the Zhu and Lakkis formulas (asymptotic variance under the alternative and under the null by all three methods, sample size and power formulas), of the framing that the Poisson model assumes mean equals variance and fits the data poorly once overdispersion occurs, of the note that asymptotic tests are appropriate above 50 subjects per group, and of the numerical validation table with 1,311, 1,490, 1,668 and 1,965 subjects per group for dispersion of 0.4, 0.7, 1.0 and 1.5. ncss.com.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the rule of thumb of 355 times the square of the skewness coefficient as the minimum number of independent observations for the mean to be approximately normally distributed, of the recommendation to apply it when absolute skewness exceeds 1, and of the table of real Bing metrics in which sessions per user appears with skewness 3.6 and 4.70 thousand observations against revenue per user with 17.9 and 114 thousand. exp-platform.com.
• Liu, M., Sun, X., Varshney, M. and Xu, Y. Large-Scale Online Experimentation with Quantile Metrics. arXiv 1903.08762, 2019. Source of the record that experimentation platforms at scale report predominantly average metrics, such as revenue per member or clicks per impression, because the average fits directly into the two-sample t-test procedure, and that metrics not well summarised by an average require their own treatment. arxiv.org.

Read next: Sample size for revenue and continuous metrics · Outliers and metric capping · Randomization unit · Ratio metrics · Bot traffic · Sample size calculator · Leia em português