Regression to the Mean: Why the Peak Never Repeats

Any unit chosen for sitting at an extreme tends to drift back toward the average on its own at the next measurement, with nobody doing anything. In a simulation of 200 pages with an identical true conversion rate of 3 per cent, the 20 worst pages of period one rose from 2.085 per cent to 2.885 per cent in period two, an apparent 38.37 per cent improvement with a p value of 0.00000028 and no change shipped. That is regression to the mean, and it is why before versus after readings on pages selected for poor performance almost always look like they worked. This guide covers the arithmetic, the reproducible simulation and the design that immunises the experiment. It is part of our complete guide to A/B testing and the counterpart, on the axis of selection before the test, of what the winner’s curse covers on the axis of selection after it.

The phenomenon has been named since 1886

Galton analysed 930 adult children and their 205 respective parentages, transmuting female statures to their male equivalents, and set what he called the level of mediocrity for that population at 68 and a quarter inches. The law he stated is about deviations, not absolute values: the height deviate of the offspring is, on the average, two thirds of the height deviate of its mid parentage. Very tall parents have tall children, but less tall than themselves; very short parents have short children, but less short. Galton records that the ratio appeared with unexpected coherence and precision, and that the same pattern had already emerged in his earlier seed experiments, where the produce converged toward an average size.

The modern translation closest to anyone who measures things comes from Barnett, van der Pols and Dobson, who define the effect as a statistical phenomenon capable of making natural variation in repeated data look like real change, happening when unusually large or small measurements tend to be followed by measurements that are closer to the mean. The authors record that the effect becomes more pronounced with increasing measurement error and when follow up measurements are examined only on sub samples selected by baseline values, and conclude that it can be alleviated through better study design and the use of suitable statistical methods.

That second condition describes, word for word, what an optimisation team does every week: pick the page with the worst conversion rate and go change it.

The arithmetic in one line

If an observed measurement sits d units away from the population mean and the correlation between the first and second measurement is r, the expected value of the second measurement sits r times d from the mean. The share that comes back on its own is therefore (1 minus r) times d.

observed deviation in period 1	correlation r between periods	expected deviation in period 2	recovery with no intervention
1.00 pp below the mean	0.0	0.00 pp	1.00 pp
1.00 pp below the mean	0.2	0.20 pp below	0.80 pp
1.00 pp below the mean	0.4	0.40 pp below	0.60 pp
1.00 pp below the mean	0.6	0.60 pp below	0.40 pp
1.00 pp below the mean	0.8	0.80 pp below	0.20 pp
1.00 pp below the mean	1.0	1.00 pp below	0.00 pp

The two ends of the table anchor the intuition. A correlation of 1 means the difference between units is entirely real and stable: no regression, what was worse stays worse. A correlation of 0 means the observed difference was entirely noise: regression is total, and the “worst” unit returns exactly to the mean with nothing happening.

Real cases are almost never at the extremes. Pages genuinely differ from one another, and each measurement carries sampling noise. The correlation between two periods mixes both, and it is what decides how much of the observed “gain” was already locked in before any change.

A reproducible simulation: 200 identical pages

The example below is a fixed seed simulation, so anyone reproduces the same numbers. The starting point is deliberately extreme in order to isolate the effect: all 200 pages have exactly the same true conversion rate, 3.00 per cent. There is no better or worse page. Every observed difference is pure sampling noise.

Each page gets 1,000 visitors in period one and another 1,000 in period two, drawn independently, with no change between the two moments.

// fixed seed generator, so the result is reproducible
function mulberry32(a){return function(){a|=0;a=a+0x6D2B79F5|0;let t=Math.imul(a^a>>>15,1|a);t=t+Math.imul(t^t>>>7,61|t)^t;return ((t^t>>>14)>>>0)/4294967296;}}
const rnd = mulberry32(20260828);
const TRUE_RATE = 0.03, PAGES = 200, VISITORS = 1000;
const draw = () => { let c = 0; for (let i = 0; i < VISITORS; i++) if (rnd() < TRUE_RATE) c++; return c; };
const p1 = Array.from({ length: PAGES }, draw);
const p2 = Array.from({ length: PAGES }, draw);
// sort by period one and take the 20 worst and the 20 best
const order = p1.map((_, i) => i).sort((x, y) => p1[x] - p1[y]);
const worst = order.slice(0, 20), best = order.slice(-20);

The result:

group	rate in period 1	rate in period 2	relative change
All 200 pages	2.990%	2.966%	minus 0.80%
The 20 worst of period 1	2.085% (417 of 20,000)	2.885% (577 of 20,000)	plus 38.37%
The 20 best of period 1	3.910% (782 of 20,000)	2.840% (568 of 20,000)	minus 27.37%
The single worst page	1.60% (16 of 1,000)	3.30% (33 of 1,000)	plus 106%

The full set row is the honest control: 2.990 per cent against 2.966 per cent, essentially nothing, exactly as it should be. The two middle rows are the entire problem in two sentences.

The statistical test does not protect you

Here is the detail that lets this bias survive technically competent teams. Take the group of 20 worst pages and run a two proportion test comparing period one against period two, using the same significance calculator the whole site uses, and the numbers are:

• 20,000 visitors with 417 conversions against 20,000 visitors with 577 conversions
• relative lift of 38.37 per cent, z of 5.1391
• p value of 0.00000028
• confidence interval of 0.495 to 1.105 percentage points on the absolute difference

And on the other side, for the 20 best: relative lift of minus 27.37 per cent, z of minus 5.9252, p value of 0.0000000031, interval of minus 1.424 to minus 0.716 percentage points.

Both results are highly significant. No change was made. The pages are statistically identical by construction, and the random number generator does not know which ones were picked.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

The significance test is not wrong. It is precisely answering a question that is not the one that matters. The question the calculator answers is: given this pair of samples, how unlikely is this difference under the null hypothesis? The question that matters is: was this difference caused by the change? The bridge between the two questions is randomisation, and it does not exist in a before versus after comparison of a unit that was selected precisely for its observed value in the first period.

Where regression to the mean shows up in real operations

1. Prioritising by the worst converting page. The canonical case. The page entered the list for sitting at an extreme, and part of the recovery was already locked in.
2. Optimising the campaign that had the worst week. Same structure, shorter period, more noise, more regression.
3. Reacting to a sudden metric drop. If the drop was partly noise, the partial recovery happens anyway, and whoever is touching things at that moment takes the credit.
4. Re testing the segment that did badly in the previous test. The segment was chosen by an extreme inside the test’s own data, which combines regression with misread heterogeneous effects.
5. Auditing vendors by the worst account of the quarter. The worst account of the quarter improves on its own, and the audit becomes a success story.
6. Recycling the test that saved a broken page as a case study. Here the effect stacks with the winner’s curse and with novelty.

The best known parallel to this dynamic comes from outside marketing. Tversky and Kahneman report, in 1974, an observation from experienced flight instructors: praise for an exceptionally smooth landing was typically followed by a poorer landing on the next try, while harsh criticism after a rough landing was usually followed by an improvement. The instructors concluded that verbal rewards are detrimental to learning and verbal punishments are beneficial. The authors record that this conclusion is unwarranted because of the presence of regression toward the mean: as in other cases of repeated examination, an improvement will usually follow a poor performance and a deterioration will usually follow an outstanding performance, even if the instructor does not respond at all to the trainee’s achievement on the first attempt.

Swap instructor for analyst, landing for week and trainee for page, and the structure is identical.

The design that immunises

In practical order:

1. Always use a concurrent control. Both halves of the chosen page carry the same regression, and the difference between them isolates the effect. That is the reason the design exists, described in how to run an A/B test.
2. Separate the selection period from the measurement period. If you picked the page using August data, do not use August as the baseline. Use September as the baseline and October as the measurement, or randomise within September.
3. Subtract the expected recovery before writing the hypothesis. Estimate the correlation between two comparable historical periods and use the table above. If the expected recovery is already 0.4 percentage points, the hypothesis has to aim above that for the experiment to be worth running.
4. Size the test on the incremental effect, not on the distance to the mean. The classic mistake is computing sample size with an MDE inflated by the distance that was going to come back anyway. The right criterion is in minimum detectable effect.
5. Read segments with the exploration ruler. A segment chosen after seeing the result is a selection by extreme inside the test itself, and the ruler is in multiple metrics and FDR.
6. Distrust a result that is too good on a unit that was broken. That is exactly the confirmation trigger described in Twyman’s law.

Common mistakes

• Reading before versus after on a page chosen for being the worst. This is the pure form of the bias.
• Assuming a small p value settles it. The simulation above produces a p value of 0.00000028 with no change at all.
• Using the same period to select and to measure the baseline. That embeds the regression inside the starting point.
• Promising in the plan the gain measured by the distance to the site average. Much of that distance returns on its own and is nobody’s achievement.
• Reading a drop at the top as fatigue or quality decay. The good extreme regresses too, and for the same reason.
• Confusing it with the novelty effect. Novelty is a real behavioural change that decays; regression is measurement noise that corrects itself. Both produce similar curves and call for different remedies.
• Applying the argument to a randomised test. With a concurrent control, regression hits both arms and cancels. Using regression to the mean to discredit a well designed test is as wrong as ignoring it in a before versus after reading.

Make this automatic in Donnu

Regression to the mean is not a computation problem, it is a design problem, and the design is decided the moment somebody chooses where to make a change. When the tool only enters after that choice, it inherits the bias fully formed.

In Donnu, every experiment starts with a concurrent control randomised over the same traffic in the same period, which is the only structure that makes regression cancel between arms, and the default reading of a result is always variant against control, never current period against previous period. If you want to check any calculation by hand, the significance calculator takes counts from any pair of groups, and the hypothesis generator asks for the expected effect before the test starts, which is the right moment to subtract what was going to come back on its own.

References

• Galton, F. Regression towards Mediocrity in Hereditary Stature. Journal of the Anthropological Institute, volume 15, pages 246 to 263, 1886. Source of the original statement of the law in terms of deviations: the height deviate of the offspring is, on the average, two thirds of the height deviate of its mid parentage; of the data used (930 adult children and 205 parentages, with female statures transmuted to their male equivalents and multiplied by 1.08); of the level of mediocrity set at 68 and a quarter inches without shoes; of the observation that the mean filial regression towards mediocrity was directly proportional to the parental deviation; and of the report that the same pattern had appeared earlier in his seed experiments, where the produce converged toward an average size. stat.ucla.edu.
• Barnett, A. G., van der Pols, J. C. and Dobson, A. J. Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, volume 34, issue 1, pages 215 to 220, 2005. Source of the definition of the phenomenon as something that can make natural variation in repeated data look like real change, happening when unusually large or small measurements tend to be followed by measurements that are closer to the mean; of the record that effects become more pronounced with increasing measurement error and when follow up measurements are examined only on sub samples selected by baseline values; and of the conclusion that the effect can be alleviated through better study design and the use of suitable statistical methods. academic.oup.com.
• Tversky, A. and Kahneman, D. Judgment under Uncertainty: Heuristics and Biases. Science, volume 185, number 4157, pages 1124 to 1131, 27 September 1974. Source of the flight instructor account on page 1127: praise for an exceptionally smooth landing was typically followed by a poorer landing on the next try, while harsh criticism after a rough landing was usually followed by an improvement, leading the instructors to the unwarranted conclusion that verbal rewards are detrimental and verbal punishments beneficial; and of the authors explanation that the conclusion does not hold because of regression toward the mean, since an improvement usually follows a poor performance and a deterioration usually follows an outstanding one even if the instructor does not respond at all to the trainee achievement on the first attempt. cs.tufts.edu.
• Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source of the context for why comparison against historical data is fragile in online environments and why controlled experiments with random, concurrent assignment are the reference standard for separating an effect from natural variation. exp-platform.com.

Read next: The winner’s curse · Minimum detectable effect · Heterogeneous treatment effects · Twyman’s law · How to run an A/B test · Significance calculator · Leia em português