Blinded Sample Size Re-Estimation in A/B Testing
How to recompute an A/B test sample size mid-flight when the planned baseline rate was wrong, without inflating false positives. Power: 42% to 78%.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
There is a legitimate way to change sample size after an A/B test has started: recalculate using only the POOLED conversion rate of both arms, without looking at the difference between them. In the simulation in this article, a plan built for a 5 percent baseline running on a true baseline of 2 percent delivered 42.0 percent power in the fixed design and 77.8 percent with blinded recalculation, while the false positive rate stayed between 4.37 and 5.08 percent across every scenario tested. This is not peeking: peeking means using the difference between A and B, and that is what breaks the test. This guide separates the two, measures the damage of a wrong baseline, quantifies when to recalculate, and records the one case where the method is not safe. It is part of our complete guide to A/B testing and complements how many visitors an A/B test needs and the peeking problem.
The plan is built on a number you guessed
Every sample size calculation needs four things: the significance level, the desired power, the minimum effect you want to detect, and the baseline rate. The first three are team decisions. The fourth is a measurement of the past, and it is the only one that can simply be wrong.
It goes wrong for mundane reasons: the history came from a different season, the page changed since the last measurement, the campaign brought traffic of a different quality, the conversion event was redefined, the bot filter changed. In the clinical trial literature this kind of parameter has a name: a nuisance parameter, a quantity that affects the distribution of the test statistic but is not what you want to test. For a binary outcome, the nuisance parameter is exactly the response rate pooled over both arms.
The reference paper on the blindrecalc package, by Baumann, Pilz and Kieser, illustrates the uncertainty with a meta analysis of 13 studies where the reported pooled morbidity rate ranged from 0.10 to 0.62. That is the same order of error a product team makes when estimating the conversion of a new checkout from the old one.
What a wrong baseline does to your sample size
Take a concrete plan: assumed baseline of 5 percent, minimum detectable effect of 10 percent relative, two-sided alpha of 5 percent, 80 percent power. The calculator returns 31,234 users per variant and 11 days at 40,000 visitors per week.
Now suppose the true baseline is something else. With the minimum effect held at 10 percent relative, which is how most teams think about it:
| true baseline | N actually needed | N in the plan | real power with the planned N | days needed |
|---|---|---|---|---|
| 2% | 80,682 | 31,234 | 41.4% | 29 |
| 3% | 53,211 | 31,234 | 57.4% | 19 |
| 4% | 39,475 | 31,234 | 70.3% | 14 |
| 5% | 31,234 | 31,234 | 80.0% | 11 |
| 6% | 25,740 | 31,234 | 87.0% | 10 |
| 8% | 18,872 | 31,234 | 95.0% | 7 |
| 10% | 14,751 | 31,234 | 98.3% | 6 |
Missing the baseline low by 3 percentage points costs half the power. And the team never finds out: the test ends, reads “not significant”, and the conclusion on record is that the variant did not work.
Note that the sign of the error flips if your minimum effect is expressed in absolute percentage points:
| true baseline | N needed (0.5 pp MDE) | real power with the planned N | days needed |
|---|---|---|---|
| 2% | 13,809 | 98.8% | 5 |
| 3% | 19,743 | 94.1% | 7 |
| 4% | 25,551 | 87.2% | 9 |
| 5% | 31,234 | 80.0% | 11 |
| 6% | 36,791 | 73.3% | 13 |
| 8% | 47,528 | 62.2% | 17 |
| 10% | 57,763 | 54.0% | 21 |
The two tables point in opposite directions because they measure different things. The practical lesson is that you need to know which of the two definitions your team uses before arguing about whether the sample is big or small. More on that in minimum detectable effect.
Blinded recalculation is not peeking
The confusion is understandable, because both involve looking at data mid-test. The separation is clean:
| what you look at mid-test | does it carry information about the effect? | consequence |
|---|---|---|
| the difference between A and B, to decide whether to stop | yes | serious false positive inflation, the peeking problem |
| each arm conversion separately, to resize | yes | unblinded recalculation, requires a formal correction |
| the POOLED conversion of both arms added together | no, under the null hypothesis | blinded recalculation, no correction needed |
The blinded estimator for a binary outcome is trivial to compute and is written exactly this way in the blindrecalc paper: add the conversions from both arms and divide by the sum of visitors from both arms. Nobody opens the split by variant. From that pooled rate, the per-arm rates under the assumed alternative are reconstructed arithmetically from the effect you had already declared in the plan, and N comes out the usual way.
The complete procedure, in Baumann, Pilz and Kieser’s formulation:
- Compute the initial N using the best available estimate of the baseline rate.
- Set the internal pilot point as a fraction of that N (they suggest 0.25, 0.5 or 0.75).
- On reaching that point, estimate the pooled rate and recalculate N.
- The final N is the minimum between the pre-specified cap and the maximum of the pilot N and the recalculated N.
- Collect the remainder and run the ordinary test on the total, with no level correction.
Step 4 hides a design choice. If you also impose the initial N as a floor, you have the restricted design, which can only grow. If you do not, you have the unrestricted design, which can shrink.
What the simulation showed
We simulated 6,000 replicates per cell, with the internal pilot at 50 percent of the planned N, a cap of 4 times the plan, and the unrestricted design. Every rate was read with the same significance function that powers the calculators on this blog.
Under the null hypothesis (true effect equal to zero), false positive rate:
| true baseline | fixed design | with blinded recalculation | mean final N |
|---|---|---|---|
| 2% | 5.12% | 4.70% | 80,628 |
| 3% | 5.07% | 4.57% | 53,157 |
| 5% | 5.03% | 5.08% | 31,212 |
| 8% | 4.77% | 4.37% | 18,861 |
The Monte Carlo standard error at 5 percent with 6,000 replicates is 0.28 percentage point, so all of these sit within two error bars of nominal. There is no type I error inflation. That matches what the literature records: in the superiority case, blinded sample size reassessment can be performed without relevant type I error rate inflation.
Under the alternative (true effect of +10 percent relative), power:
| true baseline | fixed design | with blinded recalculation | mean final N |
|---|---|---|---|
| 2% | 42.00% | 77.82% | 76,851 |
| 3% | 57.57% | 77.98% | 50,581 |
| 5% | 80.28% | 77.83% | 29,663 |
| 8% | 95.27% | 77.15% | 17,876 |
That table is the whole argument. The fixed design delivers power ranging from 42 to 95 percent depending on the luck of the initial estimate. Blinded recalculation delivers stable power around 77 percent in every scenario, and the cost of that is extra traffic when the true baseline is low and less traffic when it is high.
Why power lands at 78 and not at 80
Recalculation does not deliver exactly the 80 percent requested, and the reason is interesting enough to deserve its own section.
Under the null hypothesis, the pooled rate is an unbiased estimate of the control rate. Under the alternative it is not: it sits halfway between the control rate and the treatment rate, that is, slightly above the true baseline. And since the required N falls as the baseline rises, the recalculation returns an N that is slightly too small.
We measured the exact size of that bias with 3,000 replicates per row, with a true baseline of 5 percent:
| true effect | estimated pooled rate | recalculated N | N actually needed | deviation |
|---|---|---|---|---|
| 0% | 5.0049% | 31,201 | 31,234 | -0.11% |
| +5% | 5.1305% | 30,396 | 31,234 | -2.68% |
| +10% | 5.2550% | 29,634 | 31,234 | -5.12% |
| +20% | 5.5055% | 28,207 | 31,234 | -9.69% |
| +50% | 6.2540% | 24,624 | 31,234 | -21.16% |
Under the null the bias vanishes. Under a 10 percent effect, N comes out 5.12 percent short, which is exactly the 2 percentage points of power that went missing. And notice the upside: the larger the true effect, the more N shrinks, and that is precisely when a large effect does not need a large sample. The bias works in the right direction.
If you want the full 80 percent protected, the fix is trivial: ask for 82 or 83 percent power in the recalculation, or apply the restricted design rule described below.
Restricted or unrestricted: a business choice
The difference shows up when the true baseline comes in higher than assumed. We ran both designs with a true baseline of 8 percent and a plan built for 5 percent:
| design | false positive | power | mean N per variant |
|---|---|---|---|
| unrestricted (can shrink) | 5.15% | 77.23% | 17,892 |
| restricted (never below plan) | 5.57% | 94.92% | 31,234 |
The unrestricted design spent 43 percent less traffic and delivered the contracted power. The restricted design spent the plan and delivered power to spare. Neither is wrong: they serve different objectives.
- Pick the unrestricted design when traffic is the bottleneck and you have a queue of tests waiting. Every test that finishes early frees capacity for the next one, which is the logic behind how many A/B tests per month.
- Pick the restricted design when the cost of a false negative is high and traffic is not the bottleneck, or when the team will not accept seeing the planned sample cut mid-flight.
Declare the choice in the pre-registered analysis plan before the test starts, along with the pilot point and the cap. That is what separates blinded recalculation from improvisation.
When to recalculate matters less than you think
We tested the internal pilot point at five positions, with a true baseline of 3 percent and a plan built for 5 percent, 6,000 replicates each:
| pilot at | n1 per variant | false positive | power | mean final N | standard deviation of the estimated rate |
|---|---|---|---|---|---|
| 10% of plan | 3,123 | 5.10% | 78.28% | 50,841 | 0.2235 pp |
| 25% of plan | 7,809 | 4.85% | 78.87% | 50,628 | 0.1398 pp |
| 50% of plan | 15,617 | 5.57% | 77.78% | 50,618 | 0.0996 pp |
| 75% of plan | 23,426 | 5.23% | 77.75% | 50,573 | 0.0803 pp |
| 100% of plan | 31,234 | 5.20% | 78.72% | 50,552 | 0.0704 pp |
The N truly needed in this scenario is 53,211, and recalculation landed close to it at every position. The false positive rate stayed between 4.85 and 5.57 percent, within two error bars of nominal on every row.
What genuinely changes is the precision of the estimate: the standard deviation of the pooled rate fell by a factor of 3 between the 10 percent pilot and the 100 percent pilot. In practice that means recalculating early works as long as there is already enough data to estimate the rate stably. Baumann, Pilz and Kieser’s rule of thumb still holds: neither too early, when there is still great uncertainty about the parameter, nor too late, when there is no room left to adjust because the recalculated N has already been exceeded.
The case where this is NOT safe
There is one important exception and it has a name: non-inferiority and equivalence testing.
Glimm and Lauter show why. In a non-inferiority test with a margin, the total variance used in the recalculation is computed on data shifted by the margin, and that makes it carry the observed difference between the arm means. The result is that the supposedly blinded recalculation becomes mathematically equivalent to an unblinded one, with all the type I error bias that comes with it. Fewer additional observations are collected when the first-stage evidence favours the alternative, and more when it does not, which is exactly the recipe for inflating false positives.
The same authors record, on the other side, that in superiority testing the asymptotic guarantee holds: if the size of either stage grows, the recalculation rule implicitly fixes the total in advance and alpha is kept. The violations they construct are extremely small and occur at sample sizes that will rarely be of practical relevance. Their conclusion is explicit: the method can safely be used in practice, for superiority.
If your case is equivalence testing, the way out is an adjusted nominal level or a stage combination method, not the ordinary test. And it helps to know the price of the more conservative alternative: in Glimm and Lauter’s simulations, over 1 million runs, Fisher’s p-value combination typically loses 3 to 4 percentage points of power when the ordinary test power is below 95 percent, reaching 7 points in some scenarios (in one of them, 76.4 percent against 69.6 percent).
Run your own case
Use the calculator below twice: once with the baseline rate you assumed in the plan, and once with the pooled rate you measured at the internal pilot. The difference between the two N values is exactly the adjustment blinded recalculation tells you to make.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
The routine, in eight lines
- Compute the initial N with the best estimate of the baseline rate and record where that estimate came from.
- Choose and write down in advance the internal pilot point, the sample cap, and whether the design is restricted or unrestricted.
- Decide who performs the recalculation and make sure that person receives only the pooled rate, never the split by arm.
- On reaching the pilot point, compute the pooled rate: conversions in A plus conversions in B, divided by visitors in A plus visitors in B.
- Recalculate N with that rate, holding alpha, power and minimum effect exactly as they were.
- Apply the cap and floor of the chosen design and record the new N in the same document as the plan.
- Collect the remainder and run the ordinary test on the total, with no level correction.
- In the final report, state that a blinded recalculation happened, at what point, and which pooled rate was used.
Step 3 is what fails most often in practice, because the tool dashboard shows the split by variant on the first screen. If it cannot be hidden, the honest alternative is to record in the report that the recalculation was done with the result visible, and treat the test with the suspicion that deserves.
Make it automatic with Donnu
Blinded recalculation is a simple procedure that dies from two things: somebody opens the dashboard and sees the split by arm before recalculating, and nobody records that the recalculation happened. In Donnu the experiment carries the plan with it (assumed baseline, alpha, power, minimum effect, pilot point, cap) and the pooled read of both arms is a separate view from the per-variant read, so you can resize without opening the result. The new N enters the experiment history with a timestamp, and the final report states that a recalculation happened and with which rate. If your current tool does not separate those two views, that is the item to fix before the method can become routine.
Frequently asked questions
Short answers live in the FAQ section of this page, generated from the same measurements presented here.
References
- Lukas Baumann, Maximilian Pilz and Meinhard Kieser. blindrecalc: An R Package for Blinded Sample Size Recalculation, The R Journal 14/1, 2022. Carries the full internal pilot procedure, the formula for the final N as the minimum between cap and maximum, the distinction between restricted and unrestricted designs, the blinded estimator for binary outcomes, and the record that in the superiority case blinded re-estimation causes no relevant type I error inflation. It is also the source of the 2006 CHMP recommendation to use blinded reassessment methods whenever possible.
- Ekkehard Glimm and Jurgen Lauter. Some Notes on Blinded Sample Size Re-Estimation, 2013. Demonstrates that asymptotic type I error control is guaranteed, constructs the small-sample cases where it fails, and explains why inflation is far more serious in non-inferiority: the supposedly blinded recalculation is equivalent to an unblinded one. It also carries the power comparison between the ordinary test, the t-statistic combination and Fisher’s p-value combination, over 1 million simulation runs.
- Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu. Seven Rules of Thumb for Web Site Experimenters, KDD 2014. Reminds readers that the minimum sample size formula assumes the distribution of the mean is normal, and that many online experiment metrics are skewed enough to require a higher floor. Their rule of thumb is 355 times the square of the skewness coefficient per variant, recommended when the absolute skewness exceeds 1. It is the floor that blinded re-estimation does not replace: it corrects the baseline rate, not the skew.
Read next
Frequently asked questions
- Can I recalculate sample size after an A/B test has already started?
- You can, as long as the recalculation uses only information that does not separate control from treatment. The POOLED conversion rate across both arms is what the clinical trial literature calls a nuisance parameter, and resizing from it is the internal pilot design with blinded re-estimation. In the simulation in this article, with 6,000 replicates per cell, the false positive rate with recalculation landed between 4.37 and 5.08 percent against the nominal 5 percent, while the fixed design landed between 4.77 and 5.12 percent. There was no inflation.
- What is the difference between resizing and peeking?
- Peeking means looking at the difference between arms and deciding based on it, which inflates the false positive rate well beyond 5 percent. Blinded recalculation means looking only at the pooled rate of both arms added together, which under the null hypothesis carries no information at all about the difference. Operationally the rule is strict: whoever does the recalculation receives a single number, total conversions divided by total visitors, and never sees the split by variant.
- What happens if I do not resize and the baseline was wrong?
- You lose power without knowing it. In the example measured here, a plan built for a 5 percent baseline (31,234 users per variant) running on a true baseline of 2 percent delivers 41.4 percent power instead of the planned 80 percent. Nearly 6 out of every 10 tests with a genuine effect end without significance, and the wrong conclusion recorded is that the variant did not work. With blinded recalculation, measured power in the same scenario was 77.8 percent.
- At what point in the test should I recalculate?
- The timing matters less than you would think. We measured recalculation at 10, 25, 50, 75 and 100 percent of the planned sample and final power stayed between 77.7 and 78.9 percent at every point, with false positive rates between 4.85 and 5.57 percent. What changes with timing is the precision of the baseline estimate: its standard deviation fell from 0.2235 percentage point at 10 percent of the sample to 0.0704 at 100 percent. Recalculating early is only a problem when there is not yet enough data to estimate the rate stably.
- Can blinded recalculation shrink the sample, not just grow it?
- It can, and that is where the biggest saving shows up. In the scenario with a true baseline of 8 percent (plan built for 5 percent), the unrestricted design finished at 17,892 users per variant with 77.2 percent power, against 31,234 and 94.9 percent power for the design that never drops below plan. That is 43 percent less traffic to deliver exactly the power the team asked for. If your goal is to protect power and never cut the sample, use the restricted design and accept the surplus.
- Is there any case where blinded re-estimation is NOT safe?
- Yes: non-inferiority and equivalence testing. Glimm and Lauter show that in that context the supposedly blinded recalculation is mathematically equivalent to an unblinded one, because the total variance under the shifted null carries the observed difference between arms. The consequence is a far more serious type I error inflation that does not vanish with larger samples. For ordinary superiority testing, which is the typical A/B test, the violations are extremely small and only appear at tiny sample sizes.