Simulation Based Power Analysis for A/B Tests
Simulation based power analysis: when the closed formula lies, how to validate your simulator and how many replications to run. Four measured cases.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
The sample size formula is not a law of nature, it is a model of your data and your analysis. When the two match, it is dead on: in our test, the prescribed N returned 80.08 percent measured power against a nominal 80. When they do not match, it fails silently: the same N delivers 65.07 percent with three corrected variants and 54.66 percent with account-level assignment. This article shows how simulation based power analysis works, estimating power by counting instead of deriving, how to validate the simulator before trusting it, how many replications are enough, and four measured cases where the formula is right, lies a little, lies a lot, or does not exist. It is part of our complete guide to A/B testing and complements how many visitors an A/B test needs.
What the formula assumes without telling you
The classic two-proportion formula, the one behind every sample size calculator on the market, carries a package of assumptions nobody reads:
- The metric is a proportion, not an average of continuous values.
- Each observation is independent of the others.
- The distribution of the difference in means is approximately normal at the chosen N.
- There is exactly one comparison, between exactly two arms.
- The result will be read once, at the end.
- No adjustment, capping, stratification or covariate enters the analysis.
Most landing page conversion tests satisfy all six. That is why the formula has the reputation it has. The problem is that the moment you most need a trustworthy sample plan, when the metric is revenue, the unit is an account, there are four variants or the reading is sequential, is exactly the moment one or more of those assumptions falls.
Simulation is the universal plan B, and the cheapest one: it needs none of that to be true, because it derives nothing.
Simulation based power analysis: the recipe in five steps
- Pick the data model. Either a parametric distribution (Bernoulli, lognormal, negative binomial) or, better still, resampling from your own historical data, which spares you from choosing a shape.
- Pick the true effect you want to be able to detect. That is the MDE, and it is a business decision, not a statistical one. See minimum detectable effect.
- Run the analysis you will actually run. Same correction, same capping, same unit, same number of looks.
- Record whether it came out significant.
- Repeat and count. The fraction of significant results is the power. Redo everything with a true effect of zero and the same fraction is the false positive rate.
Step zero: calibrate against the known case
Before trusting any exotic number your simulator produces, it has to reproduce a case whose right answer is already known. It is the same principle as an A/A test, applied to the simulation code rather than the assignment system.
Ask the formula for the simplest plan: a 5 percent baseline, a target of detecting plus 10 percent relative, 95 percent confidence, 80 percent power.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
It returns 31,234 per variant. Running 4,000 simulated experiments at that N, with the plus 10 percent effect genuinely present:
| Scenario | Expected | Simulated | Monte Carlo SE |
|---|---|---|---|
| Power with a plus 10% relative effect | 80.00% | 80.08% | 0.63 pp |
| Average observed relative gain | 10.00% | 10.09% | - |
| False positive with zero effect | 5.00% | 5.35% | 0.34 pp |
The simulator reproduces the formula to within a tenth of a percentage point on power. Only after that do the numbers from the hard cases mean anything. If this calibration fails, the problem is in the code rather than the statistics, and the usual suspect is worth checking first: a random generator with state shared across runs, which makes a “fixed” seed produce different results depending on the order the simulations ran in.
How many replications? The one-line answer
Every simulated power estimate is itself an estimate, with error. Since each replication’s outcome is a yes or no, the Monte Carlo standard error is the square root of p times (1 minus p) divided by the number of replications.
| Replications | SE for power near 80% | 95% interval | SE for alpha near 5% | 95% interval |
|---|---|---|---|---|
| 200 | 2.828 pp | plus or minus 5.54 pp | 1.541 pp | plus or minus 3.02 pp |
| 1,000 | 1.265 pp | plus or minus 2.48 pp | 0.689 pp | plus or minus 1.35 pp |
| 5,000 | 0.566 pp | plus or minus 1.11 pp | 0.308 pp | plus or minus 0.60 pp |
| 20,000 | 0.283 pp | plus or minus 0.55 pp | 0.154 pp | plus or minus 0.30 pp |
| 100,000 | 0.126 pp | plus or minus 0.25 pp | 0.069 pp | plus or minus 0.14 pp |
Two practical readings. First, 200 replications are worthless: a power measured at 80 percent could be 75 or 85, and that range covers the difference between a good plan and a bad one. Second, returns fall fast: going from 20,000 to 100,000 replications costs five times the runtime to gain 0.3 percentage point of precision.
Reporting simulated power without the Monte Carlo error is the equivalent of reporting a conversion rate without a confidence interval. Every row in this article comes with its own.
Case 1: the formula is right (binary conversion)
That is the case already shown in the calibration, and it is worth stating the conclusion explicitly because it tends to get lost in articles that only discuss the hard cases: for a conversion rate with two variants and one reading at the end, the closed formula is correct and simulating is a waste of time. It is instant, it is reproducible, it has no Monte Carlo error and it returns the same number 4,000 simulations returned.
Deng, Knoblich and Lu make this point in the general direction: at data scale, the majority of metrics approximately follow normal distributions, which opens the chance to model them directly with closed-form formulas and distributed algorithms at a fraction of the cost of simulation based procedures like bootstrap. Simulation is the tool for when the case goes off the rails, not the default replacement.
Case 2: the formula lies about the interval, not the test (revenue per user)
Here it gets interesting, and the answer is subtler than the folklore suggests.
Our metric: revenue per user, where 5 percent of visitors buy and order value follows a lognormal. The exact moments of that mixture:
| Property | Value |
|---|---|
| Mean | 7.4579 |
| Standard deviation | 49.4463 |
| Coefficient of variation | 6.630 |
| Skewness coefficient | 15.134 |
A skewness of 15 is the territory Kohavi, Deng, Longbotham and Xu cover in rule of thumb number 7. Their rule: the minimum number of independent observations needed for the mean to have a normal distribution is 355 times the square of the skewness coefficient, per variant, and they recommend applying it whenever the absolute skewness exceeds 1. For our metric that gives 81,306 users per variant.
The Bing table the authors publish shows the real order of magnitude of the problem:
| Bing metric | Skewness | Minimum sample by the rule | Sensitivity at 80% power |
|---|---|---|---|
| Revenue per user | 17.9 | 114k | 4.4% |
| Revenue per user (capped) | 5.2 | 9.7k | 10.5% |
| Sessions per user | 3.6 | 4.70k | 5.4% |
| Time to success | 2.1 | 1.55k | 12.3% |
We went and measured. And the result splits in two, which is precisely the kind of nuance only simulation reveals.
The ONE-sample confidence interval breaks badly. Across 20,000 replications per row, we asked how often the 95 percent interval around a single arm’s mean contains the true value of 7.4579:
| Users in the arm | 95% CI coverage | Interval entirely below | Interval entirely above |
|---|---|---|---|
| 100 | 79.83% | 20.08% | 0.09% |
| 300 | 87.43% | 12.29% | 0.27% |
| 1,000 | 92.25% | 7.08% | 0.67% |
| 3,000 | 93.81% | 5.10% | 1.08% |
| 10,000 | 94.56% | 3.90% | 1.54% |
| 30,000 | 95.19% | 2.95% | 1.86% |
| 81,306 (the rule) | 94.77% | 2.88% | 2.35% |
At 100 users, the interval that promises 95 percent delivers 79.83, and the error is almost all on one side: 20 percent of the time the whole interval sits below the true mean, because the sample caught none of the giant orders that make the mean. Kohavi and co-authors record exactly that behaviour, noting that at 100 and 1,000 users the distribution of the sample mean is quite skewed and the two-sided interval assuming normality would miss the true mean more than 5 percent of the time.
And look at the last row. The 355 rule promises an interval in which both tail probabilities, nominally 0.025 each, come out no greater than 0.03 and no less than 0.02. At the N it prescribes, we measured tails of 0.0288 and 0.0235: both inside the promised band. At 30,000 the upper tail drops to 0.0186 and falls out of the band. The rule delivers exactly what it promises, at the N where it promises it.
But the TWO-arm comparison behaves far better. We measured the false positive rate of the two-mean test on the same metric, under a true null, also across 20,000 replications:
| Users per arm | False positive (nominal 5%) | Left tail | Right tail |
|---|---|---|---|
| 100 | 3.18% | 1.57% | 1.60% |
| 300 | 4.31% | 2.33% | 1.98% |
| 1,000 | 4.88% | 2.44% | 2.44% |
| 3,000 | 4.74% | 2.46% | 2.27% |
| 10,000 | 4.91% | 2.54% | 2.37% |
| 30,000 | 4.93% | 2.46% | 2.48% |
At a thousand users per arm, one hundredth of what the 355 rule asks for, the test is already at 4.88 percent. At 100 users it is conservative (3.18 percent), which errs on the safe side. And at the N the closed formula prescribes to detect plus 10 percent relative on that metric (69,005 per variant), simulation returned 78.60 percent power against a nominal 80, and an alpha of 4.50.
The reason for the contrast is that the skew in both arms points the same way and much of it cancels in the difference. That is good news for anyone running A/B tests and terrible news for anyone using a single arm’s confidence interval as if it were the truth about that revenue.
And the cheapest escape is still the one Kohavi and co-authors point to in their own table: capping. Cutting revenue at the 99th percentile of the distribution (the cut lands at 212.19), our metric’s skewness collapses from 15.134 to 5.764, and the formula’s N for the same plus 10 percent relative target falls from 69,005 to 40,840 per variant. With capping, the two-sample false positive rate is already at 4.87 percent with 300 users per arm. The details are in outliers and metric capping.
Case 3: no formula exists (quantile metric)
A percentile metric has no closed-form sample size formula that fits in an article. Deng, Knoblich and Lu record that page loading time quantiles at 75, 95 or 99 percent are among the metrics that get the most attention in the web performance community, and that the two usual routes to a confidence interval, estimating the density or resampling by bootstrap, both have their own problems: the density in the tail decays to zero, so a small bias in the estimate turns into a large bias in the variance, and the bootstrap requires sorting the data repeatedly, which gets expensive at scale.
For sample planning, though, the cost is irrelevant: you run the simulator once, on your laptop, before the test even exists.
We modelled load time as a lognormal with a median of 544.6 ms, a mean of 633.5 ms and a 90th percentile of 1,102.0 ms. The variant cuts the time by 5 percent. The simulated analysis was the one a team would actually run: compute each arm’s p90 and compare with a standard error from 120 bootstrap resamples. 400 replications per row:
| Users per arm | True reduction | Result | Monte Carlo SE |
|---|---|---|---|
| 2,000 | 5% | power 41.75% | 2.47 pp |
| 3,000 | 5% | power 52.75% | 2.50 pp |
| 5,000 | 5% | power 75.50% | 2.15 pp |
| 10,000 | 5% | power 97.00% | 0.85 pp |
| 3,000 | 0% | false positive 5.25% | 1.12 pp |
| 10,000 | 0% | false positive 6.25% | 1.21 pp |
Two things come out of that, and neither would come out of any formula. The right plan to detect 5 percent on p90 sits between 5,000 and 10,000 users per arm, and the curve is steep in that band, so undershooting N is very expensive in power. And the procedure is honest under the null (5.25 and 6.25 percent, both within two Monte Carlo standard errors of nominal), which validates the analysis method itself and not just the plan.
More on the family in quantile metrics.
Case 4: the formula lies by omission (three variants)
This is the most dangerous case, because everything looks right. You open the calculator, it returns 31,234 per variant for 80 percent power, you multiply by 4 arms to get total traffic and go to sleep. Except the calculator answered about one comparison, and your test has three.
We simulated 1,500 programs of one control against three variants, where one variant has a real plus 10 percent effect and the other two are null, applying Bonferroni over the three comparisons:
| Scenario | Detects the true variant | Declares some winner | False positive on a null |
|---|---|---|---|
| n = 31,234, with Bonferroni | 65.07% | 65.13% | 1.33% |
| n = 31,234, no correction | 80.07% | 80.07% | 4.27% |
| n = 43,000, with Bonferroni | 82.00% | 82.00% | 1.13% |
| n = 31,234, no true winner, with Bonferroni | - | 1.73% | 1.73% |
| n = 31,234, no true winner, no correction | - | 5.33% | 5.33% |
Power falls from 80.07 to 65.07 percent with nothing changing about the experiment except the number of arms. To get back to 80, N has to rise from 31,234 to roughly 43,000 per variant, a 38 percent increase in each arm, four times over. That is the toll for comparing more things at once, and it is invisible in the calculator.
Note the right-hand column: the correction works. Without Bonferroni, 5.33 percent of programs with no true winner declare a false one; with Bonferroni, 1.73. More in testing multiple variants.
Case 5: the formula is off by 25 points (cluster assignment)
The largest deviation we measured did not come from an odd metric, it came from the randomization unit. When the experiment assigns the account and the analysis counts the user, the plan of 31,234 per variant, which promises 80 percent, delivers 54.66 percent power with 5 users per account and an intracluster correlation of 0.20, and 38.06 percent with 10 users per account.
The formula has no idea, because it has no way of knowing: nothing on the calculator’s form asks how assignment was done. Simulation knows, because the data model you write includes the account structure. We cover the whole case, with the design effect correction, in cluster randomization.
Four traps for people who simulate
Simulating the model instead of the data. If you pick a lognormal because it is pretty and your real revenue is not lognormal, the power you get describes the lognormal. The defence is resampling: take a month of real per-user data, draw with replacement to build both arms, and apply the effect by multiplying the treated arm’s metric. That preserves the real shape of your distribution, including the parts you do not know are there.
Simulating a different analysis from the one you will run. This is the most common error and the most expensive. If the simulator runs a plain z test and the team will check the dashboard every Monday, the simulated power describes a test that is not yours. The same goes for capping, outlier exclusion, bot filters and covariate adjustment: everything in the real analysis has to be in the simulation. See the peeking problem.
A seed that is not fixed, or worse, fixed only in appearance. A generator with global state shared across runs makes the same seed return different results depending on the order the simulations ran in. The symptom is a number that will not reproduce when you run the script on its own. Every table in this article was generated with an explicit per-row seed and checked by running in isolation.
Reporting power without the Monte Carlo error. Said above and repeated here because it is what most often vanishes from the final report. A 2 percentage point difference between two plans means nothing if the simulation error is 2.8.
When NOT to simulate
Simulation costs writing time, running time and one more artefact to maintain. It is not worth it when:
- The metric is binary, there are two arms and a single reading. Use the formula.
- You need an answer in thirty seconds in a meeting. Use the formula and note the doubt.
- The goal is a confidence interval for a quantile metric in production, every day, at scale. There the Deng, Knoblich and Lu recommendation holds: a closed-form formula, like the outer interval they propose, costs a fraction of the bootstrap.
The rule of thumb: simulate to plan, use the closed formula to operate. The plan runs once and can take minutes; the reading runs every day and has to be cheap.
Make this automatic with Donnu
The reason almost nobody does simulation based power analysis is not conceptual difficulty, it is that it demands writing throwaway code with nowhere to live, then redoing all of it when the plan changes. At Donnu the sample plan is computed both ways and shows both side by side: the closed-formula number and the number that comes from simulating your analysis, with your own history resampled instead of an invented distribution. When the experiment design includes something the formula does not cover, more than two variants, account-level assignment, capping, sequential reading, the dashboard tells you what that detail costs in power before you turn the test on, with the Monte Carlo error declared alongside.
Frequently asked questions
The short answers live in the FAQ section of this page, built from the same calculations presented here.
References
- Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu. Seven Rules of Thumb for Web Site Experimenters, KDD 2014. Rule 7 is the source of the rule of thumb of 355 times the square of the skewness coefficient as the minimum number of independent observations per variant for the mean to be normally distributed, of the recommendation to apply it when absolute skewness exceeds 1, of the definition of the skewness coefficient used here, of the Bing table (revenue per user at skewness 17.9 and a sample of 114k for 4.4 percent sensitivity; capped revenue at 5.2 and 9.7k for 10.5 percent; sessions per user at 3.6 and 4.70k for 5.4 percent; time to success at 2.1 and 1.55k for 12.3 percent), of the promise that both tail probabilities land between 0.02 and 0.03, of the record that the rule derives from the work of Boos and Hughes-Oliver, and of the observation that at 100 and 1,000 users the distribution of the sample mean is quite skewed and the two-sided interval assuming normality would miss the true mean more than 5 percent of the time.
- Alex Deng, Ulf Knoblich and Jiannan Lu. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas, KDD 2018. Source of the observation that at scale the majority of metrics approximately follow normal distributions, which allows modelling them with closed-form formulas and distributed algorithms at a fraction of the cost of simulation based procedures like bootstrap; and from section 4, of the record that page loading time quantiles at 75, 95 and 99 percent are central metrics for the web performance community, of the criticism of tail density estimation (the density appears in the denominator and decays to zero, so a small bias becomes a large bias in the variance) and of the record that bootstrap works well at small scale but gets expensive at large scale because computing quantiles requires repeated sorting.
- Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi and Nathaniel T. Stevens. Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology, arXiv 2212.11366, 2024. Used here as an overview of the designs that fall outside the case the closed formula covers, including the record that under cluster randomization the effective sample size and hence power is dramatically reduced, which is the scenario of case 5.
Read next
Frequently asked questions
- What is simulation based power analysis?
- It is estimating a test's power by generating thousands of fake experiments with an effect you chose yourself, running in each one exactly the analysis you will run for real, and counting the fraction that come out significant. No formula is involved: the number comes from counting. That lets you compute power for any metric and any analysis, including ones with no closed form at all.
- When does the closed sample size formula stop holding?
- It holds very well for the case it was built for: comparing two proportions with a single look. Our simulation returned 80.08 percent power at the N the formula prescribed, against a nominal 80. It stops holding when the analysis changes: with three variants and a Bonferroni correction the same N delivers 65.07 percent, and with account-level assignment and an intracluster correlation of 0.20 it delivers 54.66.
- How many replications do I need?
- It depends on the precision you want. The Monte Carlo standard error of a power estimate near 80 percent is the square root of 0.16 divided by the number of replications. At a thousand replications the interval is plus or minus 2.48 percentage points, at 20,000 it is 0.55 and at 100,000 it is 0.25. For a planning decision, 2,000 to 10,000 replications usually suffice; for publishing a table, go higher.
- How do I know my simulator is not broken?
- By calibrating it against a case where the right answer is already known. Run the simulator on the simplest scenario, two proportions and a single look, at the N the closed formula prescribes: it must return 80 percent power and 5 percent false positives. Ours returned 80.08 and 5.35. If that case does not close, no exotic result from the same code deserves any trust.
- Does revenue per user skew break the A/B test?
- It breaks the one-sample confidence interval, not the two-arm comparison. In our simulation with a skewness of 15.13, the 95 percent interval for a single arm's mean covered the true value only 79.83 percent of the time at 100 users, but the comparison between the two arms held false positives at 4.88 percent from a thousand users per arm onward. Much of the skew cancels between the two sides.
- Does simulation replace the closed formula?
- No, it complements it. For a binary metric with a single look, the formula is instant and correct, and simulating would be a waste. Deng, Knoblich and Lu note that resampling based procedures get expensive at scale, and propose closed-form formulas precisely as the alternative. The practical rule is to use the formula when the case is the formula's and simulate when it is not.