A/B Test Peeking Simulator
Watching an A/B test and stopping the first time it "goes significant" is the most common way to declare a winner that does not exist. This simulator shows, live, how much your real false positive rate grows with every peek. Free, no signup, with the stats explained.
This tool is about a process error, not the math of a single test. If you want the verdict of a test that already ended, use the significance calculator or the p-value calculator. Here the question is different: if I keep looking and stop early, how much does that corrode my error rate? The answer is alarming.
Every simulated test is an A/A: A and B are identical, so EVERY winner is false. The red bar is the share that declares a win by stopping at the first significant peek. The inflation depends on how many times you peek, not on your traffic.
How to use it
- Set how many times you peek at the result over the test (for example, one look per day on a ten-day test = 10).
- Pick the confidence level you use to call a win (95% is the default).
- Say whether the test is two-sided (A or B can win) or one-sided (you only bet B goes up).
- Leave the number of simulated A/A tests at 20,000 (more simulations = a more stable estimate) and read the real false positive.
- Compare the red bar (real when peeking) with the blue one (what you think you have). Click run again to watch the estimate wobble between simulations.
How it works: the simulation behind it
The tool runs thousands of A/A tests: in each one, A and B have exactly the same rate, so there is no real difference. Any "winner" that shows up is, by definition, false. In each simulated test, it looks at the result at equally spaced peeks and counts how many cross the significance threshold on ANY look, applying the "I stopped at the first significant one" stopping rule.
To model the statistics faithfully, we use the classic result of group-sequential testing: in the large-sample limit, the z-value observed over time behaves like a standardized Brownian motion. At each peek at information fraction t, the z equals B(t)/√t, with independent normal increments. It is the same math behind the O'Brien-Fleming and Pocock boundaries.
One counterintuitive consequence falls out of this model: the inflation depends on how many times you peek, not on your traffic. That is why the simulator asks for neither conversion rate nor sample size: with 1,000 or 1,000,000 visitors, ten peeks inflate the false positive in practically the same way.
Worked example (reproduces the default result)
With the values that come prefilled: 10 peeks, 95% confidence, a two-sided test and 20,000 simulated A/A tests (fixed seed).
- Of the 20,000 simulated A/A tests, about 3,920 cross the significance threshold on one of the ten peeks.
- Real false positive: 3,920 / 20,000 = 19.6%.
- You thought it was 5% (the nominal α of 95% confidence).
- Inflation: 19.6% / 5% ≈ ×3.9. That is, about 1 in every 5 tests with no real effect shows a winner that is not there.
That is exactly what the tool shows when you open the page. And the curve is known: the table below (same seed) reproduces Armitage's classic result for equally spaced looks.
| Peeks | Real false positive | Inflation |
|---|---|---|
| 1 (the correct one) | 5.0% | ×1.0 |
| 2 | 8.1% | ×1.6 |
| 3 | 10.7% | ×2.1 |
| 5 | 14.1% | ×2.8 |
| 10 | 19.6% | ×3.9 |
| 20 | 25.3% | ×5.1 |
How to read it, and where it misleads
The number that matters is the gap between the red bar and the blue one: it is the size of the self-deception. A team that peeks every day and stops on the first green day thinks it is wrong 5% of the time, but is wrong nearly 20%. Multiplied by dozens of tests a year, that is a pile of "wins" that do not hold up when you try to repeat the gain in production.
Two honest limits to this simulation. First, it assumes equally spaced peeks and the most naive stopping rule (stop at the first significant look, with no alpha spent on purpose); well-built sequential methods control the error precisely by adjusting the threshold at each look. Second, it is the large-sample scenario (Brownian approximation): with few conversions per peek, the discrete behavior drifts a little from the curve, but the lesson does not change. Peeking and stopping early without correction inflates the false positive, full stop.
How to stop peeking (without losing speed)
The defense is not looking less by willpower, it is taking the decision out of the look.
- Fix the sample and the runtime before you start, with the sample size calculator and the duration calculator, and only read the verdict at the end.
- If you truly must look mid-test (to abort something harmful), use a sequential method that spends alpha on purpose, not the 95% cutoff repeated.
- Want more tests per year without cheating? The lever is traffic and effect size. See what your traffic supports in the test velocity calculator.
- When reading the end of the test, confirm the verdict with the significance calculator instead of trusting the first day that went green.
Frequently asked questions
- What is peeking in an A/B test?
- Peeking is looking at an A/B test's result while it is still running and deciding to stop the moment the p-value crosses the significance threshold. The problem is that every look is a fresh chance for randomness to cross that threshold. When you stop at the first significant look, your real false positive rate ends up far above the 5% you think you have. It is the number one cause of fake wins in experimentation.
- Why does peeking inflate the false positive rate?
- A test at 95% confidence accepts a 5% false positive chance on ONE look. If you look ten times over the test and stop at the first significant one, that is ten chances for noise to cross the line, not one. The looks are correlated (tomorrow's data includes today's), so the inflation is not 10 × 5%, but it grows fast: ten peeks push the real false positive near 20%.
- How many times can I peek without ruining the test?
- In a fixed-horizon analysis, the honest number of decision looks is one: at the end of the planned runtime. Every extra look with a stopping rule raises the false positive. If you truly must look mid-test (to abort a harmful test, say), use sequential methods that spend alpha on purpose, like O'Brien-Fleming or Pocock boundaries, instead of reusing the 95% cutoff at every look.
- Does the inflation depend on my traffic or conversion rate?
- Almost not. What matters is how many times you peek, not the sample size or the base rate. That is why this simulator only asks for the number of looks, the confidence level and the test type: in the large-sample limit, the sequential statistic becomes a standardized Brownian motion, and the false-positive inflation is practically the same for 1,000 or 1,000,000 visitors.
- How do I stop peeking without losing speed?
- Fix the sample and the runtime before you start (use the sample size and duration calculators) and only read the verdict at the end. If you want to run more tests per year without cheating, the right lever is traffic and effect size, not looking earlier. See how many tests your traffic supports in the test velocity calculator.
- Is this simulator reproducible?
- Yes. It runs a Monte Carlo simulation with a fixed seed, so opening the page always shows the same number for the same parameters. The "run again" button changes the seed so you can watch the estimate wobble slightly between runs, which is the natural variation of any simulation. Raising the number of simulated tests makes the estimate more stable.
Keep going
With the cost of peeking clear, size the test with the sample size calculator, plan the runtime with the A/B test duration calculator and read the verdict only at the end with the significance calculator. The full context is in the guide what is A/B testing and the complete CRO guide. For the deep dive, read the peeking problem in A/B testing.