Sequential Testing in A/B: Peek Anytime, Explained (2026)
Sequential testing A/B: how always-valid inference (mSPRT) lets you peek at results anytime without inflating false positives, and when it pays off.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Sequential testing, also called always-valid inference, is a family of statistical methods, most notably mSPRT (mixture Sequential Probability Ratio Test), that lets you look at an A/B test result as many times as you want, at any moment, while still keeping the real false-positive rate at the level you agreed to, usually 5%. It is the actual statistical answer to the peeking problem: rather than forbidding you from checking the dashboard before the end (the fixed-sample solution), sequential testing redesigns the decision math itself so that looking repeatedly stops being a mistake.
This guide walks through what sequential testing genuinely delivers, how mSPRT gets there without reinventing statistics from scratch, what it costs in exchange for that freedom, and when adopting a method like this is worth more than simply computing this blog’s fixed sample size and having the discipline not to peek.
The problem sequential testing solves
Every classic A/B test carries an implicit promise: a p-value below 0.05 means a 5% chance you are celebrating pure noise. That promise only holds under one very specific condition almost nobody says out loud: you decided the sample size in advance and looked at the result exactly once, at the end. Every extra look before that point, combined with the temptation to stop the moment the number looks good, is another chance for pure chance to cross the significance line for an instant.
Our guide to the peeking problem walks through the math behind this in detail, citing Evan Miller’s analysis: a test designed for a 5% false-positive rate, monitored continuously and stopped at the first apparent win, can deliver a real error rate near 26% (Miller, 2010). The chart below reproduces that inflation curve, and sets next to it how a real sequential test behaves, staying under control the entire time, not because you resist the temptation to look, but because the method was built mathematically to withstand it:
How sequential testing actually works
The core idea is not “check more carefully,” it is swapping the question the statistics answer. A fixed-horizon test asks: “at this specific N, decided in advance, is the observed difference too large to be chance?” A sequential method asks something more ambitious: “at any point in time I might want to stop, is the difference observed so far too large to be chance, given that I might be looking again an hour from now?” Answering that second question in a mathematically honest way requires a different statistic from the single-look p-value.
The method most used in the industry for this is mSPRT (mixture Sequential Probability Ratio Test), the foundation of what Optimizely calls its Stats Engine and what Statsig and GrowthBook offer as “sequential testing.” The mechanics, in three pieces:
- A mixture likelihood ratio. Instead of testing against a single hypothetical effect size, the method computes the likelihood ratio of the data observed so far, mixed (a weighted average) over an entire distribution of possible effect sizes. That mixture is where the method gets its name, and it is what guarantees good behavior regardless of what the real effect turns out to be, which you do not know in advance.
- A process that never grows past a fixed ceiling under the null hypothesis. If A and B really are equal, this mixture-likelihood process has, in expectation, a mathematical ceiling it does not cross, no matter how long you let it run. That property, formalized by Ville’s inequality, is what underpins “always valid”: the probability of this process crossing a threshold at any moment, across the entire life of the test, is already controlled at 5% from the start, instead of 5% per look.
- A p-value (or confidence interval) recomputed after every new data point, always interpretable. Unlike the classic p-value, which is only valid at the planned N, the sequential p-value can be read at any point in the test. It tends to be more conservative early on, harder to declare significance at the start, precisely because it already reserves part of the error budget for the future looks that have not happened yet.
An important side effect of this mechanism: the confidence intervals and the “always-valid p-values” of a sequential method tend to be wider and more conservative than a fixed-horizon p-value computed on the same N, especially early in the test. That is not a flaw, it is the explicit price of the reading staying valid at any future point. Statsig documents this trade-off with real numbers from its own 28-day test run with multiple checks: mSPRT held the false-positive rate at about 1.1% (below the nominal 5% ceiling), versus 5.0% for a disciplined fixed-horizon single look and 17.3% for a classic z-test checked repeatedly with no correction at all, at the cost of overall statistical power around 84% of the equivalent fixed-horizon test’s power (Statsig, 2025).
“Stop whenever you want”: what that promise covers, and what it doesn’t
It is worth clearing up a common misunderstanding before moving on: sequential testing is not “look whenever you feel like it and stop thinking about sample size.” It trades a rigid rule (never look before N) for a more flexible one, but a rule nonetheless: the reading at every point in time is honest, only it reflects everything that could still happen afterward. The table below lines up the three scenarios that already showed up in the charts above:
| Approach | When you can stop | Real false-positive rate | What it demands of you |
|---|---|---|---|
| Fixed horizon | Only at the N calculated in advance, never before | 5% (the promise holds), as long as you don’t peek and decide early | Discipline: calculate the sample up front and don’t touch the decision until you get there |
| Naive peeking | Whenever it “looks good” | Climbs from 5% to ~14% (5 looks) up to ~26% (continuous) | Nothing, which is exactly why it is dangerous, it feels free |
| Sequential (mSPRT) | At any moment, with a valid reading | Capped at the nominal level by construction; ~1.1% measured in practice (Statsig, 2025) | A tool purpose-built for it; slightly lower power, wider confidence intervals early in the test |
Both fixed horizon and sequential arrive at an honest error rate. The difference is the price each one charges: fixed horizon charges patience (you cannot act before N, full stop); sequential charges a bit of statistical power and extra sample, in exchange for real freedom to act early when the effect is large enough to have already crossed the stricter threshold the method imposes at the start of the test.
The counterpoint: the classic fixed-sample test
To see the trade-off with real numbers, it is worth running this blog’s canonical scenario through the traditional fixed-sample engine. With a 5% baseline rate, a 10% relative minimum detectable effect, 95% confidence and 80% power, the fixed-horizon calculation returns 31,234 visitors per variant, which works out to about 44 days running at 10,000 weekly visitors. Adjust the numbers for your own case:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
That is the deal a fixed-sample test makes: you calculate that N once, before launch, and commit to deciding nothing until you reach it. No significance calculator, no recalculation, no “just a quick peek” halfway through counts as a decision.
Now picture that on day 15 of that 44-day test, about a third of the way through, with 10,411 visitors per variant already collected, you open the dashboard out of curiosity and see the control (A) with 521 conversions and the variant (B) with 614. Paste those numbers into the calculator below:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Running those numbers through this blog’s same engine gives a rate for A of 5.00%, a rate for B of 5.90%, and a p-value of approximately 0.0045, clearly below 0.05. An observer who only looked once, on day 44, would trust a reading like that without hesitation. But this reading showed up on day 15, a third of the way through, and a fixed-horizon test does not give you permission to act on it, because the 0.0045 p-value was computed as if you had looked only that one time, when in reality this is one of several possible looks over the course of the test. It is exactly this kind of early, tempting, seemingly solid reading that feeds the inflation curve from the first chart in this guide, a meaningful slice of these midway “signals” does not survive to the full N.
A real sequential method would treat that same day-15 data differently: it would compare the accumulated mixture likelihood ratio against a threshold deliberately stricter than 0.05 at that specific point, because part of the 5% error budget is already reserved for the looks still to come between day 15 and day 44. If the reading clears that higher bar, the decision is valid right there, with no need to wait for the rest. If it does not clear the bar, the method simply keeps going, without that counting against you, and without you having to pretend you never looked.
When sequential testing is worth it (and when fixed horizon is already enough)
The right question is not “which method is more modern,” it is “what does your operation actually need.” The two columns below summarize the most common cases:
| Scenario | Prefer sequential testing | Prefer fixed horizon + this blog’s calculator |
|---|---|---|
| Cost of waiting for the full N | High: every extra day running a losing variant costs real revenue right now | Low: the product can wait out the calculated deadline without issue |
| Stakeholders asking for status | Frequent, with real pressure to act early if the signal is strong | Rare, or the culture already accepts “still running” as an answer |
| Tooling available | A platform that already implements mSPRT or an equivalent (Optimizely, Statsig, GrowthBook) | Any tool, including a simple significance calculator |
| Discipline not to peek at the dashboard | Hard to guarantee in practice, the dashboard updates itself and everyone sees it | High: the team can genuinely treat the calculated N as untouchable |
| Expected effect size | Large and likely to show up well before the full N | Small or uncertain, requiring the entire sample anyway |
In practice, most teams with average traffic and a “we calculate the sample and wait” culture do not need mSPRT: this blog’s sample size calculator, followed by real discipline, already delivers the same honest 5%, for free, without requiring a new platform. Sequential testing earns its investment when the cost of waiting is high enough, or the temptation to peek is strong enough, that the freedom to decide early with validity is worth the statistical power it costs.
Automate This on Donnu
The fear behind this guide is real: you peek at your test’s dashboard, the number looks good, and the doubt becomes “is this real, or will acting now invalidate the result?” A full mSPRT implementation would settle that doubt formally at any moment; Donnu settles the same fear through the path already available today, with no separate statistical research platform required: it calculates your sample size and duration in advance, keeps that decision locked until N is reached, and uses a native Bayesian engine for an honest reading the moment you get there, with no p-value theater and no quiet temptation for “just one more peek.”
Start a free 14-day trial and run your next test knowing exactly when, and why, you can trust the number you are looking at. For the full picture of the problem sequential testing solves, see our guide to the peeking problem, for the statistical significance foundation behind the fixed-horizon test, see what statistical significance means in A/B testing, and for another Bayesian technique that also shortens a test’s wait, see our guide to CUPED and variance reduction.
Leia em português: teste sequencial explicado.
References
- Johari, R., Pekelis, L. & Walsh, D.J. Always Valid Inference: Bringing Sequential Analysis to A/B Testing. arXiv:1512.04922. arxiv.org/abs/1512.04922.
- Johari, R., Koomen, P., Pekelis, L. & Walsh, D. Peeking at A/B Tests: Why It Matters, and What to Do About It. KDD 2017. doi.org/10.1145/3097983.3097992.
- Statsig. Sequential Testing on Statsig (mSPRT, empirical false-positive and power data). 2025. statsig.com/blog/sequential-testing-on-statsig.
- Optimizely. Fixed horizon, Bayesian, or sequential: Choosing a stats engine. optimizely.com/insights/blog/how-to-choose-the-right-stats-engine.
- Miller, E. How Not To Run An A/B Test (the source of the ~26% real false-positive rate figure under continuous peeking). 2010. evanmiller.org/how-not-to-run-an-ab-test.html.
- Miller, E. Simple Sequential A/B Testing. 2015. evanmiller.org/sequential-ab-testing.html.
- Armitage, P., McPherson, C.K. & Rowe, B.C. Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society, Series A, 132(2), 235-244, 1969.
Frequently asked questions
- What is sequential testing (always-valid inference) in A/B testing?
- It is a family of statistical methods, most notably mSPRT (mixture Sequential Probability Ratio Test), built so you can look at an A/B test result at any moment and act on it, without inflating the real false-positive rate past the level you agreed to, usually 5%. Instead of computing a single p-value meant for one look at the end, the method computes, after every new data point, an "always-valid" reading that already reserves error budget for every future moment you might decide to stop.
- Is sequential testing the same thing as Bayesian testing?
- No, they answer different questions. Frequentist versus Bayesian is about how you express and interpret evidence (a p-value and a confidence interval on one side, the probability that B beats A on the other). Fixed-horizon versus sequential is about when you are allowed to stop validly. You can run a Bayesian test on a fixed horizon (decide the sample up front, look only at the end), and you can run a frequentist sequential test (mSPRT), which is in fact the more common setup in the industry today.
- Does sequential testing remove the need to calculate a sample size before starting?
- It reduces the dependence, but does not remove it entirely. You still benefit from estimating a plausible effect range and expected traffic to know whether the test is viable at your volume. What changes is the rigidity: instead of a fixed N you are forbidden from checking before reaching, the method already builds a valid stopping rule into every new batch of data, so an early look stops being a mistake.
- Do I need special software to run sequential testing, or can this blog's calculator do it?
- You need a tool purpose-built for it. This blog's significance calculator uses the classic two-proportion z-test, which assumes a single look decided in advance, exactly the fixed-horizon scenario. It is great for reading a test you committed not to peek at, but running it repeatedly over the same test and stopping at the first favorable reading recreates the peeking problem; it is not the same thing as a real mSPRT.
- Does sequential testing cost more sample than a fixed-horizon test?
- Generally yes, a bit. According to Statsig, in a real 28-day test monitored repeatedly, sequential mSPRT held the false-positive rate at around 1.1% (below the nominal 5% ceiling) with statistical power close to 84% of an equivalent fixed-horizon test, and it reached significance in 58% of the cases that would end up significant, already at the halfway point of the planned duration, versus 33% for Statsig's own earlier sequential method at that same point. The trade is a bit less power and slightly wider confidence intervals, in exchange for the ability to decide at any moment with real validity.