Testing Multiple Variants: A/B/n Without False Positives
How testing multiple variants inflates false positives, what Bonferroni and Sidak corrections cost you in traffic, and when a four-arm test is worth it.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Testing multiple variants means every variation you add is another chance to be wrong, and the arithmetic is not gentle: with three variations against a control at the usual 5 percent threshold, the probability of crowning at least one false winner when nothing actually works is 14.26 percent. Most teams that run A/B/n tests never apply a correction, read the smallest p-value in the table, and ship it. This guide covers exactly how much the false positive rate inflates, what Bonferroni and Sidak cost you in traffic and calendar time, a worked example where a result that looks significant is not, and how to decide whether a multi-arm test is worth running at all. It is part of our complete guide to A/B testing and builds on A/B testing statistical significance.
Why multiple variants cost you more than traffic
A p-value threshold of 5 percent is a promise about one comparison: if there is truly no difference, you will call a winner about 5 percent of the time anyway. Make the same promise three times and the promises do not add up the way intuition suggests. The chance that at least one of them fails is 1 minus the chance that all of them hold, which is 1 minus 0.95 to the power of the number of comparisons.
| Variations against control | Chance of at least one false winner | Bonferroni threshold | Sidak threshold |
|---|---|---|---|
| 1 | 5.00 percent | 0.05000 | 0.05000 |
| 2 | 9.75 percent | 0.02500 | 0.02532 |
| 3 | 14.26 percent | 0.01667 | 0.01695 |
| 4 | 18.55 percent | 0.01250 | 0.01274 |
| 5 | 22.62 percent | 0.01000 | 0.01021 |
| 6 | 26.49 percent | 0.00833 | 0.00851 |
| 8 | 33.66 percent | 0.00625 | 0.00639 |
The middle column is what statisticians call the family-wise error rate. The last two columns are the two standard ways to pull it back to 5 percent.
Johari, Pekelis and Walsh state the two error definitions cleanly in their work on always valid inference: the family-wise error rate is the worst-case probability of incurring any false positive, while the false discovery rate is the worst-case average proportion of false positives among the hypotheses you rejected. They also name the standard procedures: Bonferroni for the family-wise rate, Benjamini-Hochberg for the false discovery rate. Which one you want depends on the decision. If a single false winner would be expensive to ship, control the family-wise rate. If you are scanning many metrics and want good performance on balance, controlling the false discovery rate is the more sensible target.
Bonferroni and Sidak, in one line each
Bonferroni divides your threshold by the number of comparisons. Three variations, 5 percent target, each comparison must beat 0.01667. It assumes nothing about how the comparisons relate to each other, which is why it is slightly conservative and why it is a good default.
Sidak assumes the comparisons are independent and solves for the threshold that yields exactly 5 percent overall: 1 minus 0.95 raised to the power of one over the number of comparisons. For three comparisons that is 0.01695.
The gap between them is 0.0003. In practice the choice between the two almost never changes a decision, so pick one, write it in your experiment record, and stop debating it. What does change decisions is whether you correct at all.
Worked example: the winner that a correction removes
A SaaS team tests three new pricing page layouts against the current one. Traffic is split four ways, 18,000 visitors per arm, and the metric is trial starts.
| Variation | Visitors | Conversions | Rate | Lift | raw p-value | adjusted p-value | Verdict | Remove variation |
|---|---|---|---|---|---|---|---|---|
| - | - | - | - | - | ||||
| - | - | - | - | - | ||||
| - | - | - | - | - |
Testing several variations against the same control multiplies the chance of a false positive. The correction lowers the per-comparison threshold to hold the error of the whole family. Compare the raw p-value (what a two-variation calculator would show) with the adjusted one: a variation often passes on its own and falls after correction.
Here is the scoreboard.
| Arm | Visitors | Trial starts | Rate | Relative lift | Raw p-value | Adjusted p-value |
|---|---|---|---|---|---|---|
| Control | 18,000 | 720 | 4.000 percent | reference | reference | reference |
| B | 18,000 | 803 | 4.461 percent | +11.53 percent | 0.0298 | 0.0893 |
| C | 18,000 | 742 | 4.122 percent | +3.06 percent | 0.5569 | 1.0000 |
| D | 18,000 | 707 | 3.928 percent | -1.81 percent | 0.7255 | 1.0000 |
Bonferroni with three comparisons, adjusted p-value is the raw p-value multiplied by three and capped at 1.
Variant B is the moment of truth. Its raw p-value of 0.0298 is comfortably below 0.05, its lift is +11.53 percent, and its confidence interval on the difference is +0.045 to +0.877 percentage points, which does not include zero. Every instinct in the room says ship it.
The corrected threshold is 0.01667, and 0.0298 does not clear it. Put the other way, the adjusted p-value is 0.0893. Sidak gives essentially the same answer, 0.0866. The correct reading is that B is the most promising of the three and the test does not confirm it.
For contrast, an effect that is genuinely large survives without argument. Had B landed at 855 trial starts (4.750 percent, a +18.75 percent lift), the raw p-value would be 0.0005 and the adjusted p-value 0.0015, clearly significant under any of these corrections. And the boundary is not far away: B would have needed 812 trial starts, a rate of 4.511 percent and a lift of +12.78 percent, for the raw p-value to reach 0.0163 and clear the corrected bar. Nine extra conversions out of 18,000 visitors separate “promising” from “confirmed” here, which is a useful reminder of how thin the evidence is at the edge of any threshold.
What the correction costs in traffic and calendar
The correction does not only make the bar higher at analysis time. If you plan the test properly, it raises the sample you need before you launch, because a stricter threshold demands more evidence for the same detectable effect. And you are paying for more arms at the same time.
| Variations | Corrected threshold | Visitors per arm | Total visitors | Days at 25,000 per week |
|---|---|---|---|---|
| 1 (plain A/B) | 0.05000 | 17,943 | 35,886 | 11 |
| 2 | 0.02500 | 21,729 | 65,187 | 19 |
| 3 | 0.01667 | 23,933 | 95,732 | 27 |
| 5 | 0.01000 | 26,699 | 160,194 | 45 |
Sizing at a 4 percent baseline, 15 percent relative minimum detectable effect, 80 percent power, two-sided, with the Bonferroni-corrected threshold.
Two things stand out. The per-arm cost of the correction is modest: going from one comparison to three raises the sample per arm by only about 33 percent, because sample size responds to the threshold through the square of a critical value rather than linearly. The expensive part is the arms themselves. Three variations turns 35,886 visitors into 95,732, which is 2.7 times the traffic, and eleven days into twenty-seven.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
That ratio is the real decision. A four-arm test is not “one test that answers three questions”, it is close to three tests run in parallel with a shared control, and it should be justified the same way. The minimum detectable effect you can afford in a four-arm design is meaningfully worse than in a two-arm one at the same calendar length, and pretending otherwise is how programs end up with a drawer full of inconclusive multi-arm tests.
The multiplicity you did not notice you had
Variations are the visible source of multiple comparisons. There are three more that are just as real and much easier to miss.
Metrics. Reading five metrics at 5 percent is arithmetically identical to five variations: about a 22.62 percent chance that one crosses the line by accident. The fix is not usually a correction, it is declaring one primary metric before launch and letting it carry the decision.
Segments. Slicing a flat result by device, channel, country and new versus returning generates a dozen comparisons in a few clicks, and at least one will look significant. Segment analysis is valuable for generating the next hypothesis and is close to worthless as confirmation, unless the segment was named before launch.
Time. Checking the result every day for two weeks is fourteen looks, and it inflates the false positive rate the same way. This is the peeking problem, and it stacks on top of everything above rather than replacing it.
Sequential rounds. Running the same four ideas as four separate A/B tests and shipping the best one does not escape the multiplicity. The comparisons still happened; they were just spread over the calendar. The honest rule is that the correction follows the number of comparisons that fed the decision, not the number of rows in your test log.
At Bing, the interaction-detection system had to run hundreds of thousands of hypothesis tests, and Kohavi and colleagues report using an empirical Bayesian false discovery rate control algorithm to keep the false positive count manageable. The lesson for a smaller program is not to copy the algorithm, it is to notice that anyone testing at scale ends up needing an explicit policy about multiplicity, and that the policy has to be written down rather than improvised per test.
When a multi-arm test is the right call
| Situation | Better design |
|---|---|
| Four genuinely different value propositions, high traffic | A/B/n with a correction, sized for the corrected threshold |
| Four small variations of the same idea | Pick the best one and run a two-arm test; the effects are too small to pay the corrected bill |
| One idea, several metrics you care about | Two-arm test with one declared primary metric and the rest as context |
| Low traffic and several ideas | Sequential two-arm tests, one at a time, biggest bet first |
| Many arms and you want traffic to flow to what is working | A bandit, which optimizes total conversions rather than producing a clean per-arm verdict |
The last row deserves a caveat, because it is often oversold. A multi-armed bandit is a different objective, not a cheaper A/B/n: it maximizes conversions during the test rather than delivering a clean estimate of each arm’s effect. If you need a defensible number for a decision that outlives the test, a controlled experiment with a correction is what produces it.
Common mistakes with A/B/n tests
| Mistake | What it produces |
|---|---|
| Reporting the smallest p-value with no correction | A false winner roughly one time in seven with three variations |
| Correcting at analysis but sizing without it | An underpowered test that almost never clears the corrected bar |
| Adding a fourth variation “since we are testing anyway” | Every other arm loses power and the calendar grows by weeks |
| Dropping losing arms mid-test and re-reading | Multiplicity plus peeking, with no valid error rate left |
| Treating a corrected non-winner as a failed idea | Variant B above is a good candidate, not a dead one |
| Reading segments after the fact as confirmation | A dozen silent extra comparisons and a story that will not replicate |
Automate this with Donnu
A correction only protects a program when it is applied by default rather than remembered. Donnu A/B counts the comparisons in the test, applies the correction to the threshold before showing any verdict, and reports the adjusted p-value next to the raw one so nobody has to reconstruct which bar a result actually cleared. The sample size it recommends at design time already accounts for the number of arms, so a four-way test is not planned as if it were a two-way one.
Start a free 14-day trial and size your next multi-arm test against your own baseline.
References
- Johari, R., Pekelis, L. and Walsh, D.J. Always Valid Inference: Continuous Monitoring of A/B Tests. arXiv:1512.04922. Source of the family-wise error rate and false discovery rate definitions used here, and of the description of Bonferroni and Benjamini-Hochberg as the standard fixed-horizon procedures. arxiv.org/abs/1512.04922.
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the account of empirical Bayesian false discovery rate control across hundreds of thousands of hypothesis tests at Bing. exp-platform.com.
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. On sustained effects and the discipline of deciding what counts as a result before running. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
Read also: What is A/B testing · A/B testing statistical significance · Minimum detectable effect · A/B vs multivariate vs split URL testing · Free A/B/n significance calculator · Leia em português
Frequently asked questions
- What is an A/B/n test?
- An A/B/n test compares a control against more than one variation at the same time, splitting traffic across all arms. The mechanics are identical to an A/B test, but the statistics are not: each additional variation is another comparison against control, and each comparison carries its own chance of a false positive. With three variations tested at a 5 percent threshold, the probability that at least one produces a false winner when nothing works is about 14.26 percent, not 5 percent. That is the whole reason A/B/n needs a correction and a bigger sample.
- How much do multiple variants inflate false positives?
- The family-wise error rate, meaning the chance of at least one false positive across the whole test, is 1 minus 0.95 raised to the number of comparisons. One comparison gives 5 percent, two gives 9.75 percent, three gives 14.26 percent, five gives 22.62 percent and eight gives 33.66 percent. At eight variations you have a one in three chance of crowning a winner that does not exist, even when every variation is identical to control. Nothing about the individual p-values is wrong; the problem is reading many of them and reporting only the smallest.
- What is the Bonferroni correction and when should I use it?
- Bonferroni divides your significance threshold by the number of comparisons. With three variations and a 5 percent target, each comparison must beat 0.0167 rather than 0.05. It is the simplest correction, it is conservative, and it is the right default when a single false winner would be expensive to ship. The Sidak correction is a slightly less strict alternative that assumes independence: for three comparisons it gives 0.01695 instead of 0.01667, a difference too small to change most decisions.
- How much extra traffic does an A/B/n test need?
- Two multipliers stack. First, the corrected threshold raises the sample per arm: at a 4 percent baseline and a 15 percent relative minimum detectable effect, one comparison needs 17,943 visitors per arm and three comparisons need 23,933. Second, you now have four arms instead of two. Total traffic goes from 35,886 visitors to 95,732, which is about 2.7 times the cost. At 25,000 visitors per week that is 27 days instead of 11.
- Should I just run separate A/B tests instead?
- Running the same comparisons sequentially does not make the multiplicity go away, it only hides it, because you are still choosing the best of several results. What sequential tests do buy you is the chance to learn between rounds and to drop bad ideas cheaply. The honest rule is that the correction follows the number of comparisons you make on the way to a decision, not the number of tests on the calendar. If you genuinely want the fastest path with several ideas, an A/B/n test with a correction is usually cleaner than four sequential tests read without one.
- Do I need a correction for multiple metrics too?
- Yes, and this is the version teams forget. Reading five metrics at a 5 percent threshold gives about a 22.62 percent chance that at least one crosses the line by accident, exactly the same arithmetic as five variations. The standard fix is to declare one primary metric before launch that carries the decision, and treat the rest as context rather than as evidence. When you truly must decide on several metrics at once, apply a correction to those comparisons the same way.
- When is testing more than two variants actually a good idea?
- When the variations are genuinely different bets rather than small tweaks of each other, when your traffic can pay the bill for the corrected sample, and when losing is informative. Four distinct value propositions on a pricing page is a good A/B/n. Four shades of the same button is not, because the effects are small, the corrected sample is enormous and the answer changes nothing you will do next. If your traffic cannot support the corrected design, testing two ideas well beats testing four badly.