A/B/n test significance calculator (multiple variants)

This tool is for tests with more than one variation (A/B/n). If you only compare a single variation B against control A, use the statistical significance calculator, which is more direct. Here the problem is different: each extra variation raises the chance a result looks like a winner purely by luck, and that is exactly what the Bonferroni or Šidák correction fixes.

How to use it

1. Fill in the control (A): visitors and conversions of the original version.
2. Fill in each variation (B, C, D...) with its visitors and conversions. Use the button to add as many as you need.
3. Pick the correction (Bonferroni is the safe default) and the confidence (95% is the convention).
4. Read the verdict at the top and the per-variation table: rate, lift, raw p-value, adjusted p-value and whether each one wins.
5. Compare the raw p-value with the adjusted one: that is where the correction shows its work.

How it works: the formula

Each variation is compared against the control with the same two-proportion z-test as the plain significance calculator. What changes is the threshold. Call m the number of variations (comparisons). The correction lowers the alpha accepted per comparison:

A variation wins when its raw p-value falls below that corrected threshold. Equivalently, the tool also shows the adjusted p-value (Bonferroni: p × m; Šidák: 1 − (1 − p)^m), which you compare directly against the original 5% alpha. Both readings give the same verdict.

Worked example (reproduces the default result)

With the values already filled in, control A = 8,000 visitors / 400 conversions (5.00%) and three variations, Bonferroni at 95%, so m = 3 and the per-comparison threshold becomes α_adj = 0.05 / 3 = 1.667%:

• B (8,000 / 520 = 6.50%, +30.0%): raw p-value < 0.0001, adjusted 0.0001. Below 1.667%, so it wins with significance.
• C (8,000 / 462 = 5.78%, +15.5%): raw p-value 0.0299, adjusted 0.0898. The raw value would pass on its own at 95% (0.0299 < 0.05), but it sits above the corrected threshold (0.0299 > 0.0167), so it does not win. That is the false positive the correction holds back.
• D (8,000 / 416 = 5.20%, +4.0%): raw p-value 0.5653, adjusted 1.0000. Far from any threshold, not significant.

Result: the winner is B. Without the correction you would look at C, see 0.0299 and declare two winners; one of them would be luck. This is exactly what the tool shows above when you open the page.

How to read it and where it fools you

The per-variation verdict answers one question: does this variation beat the control in a way that survives the number of bets you made. A variation marked not significant is not necessarily equal to the control, it almost always means there is not enough sample for that effect, all the more once the correction tightens the threshold.

Limits to keep in mind: the math assumes stable traffic and a binary metric (converted or not), and single-step corrections (Bonferroni, Šidák) are conservative by nature, they trade false positives for power. If your goal is to maximize detection with many variations, the definitive path is to plan the sample with that lower threshold in mind, using the sample size calculator, not to loosen the correction. Before trusting the verdict, also confirm the traffic split is not broken with the SRM checker.

Best practices with A/B/n tests

• Test fewer variations at a time. Each extra one steals power and sample from the whole set.
• Set the primary metric and the number of variations before you start the test, never add a variation mid-flight.
• Use Bonferroni as the default. Only switch to Šidák if the comparisons are genuinely independent and you want a sliver more power.
• Run full weekly cycles and do not stop at the first spike. The correction fixes the multiple-variation problem, not the peeking one.

Frequently asked questions

What changes when I test three or more variations at once?

Every extra comparison against the control adds a chance of a false positive. With one variation at 95% confidence the error risk is 5%. With five variations, the chance of at least one false positive by luck climbs to about 23%. That is why an A/B/n test has to correct the significance threshold: you cannot just run several two-proportion tests and look at the smallest p-value.

What is the difference between Bonferroni and Šidák?

Both hold the family-wise error by splitting the confidence across the comparisons. Bonferroni uses alpha divided by the number of comparisons (0.05 / 3 = 0.0167 for three variations): it is the simplest and the most conservative. Šidák uses 1 minus (1 minus alpha) to the power of 1 over the number of comparisons: a touch less strict, and it assumes the comparisons are independent. In practice they land very close; Bonferroni is the safe default.

Why does a variation pass on its own and fail after correction?

Because the threshold moved. A p-value of 0.03 rejects the null at 95% if you looked at that variation in isolation. But in a test with three variations at 95%, the per-comparison threshold drops to 0.0167 (Bonferroni), so 0.03 no longer passes. The variation did not get worse: the math is now charging the price of having tested several bets at the same time.

Is this the same as the two-variation significance calculator?

No. The plain significance calculator compares one variation B against control A and tells you if that single duel is real. This one is for when you have B, C, D or more against the same control: it runs each duel and corrects the threshold for the whole set. If your test has a single variation, use the statistical significance calculator, which is more direct.

Can the correction hide a real gain?

It can. Being stricter about false positives costs statistical power: real but small effects get harder to detect. The fix is not to drop the correction, it is to plan. Test fewer variations at a time, or size the sample knowing the threshold will be lower. Running five variations on traffic that barely funds two is the shortest path to an inconclusive result.

Keep going

For a two-variation duel, the statistical significance calculator is more direct. If you want to understand the p-value itself, see the p-value calculator. And before you run, size it with the sample size calculator to see if your traffic can fund all those variations.