A/B Test Significance Calculator
Find out whether the difference between your versions is real or luck. Paste the test numbers and read the verdict, with p-value, lift and confidence interval.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
How to use it
- Fill in visitors and conversions for the control (A).
- Fill in the same two numbers for the variation (B).
- Keep confidence at 95% or switch to 90% or 99%.
- Read the verdict at the top and the details: each version's rate, relative lift, p-value and confidence interval of the difference.
How it works: the two-proportion z-test
The calculator runs a two-sided two-proportion z-test. It compares the two rates using a pooled standard error and returns the p-value:
Where p₁ and p₂ are the rates of A and B, n₁ and n₂ the visitors of each, and p̄ the pooled rate (all conversions over all visitors). The two-sided p-value comes from 2·(1 − Φ(|z|)), where Φ is the standard normal.
Worked example (reproduces the default result)
With the numbers that come pre-filled: A with 4,200 visitors and 210 conversions (5.00%) and B with 4,200 visitors and 273 conversions (6.50%). The pooled rate is 483/8,400 = 5.75%. The standard error is about 0.00508, so z = 0.015 / 0.00508 = 2.95. That yields a p-value of 0.0031, well below 0.05.
Result: relative lift is +30.0%, the 95% confidence interval of the difference runs from +0.5% to +2.5% (points) and the verdict is B wins with significance. That is what the tool shows above when you open the page.
How to read it, and where it fools you
The p-value measures false positive risk, not the size of the effect. A tiny p-value with a +0.3% lift may not pay for the switch. So also look at the confidence interval: if it includes zero, the gain is still uncertain, even when the central number looks good.
The costliest mistake is confusing "not significant" with "the versions are equal". It is almost always a lack of sample. Size it first with the sample size calculator and plan the runtime with the duration calculator. And never stop the test at the first significance: that is the peeking problem.
Best practices when reading the result
The verdict at the top is a summary, not a standalone sentence. To avoid making the wrong call with the right numbers, check these points before you end the test and switch the page.
- Look at the confidence interval, not just the p-value. If the interval touches zero, the gain is still uncertain even with a positive verdict.
- Confirm the planned sample was reached. Reading before that is peeking and inflates the false positive rate.
- Keep 95% confidence as the default. Only raise it to 99% when the cost of an error is very high.
- Translate the relative lift into revenue before celebrating. A +30% gain on a tiny base can be worth little at the register.
Frequently asked questions
- What does statistically significant mean?
- It means the difference between A and B is too large to be explained by chance alone, given the confidence level you chose. With a p-value below 0.05 (95% confidence), the market treats the result as significant. It is not a guarantee of absolute truth, it is a ruler for the risk of error.
- What is the p-value?
- It is the probability of seeing a difference as large as yours (or larger) by chance alone, if A and B were truly equal. A p-value of 0.03 means a 3% chance the result is luck. The smaller it is, the stronger the evidence that the difference is real.
- Does "not significant" mean the versions are equal?
- Almost never. In practice, "not significant" almost always means not enough sample, not that A and B tied. Before concluding there is no effect, check whether you had enough visitors with the sample size calculator.
- Can I read the result at any point during the test?
- No. Peeking at the p-value and stopping the moment it crosses 0.05 badly inflates the false positive rate (the peeking problem). Fix the sample before you start, run until it is complete, then read the verdict here.
- What is the difference between relative and absolute lift?
- Absolute is the difference in percentage points (from 5% to 6.5% is +1.5 points). Relative is the proportional gain (from 5% to 6.5% is +30%). Both describe the same result, the relative one usually sounds larger.
Keep going
The formula annotated term by term and the mistakes that invalidate the reading are in the guide on statistical significance in A/B testing and the overview of what A/B testing is.