A/B Testing Statistical Significance Guide

Statistical significance in A/B testing is the measure of how unlikely it is that the difference you observed between two versions happened by chance alone. A result is usually called “significant” when its p-value falls below 0.05, meaning less than a 5% probability that a difference this large (or larger) would appear if the two versions were, in truth, identical.

That one-sentence answer is where most guides stop. This one doesn’t. Below you’ll find the actual z-test formula worked term by term, live calculators that reproduce it exactly, the peeking mistake that quietly inflates false positives, and the honest answer to “it’s significant, now what?” It’s the reference we wish existed when we started building an A/B testing product ourselves, one that treats the reader as someone who has to defend a real business decision with this number, not just pass a stats quiz.

What is statistical significance in A/B testing?

Every A/B test answers one question: is the difference between your control (A) and your variation (B) signal, or is it noise? Statistical significance is the tool that tells them apart.

Here’s the trick that makes it click. Imagine you split your traffic into two identical pages, A and A, with zero real difference between them. Because of pure randomness, they will almost never show the exact same conversion rate. One will always look a little better than the other, purely by chance. Statistical significance is the discipline of asking: is the gap I’m looking at bigger than the gap two identical pages would show anyway, just from randomness?

Formally, every test starts from a null hypothesis: “there is no real difference between A and B.” The whole statistical machinery exists to measure how much evidence you have against that null hypothesis. You never technically “prove” B is better, you gather enough evidence to reject the assumption that they’re the same.

The p-value, demystified

The p-value answers a very specific question: if A and B were truly identical, what is the probability of seeing a difference at least this large, purely from random sampling? A p-value of 0.03 means a 3% chance of seeing this result (or a more extreme one) if there were no real effect at all.

That precision matters because the p-value gets misread constantly. Three interpretations to avoid:

• “The p-value is the probability B is better.” No. A p-value of 0.03 is not a 97% chance that B wins. It only describes how surprising your data would be under the assumption of no difference.
• “Not significant means A and B are equal.” No. It almost always means you don’t have enough evidence yet, usually because the sample is too small, not that the effect is zero.
• “p = 0.049 is meaningfully different from p = 0.051.” No. The 0.05 line is a convention, not a cliff edge. Treat values near the threshold as “still gathering evidence,” not a hard pass/fail.

Confidence level and significance level (alpha)

The significance level (alpha) is the false-positive rate you’re willing to accept, decided before the test runs. The confidence level is just 1 minus alpha, expressed as a percentage. A 95% confidence level means alpha = 0.05: a 5% chance of calling a winner when there isn’t one.

Confidence	Alpha	What it costs you
90%	0.10	Faster calls, but 1 in 10 “wins” is a false alarm
95%	0.05	Market standard; balances speed and reliability
99%	0.01	Very few false positives, but needs much more traffic

Use 90% only for low-risk, easily reversible changes where speed matters more than certainty. Reserve 99% for expensive or hard-to-reverse decisions (pricing, checkout architecture), where the cost of a false positive is high enough to justify the extra traffic.

The two errors that ruin decisions: Type I and Type II

Every test can fail in exactly two ways. A Type I error (false positive) is declaring a winner that doesn’t exist; your chosen confidence level controls this directly (5% at 95% confidence). A Type II error (false negative) is missing a real improvement because your test wasn’t sensitive enough to detect it; statistical power controls this.

Statistical power and Minimum Detectable Effect (MDE)

Power is the probability your test will detect a real effect if one exists. The industry standard is 80% power, meaning you accept a 20% chance of missing a genuine improvement. Power and sample size are directly linked: with too few visitors, even a real, meaningful lift can slip through undetected.

The other half of the equation is the Minimum Detectable Effect (MDE): the smallest lift you’ve decided is worth finding. Setting an MDE isn’t just a statistics exercise, it’s a business decision. Detecting a huge +50% lift is cheap in traffic; detecting a tiny +2% lift can require dozens of times more visitors, because the signal is that much closer to the noise floor of ordinary variation.

The math behind it: the two-proportion z-test, term by term

This is the part almost every guide skips. For two proportions (your control rate and your variation rate), the standard test computes a z-score, the distance between the two rates measured in “standard deviations of pure chance”:

Term by term: p_A and p_B are the conversion rates of each version; n_A and n_B are the visitor counts; p̄ (p-bar) is the pooled rate across both groups, the estimate of “what the rate would be if A and B were identical.” The bigger the z-score, the further the observed difference sits from what pure chance would produce, and the p-value is simply that z-score translated into a probability.

A fully worked example

Control (A) had 210 conversions out of 4,200 visitors; variation (B) had 273 out of 4,200.

• Rate of A: 210 ÷ 4,200 = 5.00%. Rate of B: 273 ÷ 4,200 = 6.50%.
• Relative lift: (6.5 − 5.0) ÷ 5.0 = +30%. Looks great, but is it real?
• Pooled rate p̄: (210 + 273) ÷ 8,400 = 5.75%.
• Standard error: √[0.0575 · 0.9425 · (1÷4,200 + 1÷4,200)] ≈ 0.00508.
• z-score: (0.065 − 0.050) ÷ 0.00508 ≈ 2.95.
• Two-tailed p-value ≈ 0.003.

A p-value of 0.003 clears the 0.05 bar comfortably: the result is significant, B wins. This is exactly the calculation the tool below performs for you, live, with your own numbers.

z-test vs t-test, one-tailed vs two-tailed

Conversion rates (a visitor either converts or doesn’t) use a z-test, which is what powers the calculator on this page. Continuous metrics like revenue per visitor or time on page need a t-test, because their variance behaves differently, and with small samples a t-test’s slightly wider tails better account for the extra uncertainty of estimating variance from limited data. On tails: a two-tailed test checks for a difference in either direction and is the safe default; a one-tailed test only checks for improvement in one specific direction, which is riskier because it can’t catch a variation that quietly makes things worse. Unless you have a very specific reason to rule out one direction entirely, default to two-tailed.

Try it now: statistical significance calculator

Paste in your own visitors and conversions for each version. The calculator returns the rates, the relative lift, the p-value, the confidence interval of the difference, and an honest verdict:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Confidence intervals: report this, not just the p-value

A p-value tells you whether there’s a significant difference. A confidence interval tells you the plausible range of that difference, which is far more useful for a business decision. If B’s 95% confidence interval for the lift is “+0.5 to +2.5 percentage points,” even the worst plausible case in that range is still an improvement.

A confidence interval that crosses zero (for example, “-0.3 to +2.1 points”) is the visual definition of “not significant”: the true effect could plausibly be negative, zero, or positive, so you don’t yet have a reliable direction.

How big should your sample be?

Sample size depends on three inputs you control directly: your baseline conversion rate, the MDE you’ve decided is worth detecting, and your chosen confidence and power. Set the numbers for your own case below and get visitors-per-variant and estimated duration:

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

As a reference, holding confidence at 95% and power at 80%, with a 5% baseline rate:

Minimum detectable effect (relative)	Approx. sample per variant
50%	~1,500
20%	~8,000
10%	~31,000
5%	~122,000

The relationship isn’t linear, it’s brutal: halving the effect you want to detect roughly quadruples the sample you need. That’s why micro-tests (“changed one word”) almost never reach significance on medium-traffic sites, the effect is small and the required sample is enormous. For the full walkthrough of this table, see how many visitors an A/B test needs.

How long should you run the test?

Even if you hit your calculated sample size early, run the test for at least one to two full weeks. Monday behavior differs from Sunday behavior, and payday shoppers aren’t weekend browsers. Stopping after three days “because it already looks done” captures a biased slice of your actual audience. On the other end, running a test for six-plus weeks without progress usually means the effect is smaller than your MDE, or traffic genuinely isn’t sufficient; recalculating with a bigger MDE is more useful than waiting it out.

The #1 mistake that fakes significance: peeking

Checking your dashboard daily and stopping the instant it shows “significant” feels efficient, but it quietly destroys the test’s reliability. Every additional look is another chance for random noise to cross the significance line. A test designed for a 5% false-positive rate can climb to nearly 25-30% if you peek and stop opportunistically:

Sequential testing: peeking done right

Fixed-horizon testing (calculate a sample size, don’t look until you hit it) isn’t the only valid design, it’s just the simplest to explain the math for. Sequential testing and always-valid inference are methods built specifically to let you check results as often as you like without inflating your false-positive rate, by spending your alpha budget gradually across looks instead of all at once.

The trade-off is complexity: sequential methods (group sequential designs, mixture sequential probability ratio tests, and similar always-valid confidence sequences) require the tool running your test to implement them correctly from the start, they aren’t something you bolt on by checking a fixed-horizon p-value more often. If your testing platform explicitly supports “peek anytime” as a feature, it’s almost certainly running one of these methods under the hood. If it doesn’t say so, assume it’s fixed-horizon, and treat early peeking exactly like the chart above: a silent tax on your false-positive rate.

Multiple comparisons: testing many variants or metrics

Every extra variant or extra metric you check is another roll of the dice for a false positive to sneak through. Test 20 unrelated metrics at 95% confidence and, on average, one of them will look “significant” purely by chance, even if nothing real changed. This is why a test with a control plus four variants (A/B/C/D/E) needs more caution than a simple A/B, and why “we found a winner in one of our twelve secondary metrics” is a weak claim on its own.

The standard fix is a correction, most simply the Bonferroni method: divide your significance threshold (0.05) by the number of comparisons you’re making. Testing 5 variants against control means using roughly 0.05 ÷ 5 = 0.01 as your effective significance bar for each one. It’s conservative, but it keeps your real false-positive rate honest. The simplest practical guardrail: decide your primary metric and your variant count before the test, and treat anything found by digging through the rest as a hypothesis for your next test, not a conclusion from this one.

Sanity checks before you trust any p-value

A textbook-perfect p-value can still be worthless if the data collection was broken. Two checks to run before you trust any result:

• Sample Ratio Mismatch (SRM). If you configured a 50/50 split but the actual traffic arrived at, say, 55/45 with no good reason, something is broken in your randomization, caching, or bot filtering, and no p-value from that test can be trusted, no matter how convincing it looks. A simple chi-square test on the visitor counts (not conversions) catches this in seconds.
• Novelty effects, seasonality, and instrumentation bugs. A redesigned page sometimes wins early simply because it’s new and curious visitors click around more, an effect that fades over the following weeks. Comparing week-over-week instead of running both variants simultaneously invites seasonality to masquerade as a real lift. And a tracking bug that double-fires an event, or fails to fire on one variant, produces numbers that are internally consistent and completely wrong.

Continuous metrics: revenue, AOV, and time on page

Everything covered so far assumes a binary outcome: a visitor converts, or doesn’t. Metrics like revenue per visitor, average order value, or time on page are continuous, and they need a different tool: typically a t-test (or a non-parametric alternative like the Mann-Whitney U test when the data is heavily skewed), not the two-proportion z-test used above.

The practical headache with continuous metrics is variance. A handful of large orders can swing your average wildly, which inflates the standard error and demands far more traffic to reach significance than a simple conversion-rate test would. Two common fixes: cap or winsorize outliers (a single $50,000 order shouldn’t single-handedly decide a test meant to measure typical behavior), and report the median alongside the mean, since the median is far less sensitive to a handful of extreme values.

Statistical vs practical significance

A result can be statistically significant and still not matter to the business. If a p-value of 0.02 corresponds to a lift of +0.1 percentage points, ask whether that gain is worth the engineering cost of shipping it. Before running the test, define a minimum practically significant effect, the smallest lift that would actually justify implementation, separate from the smallest lift you can statistically detect (your MDE). Statistical significance answers “is this real?”; practical significance answers “is this worth doing?”

This distinction matters most with very large sample sizes. Run a test on millions of visitors and even a genuinely tiny, commercially irrelevant effect (say, +0.05 percentage points) will eventually clear the 0.05 significance bar, because with enough data almost any real difference, however small, becomes detectable. Large companies with huge traffic sometimes fall into the trap of shipping a long list of “statistically significant” changes that, added together, don’t move any dashboard a business leader actually watches. Setting your MDE deliberately, instead of just running the test until something turns green, is what keeps significance tied to something that matters.

Bayesian vs frequentist: two ways to define “significance”

Everything above is the frequentist approach: p-values and a fixed significance level. The Bayesian approach answers the same question differently, it gives you the direct probability that B is better than A (“B has a 97% probability of beating A”) along with the expected loss if you choose wrong.

	Frequentist	Bayesian
Output	p-value, significant/not	Probability B beats A
Reads like	“5% chance this is a fluke”	“97% chance B is better”
Early stopping	Requires sequential methods	Naturally more tolerant, with the right priors
Industry status	Long-standing default	Growing, especially in product-led SaaS

Neither is universally superior. Bayesian output tends to be more intuitive for a non-statistician making a ship/no-ship call; frequentist remains the academic and industry default. What doesn’t change in either school: you still need enough sample and time, and neither approach fixes a poorly designed test.

Low-traffic sites: testing without enough volume

If your traffic can’t reach the sample size a small MDE requires, you have three honest options. First, raise your MDE: stop chasing +2% and test changes big enough to plausibly move the needle by +20% or more, since larger effects need dramatically less traffic. A bolder redesign of a whole page section is often easier to validate statistically than a tiny copy tweak, simply because the expected effect is bigger. Second, look into variance-reduction techniques like CUPED (Controlled-experiment Using Pre-Experiment Data), which uses each visitor’s historical behavior before the test to strip out predictable noise, effectively increasing your test’s sensitivity without adding a single visitor. Third, when volume genuinely can’t support any reliable test, lean on qualitative research, session recordings, user interviews, and usability tests, to make informed decisions without a quantitative verdict.

There’s also a middle path worth knowing about: pooling tests across similar pages. A small e-commerce store might not have enough traffic on any single product page to test a checkout change, but if the same change is applied across every product page and measured as one aggregate test, the combined traffic can be enough to reach a real answer. The trade-off is that you lose the ability to say which specific page drove the result, so use this when you care about the change in general, not about ranking individual pages against each other.

None of these workarounds change the underlying math, they change how much traffic you need to feed it. A low-traffic site that raises its MDE, pools similar pages, and picks bigger, bolder changes can absolutely run valid experiments; it just has to be more deliberate about which questions are genuinely worth asking with a full A/B test in the first place, and which ones are better and faster to answer by simply talking directly to real customers instead.

The final checklist: can you declare a winner?

• Was the sample size calculated before the test started, based on a real baseline and MDE?
• Did the test run for at least one to two full business cycles?
• Is the p-value below your chosen alpha, and does the confidence interval avoid crossing zero?
• Did you check for Sample Ratio Mismatch (traffic split matches your configured ratio)?
• Is the effect size large enough to be practically significant, not just statistically?
• Did you avoid peeking-and-stopping, or use a method designed for it?

If every box is checked, you can call the winner with confidence. If any is missing, treat the result as provisional, keep collecting data, or rerun the test with the gap fixed rather than reporting a number you can’t fully stand behind.

None of this checklist is about being a statistician. It’s about the difference between a test that quietly gets challenged three months later when the “win” doesn’t hold up in the aggregate numbers, and one that survives scrutiny because every step was done in the right order: plan the sample first, run it clean, and read the result once.

Quick glossary

• Alpha: the significance level you accept, typically 0.05.
• Beta: the false-negative rate you accept, typically 0.20 (power = 1 − beta).
• Baseline rate: the conversion rate of your control before the test.
• MDE: Minimum Detectable Effect, the smallest lift worth finding.
• Pooled proportion (p̄): the combined conversion rate across both variants, used to estimate what “no difference” would look like.
• SRM: Sample Ratio Mismatch, when traffic doesn’t split the way you configured it, a sign the test’s data collection is broken.

Do this automatically on Donnu

You just walked through everything a statistically honest A/B test demands: the right sample size, the right duration, avoiding peeking, checking Sample Ratio Mismatch, and reading significance without fooling yourself. Donnu builds all of it in by default. You set the hypothesis and the goal metric; Donnu sizes the test, collects data with a snippet that never blocks your page, and calls the winner with honest statistics, Bayesian or frequentist. No guesswork, no p-value theater.

Start a free 14-day trial and run your next experiment on a foundation that already does the math right. For the fundamentals, read what A/B testing is and how it works, or read this guide in Portuguese.

References

• Kohavi, R., Tang, D. & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion site at experimentguide.com.
• Miller, E. How Not To Run an A/B Test (the peeking problem), 2010. evanmiller.org.
• Georgiev, G. Statistical Methods in Online A/B Testing. analytics-toolkit.com.
• Kohavi, R. et al. Seven Rules of Thumb for Web Site Experimenters. KDD, 2014.