What is A/B testing? The definitive guide (2026)
A/B testing from zero to advanced: how it works, how many visitors you need, the mistakes that invalidate results, and how to call a winner honestly.
A/B testing is the method of comparing two or more versions of a page (or element) by showing each one to a random slice of visitors and measuring which converts better. Instead of changing your page on a hunch and hoping, you let data from your own audience decide.
This guide covers everything from the concept to the part most articles skip: the statistics that separate a real result from a false positive. It’s the article we wish we’d found when we started.
How A/B testing works, in practice
The flow is always the same, regardless of the tool:
- Hypothesis. “Changing the button label from Buy to Add to cart will increase clicks.”
- Variations. A = control (your current page), B = variation (the change).
- Random split. Each visitor is assigned to A or B by a stable coin flip (the same visitor always sees the same version).
- Collection. You count visitors and conversions on each side.
- Decision. When there’s enough evidence, you call a winner. If there isn’t, the test is inconclusive, and that’s fine.
An A/B test doesn’t “prove” B is better. It estimates the probability that B beats A. Anyone promising certainty is selling an illusion.
When to use it (and when not to)
A/B testing shines when you have enough traffic and a clear metric. It’s the wrong tool when:
- Volume is too low (a few hundred visitors/month): the test would take months to conclude.
- The change is obvious and risk-free (fixing a bug, a broken link): just do it.
- You want to understand why something happens: that calls for qualitative research, not A/B.
How many visitors do you need?
This is the question that sinks most tests. Sample size depends on three things:
| Factor | Effect on sample size |
|---|---|
| Current conversion rate | The lower it is, the larger the sample |
| Minimum detectable effect (MDE) | The smaller the gain you want to detect, the larger the sample |
| Desired confidence | The more rigor, the larger the sample |
Rule of thumb: to detect a ~10% relative lift on a 5% rate, you need thousands of visitors per variation. Running with 200 people and “seeing who won” is the fastest path to a wrong decision.
The 5 mistakes that invalidate a test
- Stopping early (peeking). Checking the result every day and ending when it “shows a win” inflates false positives badly.
- Small sample. Without enough visitors, the winner is noise.
- Testing too many things at once. If you changed 8 things, you don’t know which moved the needle.
- Ignoring an uneven split (SRM). If A got 60% of traffic and B 40%, something broke, and the result is void.
- Running for too few days. Monday behavior differs from Sunday; run at least one or two full weeks.
How to call a winner honestly
There are two schools: frequentist (the classic “p-value < 0.05”) and Bayesian (the “probability that B beats A”). The Bayesian approach tends to be more intuitive for decision-makers: instead of an abstract p-value, you read “B has a 97% chance of being better.”
Whichever school, an honest winner requires three things at once:
- Enough visitors per variation;
- Enough time (full weeks, not spikes);
- A statistically solid difference, not one that showed up on a lucky day.
A/B testing with Donnu
Donnu A/B runs real experiments on your site with a lightweight snippet that never breaks the page, Bayesian statistics that call the winner honestly, and per-account data isolation. No guessing, no p-value theater.