What is A/B testing? The definitive guide (2026)
A/B testing from zero to advanced: how it works, how many visitors you need, the mistakes that invalidate results, and how to call a winner honestly.
A/B testing is the method of comparing two or more versions of a page (or element) by showing each one to a random slice of visitors and measuring which converts better. Instead of changing your page on a hunch and hoping, you let data from your own audience decide.
This guide covers everything from the concept to the part most articles skip: the statistics that separate a real result from a false positive. It’s the article we wish we’d found when we started.
How A/B testing works, in practice
The flow is always the same, regardless of the tool:
- Hypothesis. “Changing the button label from Buy to Add to cart will increase clicks.”
- Variations. A = control (your current page), B = variation (the change).
- Random split. Each visitor is assigned to A or B by a stable coin flip (the same visitor always sees the same version).
- Collection. You count visitors and conversions on each side.
- Decision. When there’s enough evidence, you call a winner. If there isn’t, the test is inconclusive, and that’s fine.
An A/B test doesn’t “prove” B is better. It estimates the probability that B beats A. Anyone promising certainty is selling an illusion.
When to use it (and when not to)
A/B testing shines when you have enough traffic and a clear metric. It’s the wrong tool when:
- Volume is too low (a few hundred visitors/month): the test would take months to conclude.
- The change is obvious and risk-free (fixing a bug, a broken link): just do it.
- You want to understand why something happens: that calls for qualitative research, not A/B.
How many visitors do you need?
This is the question that sinks most tests. Sample size depends on three things:
| Factor | Effect on sample size |
|---|---|
| Current conversion rate | The lower it is, the larger the sample |
| Minimum detectable effect (MDE) | The smaller the gain you want to detect, the larger the sample |
| Desired confidence | The more rigor, the larger the sample |
Rule of thumb: to detect a ~10% relative lift on a 5% rate, you need thousands of visitors per variation. Running with 200 people and “seeing who won” is the fastest path to a wrong decision.
The 5 mistakes that invalidate a test
- Stopping early (peeking). Checking the result every day and ending when it “shows a win” inflates false positives badly.
- Small sample. Without enough visitors, the winner is noise.
- Testing too many things at once. If you changed 8 things, you don’t know which moved the needle.
- Ignoring an uneven split (SRM). If A got 60% of traffic and B 40%, something broke, and the result is void.
- Running for too few days. Monday behavior differs from Sunday; run at least one or two full weeks.
How to call a winner honestly
There are two schools: frequentist (the classic “p-value < 0.05”) and Bayesian (the “probability that B beats A”). The Bayesian approach tends to be more intuitive for decision-makers: instead of an abstract p-value, you read “B has a 97% chance of being better.”
Whichever school, an honest winner requires three things at once:
- Enough visitors per variation;
- Enough time (full weeks, not spikes);
- A statistically solid difference, not one that showed up on a lucky day.
A/B testing with Donnu
Donnu A/B runs real experiments on your site with a lightweight snippet that never breaks the page, Bayesian statistics that call the winner honestly, and per-account data isolation. No guessing, no p-value theater.
Frequently asked questions
- What is the difference between an A/B test and a split URL test?
- In a classic A/B test the variations live on the same URL and the tool swaps page elements in the browser. In a split URL test each variation is a separate page with its own address, and traffic is divided by redirect. Use split URL when the versions are very different from each other, or when the variation already exists as another page.
- How many visitors do I need for an A/B test?
- It depends on three things: your current conversion rate, the size of the effect you want to detect (the MDE) and the level of rigor you want. To detect a 10% relative lift on a 5% rate, with 95% confidence and 80% power, you need roughly 31,000 visitors per variation. Run your own numbers in the sample size calculator instead of guessing.
- Why can I not stop the test as soon as it reaches significance?
- Because checking the result over and over and stopping the first time it "shows a win" inflates the false positive rate. A test designed for 5% error can drift toward 25% to 30% if you keep peeking and stopping. Define the sample and the duration up front and decide only at the end, or use sequential methods built to be looked at without that penalty.
- How long should I run an A/B test?
- At least one to two full weeks, so the test covers complete behavior cycles (weekdays and weekend), even if the sample target is reached earlier. Running only the "good days" or a handful of days distorts the result with seasonality.
- Does A/B testing hurt SEO?
- Not when it is done properly. Google recommends using rel="canonical" pointing to the original URL in split URL tests, avoiding cloaking (serving different content to the crawler) and running the test only for as long as it needs. Done that way, testing is an accepted practice.
- Frequentist or Bayesian: which one is better?
- Neither is universally better. The frequentist approach uses the classic p-value and significance level; the Bayesian one states the result as "the probability that B beats A", which tends to be more intuitive for whoever has to decide. What matters is not cheating with peeking and having enough sample and enough time, in either school.