How to Run an A/B Test: Step-by-Step Guide

Running an A/B test means splitting real traffic randomly between a control and a variation, collecting conversions on each side, and using statistics to decide, honestly, which one wins. Get the process right and you replace opinion with evidence. Get the mechanics wrong, wrong sample size, peeking, testing during an unusual week, and you get a confident-looking number that means nothing.

This guide walks through every step: from finding what to test, to sizing the test correctly with a live calculator, to reading a result you can actually trust. For the statistics behind the calls this guide makes, see our complete guide to A/B testing statistical significance.

A/B, A/B/n, split URL, or multivariate?

“A/B testing” is often used as an umbrella term, but the mechanics differ:

Type	What it is	When to use it
A/B	Control vs. one variation, same URL	The default case: one isolated, clear change
A/B/n	Control vs. several variations (C, D…)	When you have several strong ideas and plenty of traffic
Split URL	Each version is a separate page, split by redirect	Big redesigns, or when the variation already exists as another page
Multivariate (MVT)	Tests combinations of several elements at once	Finding the best combination, only with high traffic (needs a much larger sample)

Start with simple A/B. Multivariate testing looks powerful, but the number of combinations, and the sample size required to tell them apart, explodes fast.

Before you start: do you even have enough traffic?

A/B testing is the wrong tool in a few common situations. If your volume is too low, hundreds of visitors a month, a valid test would take many months to complete. If the fix is obvious and risk-free, a broken link, a typo, just ship it; you don’t need statistical proof that fixing a bug is good. And if your real question is why something happens rather than whether a change helps, that calls for qualitative research (interviews, session recordings), not a quantitative test.

If none of those apply, the next question is simply: does your traffic support the effect size you care about? A page with 500 monthly visitors and a 2% conversion rate will take a very long time to detect a modest +10% lift. It can, however, detect a bold +50% redesign in a reasonable window. Traffic doesn’t disqualify you from testing, it just tells you how bold your changes need to be.

Step 1: find what’s worth testing

Not every element on a page is equally worth your testing budget. The highest-leverage places to look are usually the value proposition and headline, the primary call-to-action, form length and friction, social proof placement, and pricing presentation. A funnel with a clear, large drop-off point is the best place to start: a percentage-point gain at the biggest leak in your funnel is worth more than the same gain applied somewhere with little traffic passing through.

Once you have a shortlist of ideas, prioritize instead of testing whatever’s easiest to build. Two common frameworks:

• ICE (Impact, Confidence, Ease): score each idea 1-10 on each dimension and multiply, or average, to rank.
• PIE (Potential, Importance, Ease): similar logic, weighted toward how much traffic and attention the page already gets.

Neither framework is precise science, their value is forcing you to write down why an idea is worth testing before you build it.

Step 2: write a hypothesis that can actually win

A test without a hypothesis is a guess wearing a lab coat. A strong hypothesis connects an observation to a change, an expected effect, and the metric that will measure it:

A concrete example: “because 62% of carts are abandoned at the shipping step (funnel data), we believe showing shipping cost on the product page will reduce abandonment, measured by checkout completion rate.” Notice what that sentence gives you for free: the primary metric, the expected direction, and the evidence it came from, not a hunch. Hypotheses that start with “what if we tried…” are weak; hypotheses that start with a number or an observation are strong.

Step 3: choose your metrics

Pick one primary metric, the number that decides the test. Everything else plays a supporting role:

• Primary (the OEC, overall evaluation criterion): the one conversion that matters. If you have two “primaries,” you effectively have none.
• Secondary metrics: help explain the why (clicks, time on page, funnel step reached) but don’t decide the test.
• Guardrail metrics: what must not get worse while the primary improves. A checkout redesign that raises conversion but spikes refunds or cancellations is not a win; the guardrail catches that before you celebrate too early.

Choosing your primary metric after looking at the data is a trap: if you run the test and then go looking for whatever number “won,” you’ll always find one by chance. Lock it in before launch.

Step 4: size the test, sample size and duration

This is the step that decides whether your result will mean anything. Sample size depends on three inputs: your baseline conversion rate, the Minimum Detectable Effect (the smallest lift worth finding), and your chosen rigor (95% confidence and 80% power are the market standard). The relationship between effect size and sample isn’t linear, it’s steep: halving the effect you want to detect roughly quadruples the sample required, which is exactly why testing a single word on a medium-traffic page rarely reaches a clean result.

Plug in your own numbers and see visitors-per-variant and estimated duration live:

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

For the math behind this calculator and a fully worked example, see the sample size and duration section of our statistics guide, or use the dedicated sample size calculator.

Step 5: set it up so you don’t fool yourself

The statistics assume clean data collection. A few execution details separate a trustworthy test from data that only looks trustworthy:

• Run an A/A test first. Pit two identical versions against each other. If they “detect” a significant difference, your tool or tracking has a bug, and nothing downstream can be trusted until it’s fixed.
• Watch for the flicker effect (FOOC). In browser-based tests, the original page can flash for an instant before the variation loads, biasing results and hurting user experience. A good testing tool hides and reveals cleanly without blocking the page.
• Client-side vs. server-side. Client-side is easy to install and fine for marketing pages; server-side is more robust, has no flicker, and is the standard for product and SaaS experiments. Choose based on the use case, not just ease of setup.
• Run variants simultaneously. Never compare “this week with B” against “last week with A.” Seasonality and external noise will contaminate the comparison. Simultaneous, randomized traffic is what isolates your change from everything else happening in the world.
• Use a stable randomization unit. The same visitor should always see the same version. If they flip between A and B across sessions, their conversion doesn’t belong to either group and pollutes both.

Step 6: let it run, and don’t peek and stop

Checking the dashboard daily and stopping the moment it shows “significant” feels efficient but quietly destroys the test’s reliability. Every extra look is another chance for random noise to cross the significance line. A test designed for a 5% false-positive rate can climb toward 25-30% if you peek and stop opportunistically:

Also run at least one to two full weeks, even if you hit your sample size sooner. Monday behavior isn’t Sunday behavior, and payday shoppers aren’t weekend browsers. Stopping after three days “because it already looks done” captures a biased slice of your real audience.

Step 7: read the results without fooling yourself

A p-value below 0.05 means there’s less than a 5% chance a difference this large would appear if the versions were truly identical, it does not mean “95% chance B is better.” A confidence interval is often more useful: it shows the plausible range of the lift, and if that range crosses zero, the result isn’t reliable yet, whatever the raw numbers look like.

Also separate statistical significance from practical significance. A p-value of 0.02 attached to a +0.1 percentage-point lift is technically real and probably not worth shipping. Decide, before the test, what size of lift would actually justify the engineering cost of implementing the change.

If the result comes back not significant, that’s a legitimate, common outcome, not a failure. Check whether you stopped before reaching your calculated sample, whether the true effect might be smaller than the MDE you targeted, or whether the change itself was simply too small to move a metric with this much traffic. What you shouldn’t do is slice the data into segments after the fact hunting for a subgroup where it “worked,” that’s fishing, and it manufactures false positives.

Step 8: ship the winner, document, and run the next test

When a variation wins cleanly, roll it out, but consider a gradual ramp for high-stakes changes rather than flipping 100% of traffic overnight, so you can catch any guardrail metric that slips once the whole audience sees it. Document what you tested, why, and what happened, win, loss, or inconclusive, in a shared log. That log becomes your institutional memory: it stops the same idea from being retested blindly six months later by someone who forgot, and it steadily turns a series of individual, disconnected tests into a real, compounding experimentation program.

Do this automatically on Donnu

Sizing the test, avoiding peeking, watching for a flicker-free rollout, and reading significance honestly, that’s the full discipline a real A/B test demands. Donnu builds it in by default: you set the hypothesis and the goal metric, Donnu sizes the test, collects data with a snippet that never blocks your page, and calls the winner with honest statistics.

Start a free 14-day trial and run your next test on a foundation that already gets the mechanics right. For the statistics behind every call this guide makes, read our complete guide to A/B testing statistical significance, or see what A/B testing is from the ground up.

References

• Kohavi, R., Tang, D. & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. experimentguide.com.
• Miller, E. How Not To Run an A/B Test (the peeking problem), 2010. evanmiller.org.
• Kohavi, R. et al. Seven Rules of Thumb for Web Site Experimenters. KDD, 2014.