How to Write an A/B Test Hypothesis (with a Free Template)
Learn how to write a strong A/B test hypothesis with a free template, worked examples, and a sample size calculator to size it right.
📚 This article is part of the guide What is A/B testing? The definitive guide (2026).
An A/B test hypothesis is a testable statement that ties an observation to a change, a predicted effect, and the metric that will prove it right or wrong. “Let’s just try a new headline and see what happens” is not a hypothesis, it’s a guess with a deploy button. The difference matters because a hypothesis is what tells you, before you spend a single visitor’s worth of traffic, what you’re claiming, how you’ll know if you’re right, and when to stop.
This guide gives you the exact structure to write one, a free copyable template, two fully worked examples (ecommerce and SaaS), and a live calculator to size the resulting test with real numbers instead of guesswork.
What Is an A/B Test Hypothesis? (and What It Is Not)
A hypothesis is a falsifiable claim. It says: given this evidence, I expect this specific change to move this specific metric, in this specific direction. It is not a vibe, a wish, or a design opinion dressed up as a plan.
Here is the distinction in practice:
- Not a hypothesis: “Let’s redesign the pricing page, it feels outdated.”
- A hypothesis: “Because 40% of pricing-page visitors leave without scrolling past the first plan (from session recordings), we believe reordering plans to show the most popular one first will increase plan-selection clicks, measured by clicks on any ‘Select plan’ button.”
The second version gives you three things the first one never will: a reason to believe the change will work, a single number that will tell you if it did, and a claim specific enough that the test can actually fail. If a statement can’t fail, it isn’t a hypothesis, it’s marketing copy for your own roadmap.
There’s also a formal counterpart worth knowing, even if you never write it down: the null hypothesis, the assumption that your change makes no difference at all. Every A/B test is built to challenge that assumption with data, and the statistics that decide the test (p-values, confidence intervals) are really just a formal way of asking “how surprising would this result be if the null hypothesis were actually true?” You don’t need to run the math by hand to write a good hypothesis, but knowing that the test’s whole job is to try to disprove “no effect” helps explain why a vague hypothesis is so hard to test cleanly, there’s nothing precise to disprove.
A hypothesis also isn’t the same thing as an idea. Ideas are cheap and everyone has a backlog full of them (“try a countdown timer,” “add more testimonials”). A hypothesis is what you get once you’ve forced an idea through evidence, a specific mechanism, and a metric. Most ideas never survive that process, which is exactly the point, it filters out the ones that were never going to teach you anything.
The Hypothesis Template That Actually Works
The structure that holds up across ecommerce, SaaS, and lead-gen tests is a single sentence with four slots:
Fill in each bracket literally, in order, and you get a sentence that is specific enough to build a test from and specific enough to prove wrong. Skip a slot and the hypothesis quietly degrades: no data means you’re guessing, no named change means the test isn’t isolating anything, no predicted effect means you won’t notice if the result contradicts you, and no metric means you’ll pick one after the fact that happens to look good.
The 3 Ingredients of a Strong Hypothesis
Every strong hypothesis, regardless of industry or page type, rests on the same three ingredients.
| Ingredient | What it means | Weak version | Strong version |
|---|---|---|---|
| Evidence | A real observation: analytics, session recordings, support tickets, a prior test | “I have a feeling users are confused” | “38% of mobile checkout sessions abandon on the payment-method step (funnel data)” |
| Causal change | One isolated, specific change you can point to | “Improve the checkout flow” | “Move Apple Pay above the credit card field on the payment step” |
| One primary metric | A single number, chosen before launch, that decides the test | “See if conversion goes up” | “Payment-step completion rate, tracked as the primary metric before launch” |
Notice that none of the three ingredients require a big team or a research budget. Session recordings, a support ticket tag, or last quarter’s funnel report are all legitimate evidence. What disqualifies a hypothesis isn’t the size of the evidence, it’s the absence of any evidence at all.
It helps to run every draft hypothesis through a quick gut check before it goes on the backlog: could this hypothesis lose? If you genuinely cannot picture a version of the test where the variation ties or loses, you probably haven’t isolated a single change, you’ve bundled a redesign with a story attached. A hypothesis that can only win isn’t a hypothesis, it’s a foregone conclusion looking for a test to rubber-stamp it.
Worked Examples: Ecommerce and SaaS
Templates are easier to internalize with full, concrete examples. Here are two, each grounded in a plausible data point, each naming all three metric roles.
Ecommerce, checkout page
Because we observed that 3% of checkout sessions convert to a completed purchase, and exit-survey responses cite “unexpected shipping cost” as the top reason for abandoning at the review step, we believe showing the estimated shipping cost on the product page (instead of only at checkout) will cause a relative increase in checkout completion rate, measured by the percentage of sessions that complete a purchase.
- Primary metric: checkout completion rate (purchases / checkout sessions started)
- Secondary metric: time spent on the review step
- Guardrail metric: average order value, to catch a scenario where showing shipping cost early pushes customers toward cheaper baskets
SaaS, signup flow
Because we observed that 8% of pricing-page visitors start a free trial, and support tickets show recurring confusion about which plan includes the feature prospects ask about most, we believe adding a feature-comparison table directly on the pricing page will cause a relative increase in trial starts, measured by the percentage of pricing-page visitors who begin a trial.
- Primary metric: trial-start rate (trials started / pricing-page visitors)
- Secondary metric: scroll depth and time on the pricing page
- Guardrail metric: trial-to-paid conversion rate, to catch a scenario where the table attracts trials that were never a good fit
Both examples follow the same anatomy. Evidence leads to a change, the change predicts an effect, and the effect is pinned to one measurable number:
Prioritizing Your Hypotheses: ICE vs. PIE
A healthy backlog has more hypotheses than traffic to test them. Two frameworks give you a repeatable way to rank them instead of testing whatever is easiest to build:
| Framework | Dimensions | Best for |
|---|---|---|
| ICE | Impact, Confidence, Ease | Small teams, fast triage, mixing product and marketing ideas on one list |
| PIE | Potential, Importance, Ease | Teams that already segment pages by traffic and want to weight high-traffic pages higher |
Both frameworks work the same way: score each dimension from 1 to 10, then average (or multiply) the scores to rank ideas. Neither is precise science, their real value is forcing the team to write down why an idea deserves testing time before anyone touches a design tool.
A scored example, using the two hypotheses above:
Common Hypothesis Mistakes
Most hypotheses that fail don’t fail because the test ran badly, they fail because the hypothesis was broken before the test ever launched.
| Mistake | What it looks like | Fix |
|---|---|---|
| Vague hypothesis | “Improve the homepage” with no named change or metric | Rewrite using the four-slot template until it names one specific change |
| Testing multiple changes at once | New headline, new CTA color, and a shorter form, all in one variation | Isolate one change per test, or use a multivariate design built to separate combinations |
| No primary metric defined upfront | “We’ll see which number looks best afterward” | Lock in one primary metric before launch, everything else is secondary or guardrail |
| Writing the hypothesis after seeing the data | The “hypothesis” is written to match a result that already happened | Draft and date the hypothesis before the test starts, treat it as a commitment |
The last row is the most common and the hardest to self-catch. If a “hypothesis” only exists in a slide deck built after the test ended, it isn’t a hypothesis, it’s a rationalization.
A related trap worth naming separately: reading a test as a win because some metric moved, even if it wasn’t the one written into the hypothesis. Look at enough secondary metrics on any test and a few will cross a significance-looking threshold by chance alone, that’s not evidence the change worked, it’s exactly what random noise produces when you check enough numbers. The primary metric named in the hypothesis is the only one allowed to decide the outcome; everything else is context.
From Hypothesis to Test: Size It
A hypothesis predicts an effect, and that predicted effect is what sizes the test. The bigger the lift you’re claiming, the fewer visitors you need to detect it; the smaller the claimed lift, the more traffic the test demands, and the relationship isn’t gentle, it’s steep. Halving the effect you want to detect roughly quadruples the required sample.
Take the ecommerce hypothesis above: baseline checkout completion rate of 3%, and a predicted relative lift of 15% (moving the rate toward roughly 3.45%). At 95% confidence, 80% power, and a two-sided test, that requires 24,193 visitors per variation. With 8,000 weekly checkout sessions split across both variations, the test needs about 43 days to reach that sample.
For the SaaS hypothesis, baseline trial-start rate of 8% and a predicted relative lift of 12% requires 13,219 visitors per variation. With 3,000 weekly pricing-page visitors, that test needs about 62 days.
Plug in your own hypothesis’s baseline and predicted lift to see visitors-per-variation and estimated duration live:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
If your hypothesis needs a sample size your traffic can’t reach in a reasonable window, that’s useful information too, it tells you the predicted effect is either too small to chase right now or the change needs to be bolder. For the statistics behind this calculator, see our complete guide to A/B testing statistical significance.
A Free Copyable Hypothesis Doc Template
Copy this table into your experiment log for every test. Filling every column before launch is what keeps a hypothesis honest.
| Field | Example |
|---|---|
| Hypothesis | Because we observed [data], we believe [change] will cause [effect], measured by [metric] |
| Change | The single, isolated change being tested |
| Expected effect | Direction and rough size of the predicted lift |
| Primary metric | The one number that decides the test |
| Secondary metric(s) | Metrics that help explain why, but don’t decide the test |
| Guardrail metric(s) | What must not get worse while the primary improves |
| Owner | Who is accountable for launching, monitoring, and reading the result |
| Start / end date | Locked in before launch, based on the calculated sample size |
Keep this log even for tests that come back inconclusive. An inconclusive result, documented, stops the same idea from being retested blindly a year later by someone who forgot it was already tried.
Do This Automatically on Donnu
Writing a strong hypothesis is only half the discipline, the other half is sizing the resulting test correctly and reading the result without fooling yourself. Donnu handles that second half by default: you write the hypothesis and the goal metric, Donnu sizes the test against your real traffic, collects data with a snippet that never blocks your page, and reads the result with honest statistics instead of stopping the moment the number looks good.
Start a free 14-day trial and turn your next hypothesis into a properly sized test. For the full mechanics of running the test once your hypothesis is written, see how to run an A/B test step by step, and for background on the method itself, what A/B testing is from the ground up.
Read Also
- The complete guide to A/B testing
- How to optimize conversion rate (CRO)
- A/B testing statistical significance
Leia em português: Como escrever uma hipótese de teste A/B.
References
- Kohavi, R., Tang, D. & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. experimentguide.com.
- Miller, E. How Not To Run an A/B Test (the peeking problem), 2010. evanmiller.org.
- CXL. How to Write Hypotheses for Your CRO and A/B Testing Program. cxl.com.
- VWO. The Complete Guide to A/B Testing. vwo.com/ab-testing.
- Harvard Business Review. The Surprising Power of Online Experiments. hbr.org.
Frequently asked questions
- What is an A/B test hypothesis?
- A testable statement that connects an observation to a change, a predicted effect, and a metric that will confirm or deny it. The format that works: because we observed X, we believe change Y will cause effect Z, measured by metric M. A hunch like "let's just try a new headline" is not a hypothesis, it has no evidence, no predicted direction, and no defined metric.
- What makes an A/B test hypothesis strong instead of weak?
- Three ingredients: it is grounded in real evidence (data, research, or a prior test), it names one specific, isolated change rather than a bundle of changes, and it commits to a single primary metric before the test starts. Weak hypotheses usually fail on the third point, they let the team pick whichever metric moved after the fact.
- Should I write the hypothesis before or after building the variation?
- Always before. Writing the hypothesis first forces you to define the primary metric and the expected direction of effect in advance, which is what keeps you honest when the results come in. Writing it after you have already seen data is how teams talk themselves into shipping noise.
- How do I prioritize which hypotheses to test first?
- Score each one with a framework like ICE (Impact, Confidence, Ease) or PIE (Potential, Importance, Ease), rate each dimension from 1 to 10, and average or multiply the scores to rank ideas. Neither framework is precise science, their real value is forcing the team to write down why an idea is worth testing before committing engineering time to it.
- Can one A/B test have more than one hypothesis?
- Not if you want a clean read. Bundling several changes under one test (a new headline, a new CTA color, and a shorter form all at once) means that if the variation wins, you cannot say which change caused it. Test one hypothesis at a time, or use a multivariate design built specifically to isolate combinations, which needs substantially more traffic.
- How many visitors do I need to test a hypothesis?
- It depends on your baseline conversion rate and the size of the effect the hypothesis predicts. As a concrete example, detecting a 15% relative lift on a 3% baseline conversion rate, at 95% confidence and 80% power, needs about 24,193 visitors per variation. Smaller predicted effects need dramatically more traffic, use the calculator in this guide to size your own case.