Email A/B Test Sample Size Calculator

This calculator is for the email subject line A/B test, where you have a fixed list, not traffic arriving per day. You send two subject lines to two test cohorts, measure the open rate, and the winner goes to the rest of the list. To size a web page test (visitors per day and duration), use the A/B test sample size calculator.

How to use it

1. Enter your list size (subscribers available for the test).
2. Enter your current open rate, in %. It works for click rate too: swap in yours.
3. Pick the minimum detectable effect: the smallest lift that would still change your decision, relative (%) or absolute (points).
4. Leave confidence 95% and power 80% or adjust if you know what you are doing.
5. Read the result: subscribers per subject line, total test cohort, and how much is left for the winner, with the feasibility verdict.

How it works: the formula

The sample per subject line uses the two-proportion normal approximation, the same math as the site sample size calculator, applied to the open rate. The number per variation is:

Where p₁ is the current open rate, p₂ is the target open rate (base plus the MDE), p̄ is the average of the two, z_α is the confidence critical value (1.96 for 95% two-sided) and z_β is the power one (0.84 for 80%). The total test cohort is 2·n (two subjects, 50/50), and whatever is left of the list receives the winning subject.

Worked example (reproduces the default result)

With the values already filled in: a list of 50,000 subscribers, current open rate 25%, MDE +10% relative (target 27.5%), confidence 95% and power 80%, two-sided. The absolute difference is 2.5 points (0.275 − 0.25). Plugging into the formula, the numerator is about 3.0382 and the denominator 0.000625, so:

n ≈ 3.0382 / 0.000625 = 4,862 subscribers per subject line, or 9,724 across the whole test cohort. That is 19.4% of the 50,000 list, leaving 40,276 subscribers for the winning subject. It is exactly what the tool shows above when you open the page.

How to read it and where it misleads

The number per subject line is the target you lock before firing the test. The golden rule is the same as the site sample size: the smaller the effect you want to detect, the far larger the cohort. In email this bites fast, because the open rate has a high base and large variance. If the test cohort exceeds your list, the verdict flags it: raise the effect you want to detect or grow the list before testing.

Limits to keep in mind: the math assumes a binary metric (opened or not, clicked or not) and a homogeneous list. It does not cover revenue per email, send time, or the subject line effect on future engagement. Once you measure the winner, confirm the difference is real with the statistical significance calculator and read the context in the what is A/B testing guide.

Best practices for subject line testing

Before you fire, lock three things: the cohort per subject line, the minimum effect you accept, and the primary metric. Then follow these rules.

• One metric per test: open OR click, not both at once. Chasing several multiplies the false positive.
• Same-size cohorts drawn at random. A biased slice (only the most engaged in one cohort) contaminates the result.
• If the test consumes almost the whole list, the gain from sending the winner to the rest gets small. Revisit the effect you want to detect.
• Respect the base: detecting a +2% relative lift in the open rate costs a cohort most lists do not have.

Frequently asked questions

How many subscribers does a subject line A/B test need?

It depends on your current open rate and the lift you want to detect. The smaller the effect, the bigger the cohort. For a 25% open rate detecting a +10% relative lift (reaching 27.5%), at 95% confidence and 80% power, you need about 4,862 subscribers per subject line, or 9,724 across the whole test. Bigger effects cost much smaller cohorts.

Why does an email test often ask for such a large cohort?

Because the open rate is a binary metric with a high base and large variance, so telling 25% from 27.5% with confidence takes a lot of volume. That is why testing subject lines on a small list is different from testing on a big e-commerce list. If the cohort exceeds your list, raise the minimum effect or grow the list first.

Do I test with a slice of the list and send the winner to the rest?

Yes, that is the standard subject line A/B test. You send subject A to one cohort, subject B to another of the same size, measure the open rate, and the better subject goes to the remaining list. The calculator shows how much of the list the test consumes and how much is left for the winning send.

Can I use the calculator for click rate instead of open rate?

You can. The math is the same for any binary email metric: swap the base rate for your current click rate and the effect you want to detect. Since clicks usually have a much lower base than opens, the cohort tends to be even larger. Pick one primary metric per test.

How is this different from the site sample size calculator?

The site one thinks in visitors per day and test duration, because traffic arrives over time. With email you have a fixed list and a single send, so the question changes: how much of the list the test consumes and how much is left for the winner. The sample math is the same, the framing is email.

Keep going

Sized the cohort? Run the test, measure the open rate, and confirm the winner with the statistical significance calculator. To size a web page test instead of email, use the A/B test sample size calculator. The full context is in the what is A/B testing guide.