Multi-Armed Bandits for Email Subject Lines
Multi-armed bandits in email: auto-optimize subject line features are a simple bandit, not a rigorous A/B test. See the sample size math.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Most email marketers have already run a multi-armed bandit without knowing it. Mailchimp, HubSpot, Klaviyo, and ActiveCampaign all ship a subject line “auto-optimize” or “winner selection” feature: send two or three subject lines to a slice of the list, wait a few hours, then blast whichever looks best to everyone else. That is a real bandit pattern, the simplest one in the family, called explore-then-commit. It is not the same rigor as a full A/B test, and the gap matters most exactly when the decision is expensive: a small initial sample, a short evaluation window, and a single noisy metric (open rate) stand between “the tool declared a winner” and “the winner is real.”
This guide explains, with sources for every vendor-specific claim, what these auto-optimize features actually do under the hood, why the sample size they use is often only big enough to catch a huge difference rather than a realistic one, when that trade-off is the right call for email specifically, and when you should insist on a longer, click-based or conversion-based test instead.
What your email tool actually does when it “auto-optimizes” a subject line
Strip away the marketing language and the mechanism is the same across vendors: split a portion of the list between two or three subject lines, measure a single metric for a fixed window, then send whatever “won” to the rest of the list. The differences are in the details, and the details are exactly what decide whether the winner it picked is trustworthy.
- Mailchimp lets you test up to three variations of a single variable (subject line, from name, content, or send time), per its about A/B tests documentation. Its create an A/B test guide states that Mailchimp requires “you send to at least 10% of your recipients for any type of A/B test, so the slider won’t go below 10%,” recommends at least 5,000 recipients per combination “to get the most useful data,” and suggests waiting at least four hours after sending before the winning combination goes out. The winner can be chosen automatically by open rate, click rate, or total revenue, or picked manually.
- HubSpot lets you set your own test percentage and your own wait window in hours before the rest of the list gets the winning version, per its running an A/B test on a marketing email documentation. HubSpot recommends sending an A/B tested email to at least 1,000 contacts; a 50/50 split is allowed at any list size, but other split ratios need that 1,000-contact minimum, and if the final send list is too small, HubSpot skips the test entirely and sends only the single version you manually selected to everyone. Winner selection can use open rate, click rate, or click-through rate.
- Klaviyo uses an adjustable slider (its own A/B test campaign guide shows an example of 20 percent to variation A and 20 percent to variation B, with the remaining 60 percent held back for the winner) and, notably, offers an automatic option gated on a stated win probability of 90 percent or higher, not just the highest raw number, per its statistical significance in campaigns documentation. Klaviyo recommends open rate specifically “when testing subject line, preview text, or sender name,” and click rate for content changes.
- ActiveCampaign works the same way: you assign a percentage to each subject line variant, choose open rate or click rate as the deciding metric, and pick the number of hours before the remaining percentage receives the winner, according to its split test email campaign guide.
| ESP | Sample used for the test | Metric that decides the winner | Wait before the rest of the list gets it |
|---|---|---|---|
| Mailchimp | Slider from 10 percent up, across up to 3 variations; recommends 5,000+ recipients per combination | Open rate, click rate, or total revenue (if a store is connected) | User-chosen; vendor recommends at least 4 hours |
| HubSpot | User-chosen percentage; a 50/50 split is allowed at any size, other splits need 1,000+ contacts, below which HubSpot skips the test and sends only the manually chosen version | Open rate, click rate, or click-through rate | User-chosen number of hours |
| Klaviyo | Adjustable slider (example given: 20% and 20%, 60% held back) | Open rate (recommended for subject line), click rate, or placed order rate; automatic mode requires 90%+ win probability | Automatic once the threshold is met, or manual override |
| ActiveCampaign | User-assigned percentage per variant | Open rate or click rate | User-chosen number of hours |
Every one of these is, statistically, some flavor of explore-then-commit: spend a short exploration phase gathering data across all arms, then irrevocably commit all remaining traffic to whichever arm looked best. It is the simplest member of the multi-armed bandit family, simpler even than epsilon-greedy, and much simpler than Thompson Sampling, which keeps drawing from each arm’s belief distribution and gradually shifts allocation instead of flipping a single switch after a fixed clock runs out. Klaviyo’s win-probability gate is the one vendor detail here that borrows directly from Bayesian bandit logic; the rest pick whichever raw number is bigger once the timer expires, with no significance check at all.
The timeline nobody shows you in the product screenshot
The compressed timeline is not an accident or a corner someone forgot to fix, it is the entire point of the feature. Email decays fast: most opens happen in the first few hours, and by the next day the audience has mostly already decided whether to engage. Waiting two weeks for a formally significant verdict, the way this blog’s guide to bandits versus A/B testing recommends for a web page test, would mean the newsletter is stale before the “winner” ever reaches most of the list. Speed is a legitimate constraint here in a way it usually is not for a landing page. The question is not whether the shortcut is justified, it clearly is for a lot of routine sends, it is whether marketers understand what they are trading away when they take it.
Why open rate is a weaker signal than it used to be
Subject line tests are judged on open rate almost everywhere, for a sound reason: a subject line only changes one thing, whether someone opens the email, so open rate is the metric closest to the actual manipulation. The problem is that open rate itself has gotten noisier since Apple shipped Mail Privacy Protection in 2021. Apple’s proxy servers pre-fetch the tracking pixel on the company’s own infrastructure the moment an email arrives, before a human ever looks at it, which registers as an “open” whether or not the recipient reads anything. Mailchimp addresses this directly in its own Apple Mail Privacy Protection FAQ, and Litmus, which tracks email client market share from over a billion measured opens, reports that Apple Mail’s share now covers more than half of all opens industry-wide, meaning a large and growing fraction of every subject line test’s “winning” metric is not a genuine human signal at all.
None of this makes open rate useless. For a subject line specifically, it is still the metric that isolates the variable you changed. But it means a subject line “winner” declared on open rate carries an extra layer of noise on top of the ordinary sampling noise from a small test group, and it is one more reason a raw percentage difference on a short clock deserves more scrutiny than the dashboard’s green checkmark implies.
A worked example: when the sample is too small to know anything
Here is where theory turns into arithmetic, using the same statistics engine (sampleSizePerVariant and the underlying two-proportion z-test in stats.ts) that runs every calculator on this blog.
The scenario: a list of 20,000 subscribers, a subject line with a baseline open rate of 20 percent, and a variation B that, in reality, though the marketer has no way to know this in advance, opens at 23 percent, a genuine 15 percent relative lift. Following a typical ESP’s default, the auto-optimize test sends to 10 percent of the list, 2,000 subscribers, split evenly: 1,000 see subject A, 1,000 see subject B. The remaining 18,000 will get whichever looks better.
Expected opens in the test group: A gets roughly 1,000 times 20 percent, about 200 opens. B gets roughly 1,000 times 23 percent, about 230 opens. Running the two-proportion z-test on those numbers:
- Pooled rate: (200 + 230) divided by 2,000 equals 21.5 percent.
- Standard error: the square root of 0.215 times 0.785 times (1/1,000 + 1/1,000), about 0.0184.
- z-score: (0.23 minus 0.20) divided by 0.0184, about 1.63.
- Two-sided p-value: about 0.10.
A p-value of 0.10 is well above the conventional 0.05 threshold. The test is not statistically significant, even though the underlying lift is entirely real. Most ESPs do not check for this: they compare the two raw open rates after the clock runs out and send whichever number is bigger, in this case B, without ever computing whether 230 out of 1,000 is meaningfully different from 200 out of 1,000. Klaviyo’s automatic mode is the exception, since its 90 percent win-probability gate would very likely hold this exact result back from an automatic send.
So what sample would have actually been enough? Plug the same numbers into the calculator below (baseline 20 percent, minimum detectable effect 15 percent relative, 95 percent confidence, 80 percent power, the market-standard rigor this blog’s bandit versus A/B testing guide uses everywhere else) and check the sample size it returns:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
The formula returns 2,943 subscribers per variant, 5,886 in total, almost three times the 2,000 the auto-optimize test actually used. Ignore the estimated day count the calculator also shows, it assumes continuous week-over-week traffic the way a website does, and an email blast is a one-shot send rather than accumulating traffic over days; the subscriber count is the number that matters here.
It gets more revealing from the other direction. With only 1,000 subscribers per variant, the minimum lift this test could reliably detect at 95 percent confidence and 80 percent power works out to roughly 25 percent relative, about 5 percentage points on a 20 percent baseline, using the same formula run backward (mdeForSample in stats.ts). In plain terms: the ESP’s fast test was never built to catch a realistic 15 percent improvement. It was only ever powerful enough to catch something close to a 25 percent swing or bigger.
None of this means the ESP’s number is wrong or dishonest, B genuinely did convert better in this scenario, and sending it to the rest of the list was, on the raw numbers, the better bet. What it means is that the tool cannot tell you, and mostly is not trying to tell you, whether that 15 percent lift is something you can count on the next time, versus a coin flip that happened to land on the side that also matches the truth. That distinction is invisible from inside the dashboard.
When the shortcut is a reasonable trade, and when to insist on a real test
Neither approach is universally right. The question is what a specific send needs.
| Situation | Auto-optimize is a reasonable trade | Insist on a full click or conversion based A/B test |
|---|---|---|
| List size | Large list (tens of thousands or more), where even a small percentage is still a real sample | Small list, where even a 100 percent split barely reaches a few hundred recipients per side |
| What is riding on the send | Routine newsletter, low stakes if the “wrong” subject line reaches most of the list | High-stakes campaign: a launch, a renewal push, a price change, a win-back sequence |
| Metric that matters | Open rate genuinely is the goal, brand recall or re-engagement | A click, a signup, or a purchase is the real goal, and open rate is only a rough proxy for it |
| How fast the send has to go out | Time-sensitive content (breaking news, a flash sale) where speed beats certainty | You can run the send over days and need an answer that holds up to scrutiny |
| Need to defend the decision later | No one will ever ask for the p-value | The result gets reported, repeated across campaigns, or cited to justify a bigger investment |
Email is one of the few channels where “good enough, fast” is frequently the correct engineering trade-off, because opens and clicks decay within hours and a subject line rarely carries the kind of downstream risk a checkout redesign does. But the moment a send is expensive to get wrong, whether because the list is small, the metric is revenue rather than opens, or the result will be quoted in a board deck, the honest move is to run the send as a real, longer, click-based or conversion-based A/B test, sized the way the calculator above sizes it, not to trust a coin-flip-sized sample just because the dashboard reported a winner in six hours.
Automate This on Donnu
Subject line auto-optimize tools solve for speed, not certainty, and that trade is invisible unless you do the arithmetic yourself, the way the worked example above just did. Donnu runs on a native Bayesian engine built for the opposite priority: honest read of whether a lift is real, not just which raw number happened to be bigger after a short clock ran out. When a subject line, an email, or any page variation is worth defending with a real answer rather than a fast guess, size the test properly first with the calculator on this page, then run it long enough, on the metric that actually matters, to know the difference between a lucky sample and a genuine improvement.
Read the full picture in Multi-Armed Bandits vs A/B Testing and see the mechanics behind adaptive allocation in Thompson Sampling Explained. Start a free 14-day trial and size your next test on statistics that hold up, not just a raw number after a few hours.
References
- Mailchimp. About A/B Tests. mailchimp.com/help/about-ab-tests; Create an A/B Test. mailchimp.com/help/create-ab-tests.
- HubSpot. Run A/B tests for marketing emails. knowledge.hubspot.com.
- Klaviyo. How to A/B test an email campaign. help.klaviyo.com; Understanding statistical significance in Klaviyo campaigns. help.klaviyo.com.
- ActiveCampaign. How to create a Split Test email campaign. help.activecampaign.com.
- Mailchimp. Apple Mail Privacy Protection (MPP) FAQs. mailchimp.com/help/apple-privacy-faq.
- Litmus. Email Client Market Share: Trends and Statistics. litmus.com/email-client-market-share.
Frequently asked questions
- Is the "auto-optimize" subject line feature in my email tool actually a multi-armed bandit?
- Functionally, yes, though a simple one. Most email service providers send two or three subject lines to a small slice of the list, wait a few hours, then blast whichever looks best by open rate to everyone else. That is a two-phase pattern called explore-then-commit, the simplest member of the bandit family. It is not the same as an adaptive bandit like Thompson Sampling, which keeps shifting traffic gradually as evidence accumulates instead of making one irreversible switch.
- Is a 10 percent test slice enough to trust an email subject line "winner"?
- Usually not for a realistic, moderate lift. As the worked example on this page shows, a 1,000-recipient-per-variation test can reliably detect only a large swing in open rate, on the order of 25 percent relative or more, at standard 95 percent confidence and 80 percent power. A genuine but more modest improvement, like 15 percent, often will not clear statistical significance in that sample, even though it is real.
- Why do email tools use open rate instead of click rate to judge a subject line test?
- Because a subject line only influences one decision: whether the recipient opens the email at all, so open rate is the metric closest to what changed. The trade-off is that open rate has become a noisy proxy since Apple introduced Mail Privacy Protection, which pre-fetches tracking pixels on the company's servers and inflates opens regardless of whether a human read the email. Litmus reports that more than half of all email opens now happen on a device covered by Apple MPP.
- When should I skip auto-optimize and run a full click-based or conversion-based A/B test on an email instead?
- Insist on a real test when the send is high-stakes (a launch, a renewal push, a price change), when your list is small enough that even a 100 percent split barely reaches a few hundred recipients per side, or when the metric that actually matters is a click, a signup, or a purchase rather than an open. Auto-optimize is a reasonable trade for routine, large-list sends where speed matters more than a defensible verdict.