How Many A/B Tests Should You Run Per Month?
How many A/B tests per month your traffic can support: the capacity formula, a live calculator and a worked example from sample size to annual wins.
📚 This article is part of the guide How to Build an Experimentation Culture (2026).
The honest answer to “how many A/B tests should we run per month” is not a benchmark number: it is the capacity your traffic supports, and it comes out of a two-line calculation. The duration of each test is the larger of the required sample divided by daily traffic and a floor of two weeks; monthly capacity is the number of days in the month divided by that duration. This article shows the math, explains why the numbers published by Booking.com and Microsoft are not a target for almost anyone, identifies the point where extra traffic stops buying more tests and starts buying more sensitivity, and gives you live calculators plus a worked example end to end. It is part of the guide on how to build an experimentation culture.
Why the industry benchmark is not a target
Every article about experimentation velocity quotes the same giants. Microsoft documented the Bing experimentation program publicly, with thousands of experiments running at scale (Bing Search Quality Insights, Large Scale Experimentation at Bing). Booking.com is routinely cited for keeping hundreds of tests live at once, a case Stefan Thomke documented in Building a Culture of Experimentation (Harvard Business Review, 2020).
Those numbers describe a condition almost nobody has: tens of millions of sessions per month, a dedicated in-house platform and entire teams working on experimentation infrastructure. Copying the target without copying the traffic produces the worst possible outcome: an artificially high cadence, tests stopped early so they “fit the month”, and a sequence of decisions made on noise.
The useful question is not “how many tests does Booking run”. It is “how many tests can my traffic support with statistical rigor”. That second question has an exact numeric answer.
The capacity formula
Three inputs and two operations:
- Sample per variation. Depends on the baseline conversion rate, the minimum detectable effect (MDE) and the rigor you choose (market standard: 95% confidence and 80% power). It is the same calculation covered in the guide on how to run an A/B test step by step.
- Total sample for the test. Sample per variation multiplied by the number of variations (2 in a classic A/B).
- Daily traffic on the tested flow. Not sitewide traffic: only the visitors who enter the flow where the test runs.
With that: days per test = the larger of (total sample divided by daily traffic) and the duration floor, and tests per month = 30 divided by days per test.
The duration floor deserves an explanation. Even if traffic closes the sample in three days, ending a test in three days measures only Tuesday-to-Thursday behavior, and weekend audiences tend to convert differently. Two weeks is the most common market floor because it captures two full weekly cycles. That floor is what splits the calculation into two distinct regimes, and the second one surprises a lot of people.
How many A/B tests per month your traffic supports
The table below applies the formula to a common scenario: a 2.5% baseline conversion rate, two variations, a 14-day floor, and the ambition to detect a 20% relative effect (which requires 16,792 visitors per variation, 33,584 in total).
| Monthly visitors on the flow | Days bound by traffic | Actual duration per test | Tests per month | Bottleneck |
|---|---|---|---|---|
| 20,000 | 51 | 51 days | 0.6 | Traffic |
| 60,000 | 17 | 17 days | 1.8 | Traffic |
| 200,000 | 6 | 14 days | 2.1 | Calendar |
| 600,000 | 2 | 14 days | 2.1 | Calendar |
Sample by normal approximation of two proportions (95% confidence, 80% power, two-sided), 2 variations, 14-day floor, 30-day month.
Two readings come out of that. First: between 200,000 and 600,000 monthly visitors, capacity does not change at all. Tripling traffic bought zero additional tests, because the two-week floor was already the binding limit. Second: the 20,000-visitor site is not condemned to 0.6 tests per month forever, it simply cannot detect a 20% effect in that window. Chasing a bigger effect, the same operation fits the calendar again.
Compute your real capacity with your own baseline, your MDE and your traffic:
Sample per variation at 95% confidence and 80% power (two-sided), from your rate and MDE. Tweak the fields and watch capacity update live.
Capacity is a property of the flow, not of the company
The most common way to overpromise cadence is to run the formula on sitewide traffic. Capacity belongs to the flow where the change lives, and flows are much smaller than the site. A single ecommerce site usually looks like this:
| Flow | Share of sitewide traffic | Monthly visitors | What can be tested there |
|---|---|---|---|
| Home and landing pages | 100% of entries | 500,000 | Value proposition, hero, navigation |
| Category and search | ~45% | 225,000 | Sorting, filters, listing density |
| Product page | ~30% | 150,000 | Gallery, social proof, shipping info |
| Cart | ~9% | 45,000 | Order summary, upsell, urgency |
| Checkout | ~6% | 30,000 | Form fields, payment options, trust signals |
One site, five very different capacities. On the home page, the two-week floor binds almost immediately. On checkout, at 30,000 visitors a month and a 2.5% baseline, a 20% relative MDE needs 34 days per test, which is under one test per month. This is why a single sitewide “we run 4 tests a month” target is meaningless: the same team is calendar bound at the top of the funnel and traffic bound at the bottom, in the same quarter.
The practical consequence for planning: compute capacity per flow, and let the roadmap allocate slots per flow instead of a single global number. The experimentation roadmap template carries a duration column precisely so this conversation happens before the commitment, not after.
What to do with surplus traffic
When the bottleneck becomes the calendar, the traffic left over has a better use than “end the test on day 6”: buying sensitivity. With more sample inside the same 14-day window, the same test starts detecting smaller effects, and smaller effects are most of the real effects.
In the table scenario, the site with 200,000 monthly visitors that lowers its ambition from +20% to +10% relative now needs 64,199 visitors per variation. That stretches duration to 20 days and drops capacity from 2.1 to 1.5 tests per month. It is a deliberate trade and almost always a good one: fewer tests per month, but each one able to see improvements that would previously have read as a tie.
| Test ambition | Sample per variation | Duration at 200,000 per month | Tests per month |
|---|---|---|---|
| Detect +20% relative | 16,792 | 14 days (floor) | 2.1 |
| Detect +10% relative | 64,199 | 20 days | 1.5 |
In other words: “how many tests per month” and “how small an effect can I see” are the same question viewed from two sides. Fixing one defines the other.
The two-week floor is not superstition
Teams under cadence pressure often treat the duration floor as a nice-to-have and cut it first. It is worth knowing exactly what it protects against, because the mechanism is not statistical noise, it is composition.
Conversion rate varies systematically by day of week: business-hours traffic on a B2B site behaves differently from weekend traffic, paid campaigns often run on weekday schedules, and payday cycles move ecommerce. A test that runs Tuesday to Thursday does not just have a smaller sample, it has a biased sample, measuring a population that is not the one your change will eventually serve. More traffic does not fix that, because the missing days are missing at any sample size.
A practical refinement most teams can adopt immediately: make the floor whole weeks, not “14 days”. A test that starts on a Wednesday and ends 14 days later covers two Wednesdays and only one of every other day, which reintroduces a smaller version of the same problem. Starting and ending on the same weekday keeps every day of week equally weighted.
Worked example: a SaaS with 60,000 visitors per month
A SaaS product gets 60,000 monthly visitors on its signup landing page, converting at 2.5% into trial starts. The team wants to detect a 20% relative gain (from 2.5% to 3.0%).
- Sample per variation: 16,792 visitors.
- Total sample for the test (2 variations): 33,584 visitors.
- Daily traffic: 60,000 divided by 30, which is 2,000 visitors per day.
- Days required by traffic: 33,584 divided by 2,000, which is 17 days.
- Actual duration: 17 days, because 17 is greater than the 14-day floor.
- Capacity: 30 divided by 17, approximately 1.8 tests per month, or about 21 tests per year.
Check the first step of that calculation with your own baseline rate:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Now the part that changes decisions. This team has an annual capacity of 21 tests. Applying the proportion Ronny Kohavi documented for experiments at Microsoft, roughly one third of ideas improve the target metric, one third are neutral and one third make it worse (Kohavi, Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years, KDD 2015), so the realistic expectation is about 7 wins per year, several of them small.
That number tends to alarm people, which is exactly why it belongs in the plan. A roadmap that promises “we will lift conversion 40% this year with A/B tests” while holding a capacity of 21 tests is promising something the math does not support. A roadmap that plans 21 tests, expects 7 wins and treats each as a compounding gain is being honest about its own capacity.
Parallelizing without contaminating the reading
The obvious way to double cadence is running two tests at once, and it works under one condition: the tests must not compete for the same audience on the same decision. Two experiments on independent flows (one on the pricing page, one on the password recovery flow) coexist without trouble, because most visitors of one never see the other.
The dangerous case is the opposite: two tests in the same journey, or two tests moving the same primary metric. Three problems appear at once. The sample splits across four combinations instead of two, which stretches both tests rather than accelerating the program. The interaction between the changes gets baked into the result, and neither report can separate its own effect from the neighbor’s. And if both “win”, shipping both together can deliver less than the promised sum, because part of the gain was the same gain counted twice.
Two ways to organize this without giving up parallelism: mutual exclusion, where each visitor enters at most one experiment and the groups stay fully separated, or orthogonal layers, where the allocation in one test is drawn independently of the other, keeping the combinations balanced. The first is simpler and more conservative, the second uses traffic better. Under either one, the practical rule holds: one test per flow, and the decision to parallelize taken at planning time, not mid-execution.
If your capacity is low
A result of 0.6 tests per month does not mean “do not test”. It means the strategy has to change shape:
- Test big changes, not fine-tuning. With little sample, only large effects are detectable. A new value proposition has a chance of producing an effect of 25% or more; a button color, almost never. The complete guide to conversion rate optimization covers which kinds of change tend to produce effects large enough to be detectable.
- Move the primary metric up the funnel. Measuring signup started (high baseline rate) instead of purchase completed (low baseline rate) cuts the required sample by orders of magnitude. The cost is measuring a proxy for the final outcome, which has to be documented.
- Group similar pages into one test. If three landing pages share the same structure, testing the change on all three at once pools their traffic and shortens duration proportionally.
- Accept longer windows for fewer tests. Two well-sized tests per quarter are worth more than six tests stopped halfway, which mostly produce false positives.
There is a deeper version of this covered separately in CRO for low-traffic sites, including which techniques genuinely help and which ones only look like they do.
Common mistakes when setting cadence
| Mistake | Warning sign | Fix |
|---|---|---|
| Copying a giant’s benchmark | “Booking runs thousands, we will run 20 a month” | Compute capacity from your traffic, not from someone else’s target |
| Stopping early to fit the month | Test halted on day 8 because “it already hit significance” | Fix duration before starting and respect the two-week floor |
| Counting sitewide traffic | Used 500,000 sessions when the tested flow gets 40,000 | Use only the traffic entering the tested flow |
| Ignoring the duration floor | A 4-day test because traffic closed the sample | Run at least one to two full weekly cycles |
| Two tests on the same flow | Two simultaneous experiments in checkout | One test per flow; use distinct flows to parallelize |
| Measuring velocity instead of learning | A dashboard with “tests run this month” as the headline KPI | Also measure how many tests had a defensible hypothesis and what each one taught |
Make this automatic with Donnu
Working out how many tests fit in your month means computing sample size, dividing by real traffic, respecting the duration floor and reordering the roadmap whenever the math changes. Donnu handles that part for you: when you configure a test, it sizes it against your real traffic, shows the estimated duration before you start, and tracks collection without letting a glance at the dashboard turn into a premature decision. You plan cadence with numbers instead of hope.
Start a 14-day free trial and find out your traffic’s real capacity before promising a result. To organize the queue that capacity supports, see the experimentation roadmap template.
References
- Kohavi, R. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. Keynote, ACM SIGKDD 2015. exp-platform.com/Documents/2015-08OnlineControlledExperimentsKDDKeynoteNR.pdf.
- Bing Search Quality Insights. Large Scale Experimentation at Bing. Microsoft. blogs.bing.com/search-quality-insights/August-2013/Large-Scale-Experimentation-at-Bing.
- Thomke, S. Building a Culture of Experimentation. Harvard Business Review, March-April 2020. hbr.org/2020/03/building-a-culture-of-experimentation.
- Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.
- Kohavi, R. ExP Platform: accelerating innovation through trustworthy experimentation. exp-platform.com.
Read also:
- How to Build an Experimentation Culture: A Practical Framework
- The Experimentation Roadmap Template (Free)
- How to Run an A/B Test: The Complete Step-by-Step Guide
Leia em português: Quantos Testes A/B Você Deveria Rodar por Mês?
Frequently asked questions
- How many A/B tests per month should a company run?
- There is no universal number: the right answer is the capacity your traffic supports. It comes from one division: days in the month divided by the duration of each test, where duration is the larger of the required sample divided by daily traffic and a floor of two weeks to capture a full weekly cycle. A site with 60,000 monthly visitors on the tested flow, a 2.5% conversion rate and the ambition to detect a 20% relative lift gets about 1.8 tests per month. A site with 200,000 reaches 2.1, limited by the calendar rather than by traffic.
- Does running more A/B tests per month mean growing faster?
- Only up to a point, and never on its own. Velocity multiplies hypothesis quality: if most of your tests have no defensible hypothesis, raising the cadence only produces more ties, faster. According to data published by Ronny Kohavi on experiments at Microsoft, roughly one third of tested ideas improve the target metric, one third change nothing and one third make it worse. Velocity without hypothesis quality only accelerates those last two thirds.
- My site has low traffic. Should I give up on testing?
- No, but you need to change what you test. With low traffic only large effects are detectable, so radical-change tests (a new layout, a new value proposition, a new offer structure) make more sense than button-color tweaks. It also helps to move the primary metric to a more frequent event higher in the funnel, such as signup started instead of purchase completed.
- Why does more traffic stop adding tests past a certain point?
- Because the minimum duration becomes a calendar limit, not a sample limit. Every test should run at least one or two full weeks to capture the variation between weekdays and weekends. Once traffic closes the sample in three days, the two-week floor takes over, and the surplus traffic starts buying sensitivity (detecting smaller effects) instead of more tests.
- Can I run two A/B tests at the same time to double my cadence?
- Yes, as long as they do not compete for the same flow or the same primary metric. Two tests on distinct pages of distinct journeys coexist fine. Two tests on the same page share visitors, and the interaction between the changes contaminates both readings, on top of splitting the sample and stretching both tests.
- Should capacity be counted for the whole site or per flow?
- Per flow. The sample that matters is the traffic that actually enters the page or journey where the change lives. A site with 500,000 monthly sessions whose checkout receives 40,000 has checkout capacity computed on 40,000, not on 500,000. Counting site-wide traffic is the single most common way teams end up promising a cadence they cannot deliver.