Experimentation Velocity: How to Calculate Yours (Template)
Free template to calculate experimentation velocity: statistical ceiling, real cycle time, utilization and the four numbers that actually matter.
📚 This article is part of the guide How to Build an Experimentation Culture (2026).
Experimentation velocity is how many valid experiments your team completes per year, measured end to end. Almost everyone calculates it wrong, because they count only the days the test was live and ignore the days spent building, waiting for approval, and sitting on a finished result nobody has decided on. The result is a program that believes it has capacity for 26 tests a year and ships 11. This article gives you the full template for the two numbers that matter, the statistical ceiling your traffic allows and the real velocity your process delivers, plus the indicator that comes out of the ratio between them and points at exactly where the program is leaking. It is part of the guide on how to build an experimentation culture.
The two numbers that make up experimentation velocity
Experimentation velocity is not one number, it is two, and the distance between them is the diagnosis.
Statistical ceiling. How many tests your traffic can fund per year, if the process were instant. It comes from required sample divided by daily traffic, respecting the two week duration floor. It is a physical limit: no process, however good, gets past it.
Real velocity. How many tests the team actually completes per year, counting the full cycle of each one. It comes from 365 divided by average cycle time.
Utilization. The ratio between the two. This is where the conversation gets useful, because it tells you whether the next gain comes from buying traffic or from fixing the process. Utilization of 40% means 60% of your traffic’s potential is being burned on friction, and no audience increase recovers that.
Step 1: calculate the statistical ceiling
The ceiling comes from three inputs: the baseline conversion rate of the flow, the minimum detectable effect you want to be able to see, and daily traffic in that flow. Worth repeating: it is traffic in the tested flow, never sitewide traffic.
Run it with your own numbers:
Sample per variation at 95% confidence and 80% power (two-sided), from your rate and MDE. Tweak the fields and watch capacity update live.
One detail of the ceiling surprises almost everyone the first time: past a certain point, getting more ambitious on MDE does not buy more tests. In the worked example below, at 90,000 visitors a month, sizing for +15% relative and sizing for +25% relative give exactly the same capacity, because in both cases the two week floor is what binds. The extra ambition becomes pure loss of sensitivity, with no gain in cadence.
Step 2: measure cycle time honestly
Cycle time is the clock that starts when the hypothesis is approved and stops when the decision is logged. It has four pieces, and three of them collect no data.
| Cycle stage | What it counts | Sign that this stage is your bottleneck |
|---|---|---|
| Build | From approved hypothesis to variation live | The queue of ready to run tests is always empty |
| Collection | From start to end of the fixed window | The test runs less than the two week floor |
| Read and decide | From end of collection to logged decision | Finished results waiting for a meeting |
| Idle | Days between the decision and the next test starting | The slot sits empty while the next variation is debated |
Log all four for every completed test. Three or four tests are enough for the average to be informative, and the pattern shows up fast: in most young programs, build and idle together beat collection.
One measurement detail changes the result: the idle stage is the easiest to forget, because nobody owns it. It shows up in no ticket, it has no assignee, and so it rarely makes it into the math. It is also, frequently, the largest of the four.
Step 3: the fillable template
Copy the table below into your spreadsheet. The first three rows are inputs, the rest is arithmetic.
| Field | How to fill it | Example |
|---|---|---|
| Monthly visitors in the flow | Only people entering the tested flow | 90,000 |
| Baseline conversion rate | Primary metric of the flow | 3.5% |
| Target MDE | Relative effect worth detecting | +15% |
| Sample per variant | Calculator output | 20,622 |
| Total sample (2 variants) | Sample per variant times 2 | 41,244 |
| Daily traffic | Monthly visitors divided by 30 | 3,000 |
| Collection days | The larger of total sample divided by daily traffic and the floor | 14 |
| Annual statistical ceiling | 365 divided by collection days | 26.1 |
| Build days (average) | Measured across recent tests | 8 |
| Read and decide days (average) | Measured across recent tests | 4 |
| Idle days (average) | Measured across recent tests | 8 |
| Cycle time | Sum of the four stages | 34 |
| Annual real velocity | 365 divided by cycle time | 10.7 |
| Utilization | Real velocity divided by ceiling | 41% |
| Tests with a logged hypothesis | Share in the quarter | log it |
| Documented results | Share in the quarter, ties included | log it |
The last two rows are not part of the velocity math and they are there on purpose. Velocity without hypothesis quality is just producing ties faster, and a cadence number displayed on its own becomes a perverse target within weeks.
Worked example: a SaaS with 90,000 monthly visitors
The signup flow receives 90,000 visitors a month, converts 3.5% into trial starts, and the team sizes to detect +15% relative at 95% confidence and 80% power.
- Sample per variant: 20,622 visitors.
- Total sample (2 variants): 41,244 visitors.
- Daily traffic: 90,000 divided by 30, that is 3,000 visitors a day.
- Days required by traffic: 41,244 divided by 3,000 gives 13.7, rounded up to whole collection days, that is 14 days.
- Real duration: 14 days, tying with the floor.
- Statistical ceiling: 365 divided by 14, roughly 26 tests per year.
Check the sample against your own baseline rate:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Now the other side. Measuring this team’s last four tests: 8 build days, 14 collection days, 4 read and decide days, 8 idle days until the next one starts. Cycle time of 34 days.
- Real velocity: 365 divided by 34, roughly 10.7 tests per year.
- Utilization: 10.7 divided by 26.1, roughly 41%.
The team is leaving 15 tests a year on the table, and not one of them requires a single extra visitor. Note what the math says about the more expensive alternative: if this same team doubled flow traffic to 180,000 visitors a month, the ceiling would stay at 26 tests a year, because collection is already at the two week floor. The extra traffic would buy sensitivity (the ability to see smaller effects), which is valuable, but it would not buy a single additional test. Cadence only improves by attacking the 20 days outside collection.
What to do with each bottleneck
| Bottleneck | Lever | Real cost |
|---|---|---|
| Long build | Keep the next variation ready before the slot frees up | Requires deciding the queue in advance |
| Long build | Standardize the most frequent changes into reusable components | Upfront engineering investment |
| Collection above the floor | Move the primary metric up the funnel or raise the target MDE | Loses sensitivity or measures a proxy |
| Slow reads | Write the decision rule before running | Requires agreeing on criteria while nobody knows the result |
| Idle between tests | Parallelize across independent flows | Only works if the flows do not share an audience |
| Idle between tests | Book the slot in the calendar before the previous test ends | Requires treating the slot as a resource, not an intention |
The most underrated lever is the decision rule written before the test. It shortens the read stage because it removes the most expensive argument in the cycle, which is deciding the criteria after you already know the result. As a side effect it also protects against primary metric swapping, one of the quiet paths to inflating a program’s win rate. The statistical cost of deciding after seeing the data is detailed in the peeking problem in A/B testing.
A warning about parallelizing: running two tests at once raises real velocity without touching individual cycle time, but only when the two flows do not compete for the same audience and the same primary metric. Two tests in the same journey split the sample across four combinations, stretch both, and bake the interaction into both results. The rule of thumb is one test per flow.
What 41% utilization costs
Process arguments usually lose to media arguments because the cost of friction is never expressed as a number. Here it is easy to express.
The example team leaves 15.4 tests a year on the table (26.1 of ceiling against 10.7 of real velocity). Applying the proportion Ronny Kohavi documented for experiments at Microsoft, roughly one third of tested ideas improve the target metric (Kohavi, KDD 2015), those 15.4 unrun tests amount to about 5 improvements a year the team will never find, plus another 5 bad changes it will not catch in time to avoid.
The third block is the easiest to forget and usually the most expensive. Tests that do not run do not put the change on hold: the change ships anyway, unmeasured, and the damage surfaces months later as a metric drop nobody can attribute.
Two traps when shortening the cycle
Shortening the cycle is the right lever, but two ways of shortening it destroy the program instead of accelerating it.
Shortening collection. It is the obvious temptation, because collection is the largest contiguous slice of the cycle. It is also the only slice that produces data. Cutting collection from 14 days to 8 raises cadence on paper and drops the power of the test, which raises the apparent win rate while lowering the share of wins that are real. The other three stages can be compressed at zero statistical cost; this one cannot.
Building before the hypothesis is approved. Keeping the next variation ready is good practice, but only after the hypothesis, the primary metric and the duration are written down. Building before that inverts the order: the team starts picking the hypothesis that justifies the variation that already exists, and the roadmap becomes a queue of finished things instead of a queue of questions.
Cycle time is also a hiring argument
One use of this template that rarely gets made explicit: the four stage breakdown is the cleanest input for deciding what the next hire on the program should do. A team whose cycle is dominated by build days is describing an engineering or design constraint. A team whose cycle is dominated by read and idle days is describing a decision making constraint, and hiring another builder makes the queue longer without making the program faster.
The same breakdown answers the tooling question. Buying a platform shortens build and read; it does nothing for approval days that live in someone else’s calendar. Sizing that gap before signing a contract is what keeps the tool from being blamed for a bottleneck it was never able to touch. If you want the full picture of what the constraint looks like from the outside, common organizational blockers to A/B testing maps the ones that are structural rather than technical.
How to track it without creating a perverse target
Velocity is a means metric, not an ends metric, and displaying it alone on a dashboard produces the wrong behavior within weeks: tests shortened to fit the month, loose hypotheses to fill the slot, ties reclassified as wins. Three habits prevent that:
- Always report it in a pair. Velocity next to the share of tests with a hypothesis logged before running. One number rises at the expense of the other as soon as the program starts rushing.
- Measure by year, not by month. With few tests in the window, a single result moves the whole average. Annual reading is the only stable one for anybody running fewer than 20 tests per period.
- Separate ceiling from velocity in the report. They are different problems with different owners: the ceiling is a conversation about traffic and MDE ambition, velocity is a conversation about process. Mixing them makes the team ask for media budget when the problem was the queue.
To order the queue this capacity supports, the experimentation roadmap template already carries the estimated duration column, which is the bridge between this calculation and planning. And to know what to expect from the tests that fit, it is worth reading the average A/B test win rate and what the data shows, because cadence multiplied by an unrealistic expectation of success produces a promise the math does not support.
Make this automatic with Donnu
Half of this template is arithmetic the tool should be doing for you. Donnu sizes every test against your real traffic and shows the estimated duration before you start, which delivers the statistical ceiling without a spreadsheet, and it keeps the duration fixed so the collection stage is not cut short mid flight by a glance at the dashboard. What is left for the team to measure is the days outside collection, which is exactly where the velocity gain nobody buys with media is sitting.
Start a 14 day free trial and see the duration of each test before you commit the quarter. To size the cadence your traffic can fund, see how many A/B tests you should run per month.
References
- Kohavi, R. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. Keynote, ACM SIGKDD 2015. exp-platform.com/Documents/2015-08OnlineControlledExperimentsKDDKeynoteNR.pdf.
- Thomke, S. Building a Culture of Experimentation. Harvard Business Review, March-April 2020. hbr.org/2020/03/building-a-culture-of-experimentation.
- Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.
- Bing Search Quality Insights. Large Scale Experimentation at Bing. Microsoft. blogs.bing.com/search-quality-insights/August-2013/Large-Scale-Experimentation-at-Bing.
- Kohavi, R. ExP Platform: accelerating innovation through trustworthy experimentation. exp-platform.com.
Read also:
- How to Build an Experimentation Culture: A Practical Framework
- The Experimentation Roadmap Template (Free)
- How Many A/B Tests Should You Run Per Month?
Leia em português: Velocidade de Experimentação: como calcular a sua (template)
Frequently asked questions
- What is experimentation velocity?
- It is how many valid experiments a team completes per unit of time, measured end to end and not just by the days the test was live. The distinction matters because most of the slowness in a program is not in data collection: it is in build days, approval waiting days and finished tests nobody has read. A program running 14 day tests that ships 10 experiments a year has a process problem, not a traffic problem.
- How do you calculate experimentation velocity?
- Two numbers and one division. The statistical ceiling is 365 divided by the duration of each test, where duration is the larger of required sample divided by daily traffic and the two week floor. Real velocity is 365 divided by full cycle time, which adds build days, collection days, read and decide days, and idle days. Utilization is the ratio between the two, and it almost always lands well below 100%.
- What is a good experimentation velocity?
- The useful benchmark is not an absolute number, it is your own ceiling. A team using 70% of the capacity its traffic allows is doing well even at 12 tests a year; a team using 35% of a ceiling of 26 is losing half of its program to friction that no traffic increase repairs. Compare yourself to your ceiling before comparing yourself to any company.
- What do you do when the bottleneck is build time, not traffic?
- Attack the queue, not test duration. The levers that work: keep a variation already built waiting for the slot to free up, standardize the most frequent changes into reusable components, and separate the decision to launch from the end of collection by writing the decision rule before you start. None of them requires more traffic, and all three shorten the cycle without shortening the test.
- Does raising experimentation velocity always raise results?
- No. Velocity multiplies hypothesis quality, so doubling the cadence of a program with weak hypotheses just produces more ties faster. According to data published by Ronny Kohavi on experiments at Microsoft, roughly one third of tested ideas improve the target metric, one third are neutral and one third make it worse. Accelerating without improving where hypotheses come from mostly accelerates the last two thirds.
- How often should you recalculate the velocity template?
- Once a quarter for the statistical ceiling, because traffic and baseline rate move slowly, and after every completed test for cycle time, which is where improvements show up. Logging cycle time test by test is what turns the template from a one off diagnosis into a metric you can track.