Average A/B Test Win Rate: What the Data Shows
The average A/B test win rate in published data runs from 8 to 33 percent, why the range is that wide, and why a high win rate is usually a warning sign.
📚 This article is part of the guide How to Build an Experimentation Culture (2026).
The A/B test win rate, meaning the share of experiments that improve the target metric, has no universal benchmark. The most citable public data opens a wide range: Ronny Kohavi documented that at Microsoft roughly one third of tested ideas improve the metric, one third are neutral and one third make things worse; Kohavi, Deng and Vermeer compiled success rates already published by different operations and landed on a range running from 8 to 33 percent. This article covers what each of those numbers actually measures, why they diverge so much, why a high rate is usually a symptom of a loose process, and above all the calculation almost nobody runs: how many of your winners are false positives, given that most ideas do not work. It is part of the guide on how to build an experimentation culture.
What the published data actually shows
Three references support practically every win rate figure circulating in the market. It is worth reading what each one measures, because they are not measuring the same thing.
| Source | What it reports | What it actually measures |
|---|---|---|
| Kohavi, KDD 2015 (Microsoft) | About one third improve the target metric, one third are neutral, one third make it worse | Ideas tested in a mature program, counted in all three directions |
| Kohavi, Deng and Vermeer, KDD 2022 | Compilation of published success rates: 33 percent (Microsoft), 20 percent (Kaushik), 15 percent (Bing), 10 percent (Booking.com, Google Ads, Netflix), 8 percent (Airbnb Search) | Estimates from different origins, gathered to compute the false positive risk of each |
| Thomke, HBR 2020 (Booking.com) | A program running tens of thousands of experiments a year with public emphasis on learning from failure | Volume and culture, more than a comparable rate |
The distance between “one third” and “10 percent” reads like a contradiction and is not. Both ends of the range come from the same table in the 2022 paper: 33 percent is the Microsoft line, exactly the one-third proportion documented in 2015, and 10 percent is the line attributed to Booking.com, Google Ads and Netflix. Between them sit Bing at 15 percent and an Avinash Kaushik estimate at 20 percent. These are not measurements from a single study comparing companies: they are numbers published at different moments in different contexts, and the authors state plainly that counting criteria vary across sources and that the figures serve as orders of magnitude. A range from 8 to 33 percent is not measurement noise, it is the effect of four variables the next section unpacks.
Why the rates vary so much
Four variables explain almost the entire gap between teams, and none of them is talent.
- Site maturity. A site that has never been optimized has large, obvious problems waiting. After two years of a program, what remains are small effects that produce more ties. The same team, on the same site, tends to watch the rate fall over time.
- Size of the change being tested. Testing a new value proposition produces bigger effects, and bigger losses, than testing microcopy. A team that only tests details will accumulate ties regardless of hypothesis quality.
- The minimum detectable effect chosen. Sizing to detect plus 20 percent relative and sizing to detect plus 5 percent completely change how many real effects the test can see. Whoever picks an ambitious MDE records a pile of small real effects as ties. The arithmetic is in the statistical significance guide.
- The internal definition of a win. Some teams count any positive result as a win, even without significance. Others count only what actually shipped. That alone moves the rate by tens of points without anything changing in the operation.
Before comparing your rate against any benchmark, write down your definition of a win. In most cases that one sentence explains the entire difference.
A high win rate is usually a warning sign
This is the counterintuitive point most benchmark articles skip. If public sources point to something between 8 percent and one third, a program reporting 60 or 70 percent wins deserves an audit before it deserves a celebration. The four most common causes, in rough order of frequency:
| Likely cause | How to spot it | What to do |
|---|---|---|
| Stopping early after a favorable dashboard reading | Tests ending on irregular dates, always right after a favorable spike | Fix duration before starting; see the peeking problem |
| Underpowered tests | Sample per variation well below what the declared MDE requires | Size before running; accept running fewer and better tests |
| Switching the metric after the result | The primary metric in the report is not the one in the plan | Register the primary metric and guardrails before starting |
| Ties counted as wins | Wins reported without a p-value or an interval | Always report the interval, not just the direction |
The first two causes reinforce each other. An underpowered test almost never crosses the significance line through a real effect: when it crosses, it is because it caught a favorable draw. If on top of that the team checks the dashboard daily and stops when it likes what it sees, the win rate climbs precisely because the accuracy fell.
The calculation almost nobody runs: how many winners are false
Here is the part that changes decisions. The question “how many of my winners are real” is not answered by the p-value alone. It also depends on the share of your ideas that genuinely work, and that is where win rate becomes genuinely useful information.
Consider 1,000 tests run at 95 percent two-sided confidence and 80 percent power, in a scenario where 10 percent of tested ideas produce a real effect.
- 100 tests have a real effect. At 80 percent power, about 80 of them will be detected as significant.
- 900 tests have no effect. A two-sided test at 5 percent produces 5 percent false positives, but half of them point in the negative direction and nobody calls that a win. That leaves 2.5 percent in the positive direction: about 22.5 false winners.
- Total “winners”: 80 plus 22.5, so 102.5.
- Share of false winners: 22.5 divided by 102.5, approximately 22 percent.
In other words: even with correct statistics, even without peeking, even with adequate power, roughly one in five winners in that program is not a win. And it gets worse quickly as the share of good ideas falls or power drops.
| Real share of ideas that work | Test power | True winners per 1,000 | False winners per 1,000 | False among the winners |
|---|---|---|---|---|
| 5% | 80% | 40.0 | 23.8 | 37.3% |
| 10% | 80% | 80.0 | 22.5 | 22.0% |
| 20% | 80% | 160.0 | 20.0 | 11.1% |
| 33% | 80% | 266.6 | 16.7 | 5.9% |
| 10% | 50% | 50.0 | 22.5 | 31.0% |
Computed with a 5 percent two-sided alpha, counting as a “winner” only a significant result in the positive direction.
Worth recording that this calculation is not our invention: it is exactly the computation Kohavi, Deng and Vermeer apply in the table of the 2022 paper, and the values match theirs line by line. For the success rates in that compilation, the false positive risk they report is 5.9 percent (Microsoft, 33 percent), 11.1 percent (Kaushik, 20 percent), 15.0 percent (Bing, 15 percent), 22.0 percent (Booking.com, Google Ads and Netflix, 10 percent) and 26.4 percent (Airbnb Search, 8 percent). That is the reason the table exists in the original paper: to show that the lower a program’s success rate, the larger the slice of its winners that are not wins at all.
Worked example: reading a winner through this lens
A checkout flow test runs 12,000 visitors per variation. The control converts 360 times (3.00 percent) and the variation converts 420 times (3.50 percent). Running the two-proportion z-test, two-sided, at 95 percent confidence:
- Relative lift: +16.67 percent.
- Absolute difference: +0.50 percentage points.
- z statistic: 2.18.
- p-value: 0.029.
- Confidence interval for the difference: from +0.05 to +0.95 percentage points.
Check that calculation with the same numbers, or with yours:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
By the classic criterion this is a win: the p-value came in under 0.05 and the interval excludes zero. Two additional readings, however, change what to do with the win.
The first is the interval. It runs from +0.05 to +0.95 percentage points, a range that in relative terms goes from roughly +1.7 percent to +31.7 percent. The point estimate of +16.67 percent is the center of that range, not a forecast. Promising the board a 16 percent conversion gain from this test is promising the middle of an interval that nearly touches zero.
The second is the base rate. If this team historically gets about 10 percent of its hypotheses right, roughly 22 percent of its winners are false, and a result with a p-value of 0.029 (close to the line, not far from it) sits exactly in the region where false positives concentrate. The correct move is not to discard the result, it is to ship the change and confirm the effect in follow-up measurement, or to replicate the test if the change is expensive or risky to reverse.
How to calculate your own rate properly
Before comparing against any benchmark, the number has to be computed in a way that survives an audit. The most common mistake is not in the numerator, it is in the denominator.
The honest denominator is every test that started collecting data, not every test that reached the end. Tests aborted midway, tests discarded over implementation problems and tests nobody formally closed drop out of the count very often, and they drop out in a biased way: a test is far more likely to be abandoned when the first days look bad than when they look good. Excluding the abandoned ones inflates the rate without anything improving.
The honest numerator has three possible definitions, and the only mistake is failing to pick one and write it down:
- Statistical win: a significant result in the positive direction on the primary metric, with guardrails intact. This is the most comparable across teams.
- Decision win: the test led to shipping the change. It includes cases where a tie was enough to decide (simplifying a screen without losing conversion, for example), which is a good decision rather than a statistical win.
- Realized win: the change shipped and the gain held up in post-launch measurement. This is the strictest definition and the most useful one for financial planning.
Mature teams report all three, and the distance between them is diagnostic. A statistical win rate far above the realized one points to effects inflated by the winner’s curse, or to gains that decay after launch. A decision win rate far above the statistical one indicates that much of the program’s value is coming from informative ties, which is legitimate as long as it is written down.
One final caution about the time window: a win rate computed per quarter swings wildly in programs running few tests. With 5 tests in a quarter, a single result moves the rate by 20 points. Below roughly 20 tests in the window, the number says more about chance than about the team, and the annual reading is the only one that holds.
Win rate and test velocity are the same conversation
A detail that gets lost when win rate is discussed on its own: the rate only becomes meaningful next to how many tests the program can actually run. Ten wins out of 100 tests and 1 win out of 10 tests are the same rate and a completely different business outcome, and the difference is capacity, not skill.
That capacity is not a matter of ambition either. It is bounded by traffic, by the effect size the team is willing to chase, and therefore by the sample each test consumes, which is exactly the arithmetic in how many A/B tests you should run per month. A team that wants a higher win rate has two honest levers: test bigger changes, which raises the share of real effects large enough to detect, or run more tests, which raises the absolute number of wins at any given rate. Optimizing the rate itself, without touching either lever, is what produces the audit findings from the section above.
What to measure instead of win rate
Win rate is easy to compute, easy to compare and easy to inflate, which is the exact combination of a bad indicator. Three measures resist the perverse incentive better:
| Indicator | How to measure it | Why it resists manipulation |
|---|---|---|
| Cumulative realized effect over the year | Sum of gains that stayed live after launch, measured rather than estimated | It only grows if the wins are real and survive |
| Tests with a hypothesis registered before running | Share of tests with hypothesis, primary metric and duration written before the start | Cannot be inflated without improving planning |
| Documented results, including ties and losses | Share of tests with a complete entry in the repository | Measures learning, which is the program’s real product |
The last two come from the process, not the result, and that is exactly why they work: neither improves when someone stops a test early. The third depends on having somewhere to record it, which is the subject of the experimentation roadmap template and the program’s documentation routine.
A fourth indicator suits more mature teams: the share of statistical capacity actually used, meaning how many of the year’s tests had enough sample for the effect they set out to detect. A program full of underpowered tests will show a high win rate and a low cumulative effect at the same time, which is the classic signature of a pretty dashboard with no revenue behind it.
Automate this with Donnu
Reading a win properly requires three things at once: duration fixed before the start, the confidence interval reported alongside the effect, and the program’s history so you know where your result falls. Donnu delivers all three by default: it sizes the test against your real traffic and shows the duration before you start, presents the interval next to every result instead of a lone number, and keeps the record of each experiment with the numbers frozen exactly as they were read. Win rate becomes a consequence of the process rather than a target to chase.
Start a free 14-day trial and read your results with the interval in view. To size how many tests your traffic can actually support before promising a cadence, see how many A/B tests you should run per month.
References
- Kohavi, R. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. Keynote, ACM SIGKDD 2015. exp-platform.com/Documents/2015-08OnlineControlledExperimentsKDDKeynoteNR.pdf.
- Kohavi, R., Deng, A. and Vermeer, L. A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. KDD 2022. The compilation of success rates and the corresponding false positive risk are in Table 2. Paper page: exp-platform.com/abtestingintuitionbusters.
- Thomke, S. Building a Culture of Experimentation. Harvard Business Review, March-April 2020. hbr.org/2020/03/building-a-culture-of-experimentation.
- Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.
- Kohavi, R. ExP Platform: accelerating innovation through trustworthy experimentation. exp-platform.com.
Read also: How to build an experimentation culture · How many A/B tests should you run per month · A/B testing statistical significance · Leia em português
Frequently asked questions
- What is the average A/B test win rate?
- There is no single number, and the published data opens a wide range. Ronny Kohavi documented that at Microsoft roughly one third of tested ideas improve the target metric, one third are neutral and one third make it worse. Separately, Kohavi, Deng and Vermeer compiled success rates published by several operations and reported a range from 8 percent (Airbnb Search) to 33 percent (Microsoft), with 10 percent reported for Booking.com, Google Ads and Netflix. The authors themselves warn that these estimates use different counting criteria and should be read as orders of magnitude, not as comparable benchmarks.
- Is a high win rate a good thing?
- Not always, and past a certain point it is a warning sign. Rates far above one third usually indicate process problems before they indicate talent: stopping early after a favorable dashboard reading, underpowered tests where only a lucky draw crosses the line, switching the primary metric after seeing results, or counting flat results as wins. Before celebrating a high rate, audit how the number was counted.
- Why does a program win rate fall over time?
- Because the obvious improvements run out. A new program harvests large, evident problems with large, easy-to-detect effects. After a few quarters what remains are smaller effects, which need more sample to surface and produce more ties. A falling win rate alongside accumulating learning is the normal shape of a healthy program, not a symptom.
- If 10 percent of tests win, how many of those winners are false positives?
- About 22 percent, and that is the figure Kohavi, Deng and Vermeer publish for a 10 percent success rate. At 95 percent two-sided confidence and 80 percent power, every 1,000 tests produce roughly 80 true winners and 22.5 false positives in the positive direction, so about 22 percent of your winners are not wins. That calculation depends on the base rate of good ideas, not only on the p-value, which is why replicating a surprising winner before rolling it out pays for itself.
- What should you measure instead of win rate?
- Three numbers say more: the cumulative realized effect over the year (the sum of gains that survived in production, not the count of wins), the share of tests with a hypothesis written before the test ran, and the share of results documented including ties and losses. Win rate measures luck and the difficulty of the bets; none of those three can be inflated without genuinely improving the process.
- Is it worth comparing your win rate against another company?
- Only as conversation, never as a target. The rate depends on available traffic, the minimum detectable effect chosen, how optimized the site already is, and the internal definition of a win. Two companies with equally good processes can post very different rates because of those four variables, so a win rate target copied from outside mostly pressures the team to inflate the number instead of improving decisions.