Traffic Allocation in A/B Tests: Why 50/50 Wins
Traffic allocation in A/B testing: why 50/50 is optimal, what a 90/10 split costs (2.78x the traffic) and when a ramp up solves the risk fear.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Running a test at a 90/10 traffic allocation does not make the experiment safer, it makes it 2.78 times slower. The same plan that takes 11 days with half the traffic on each arm takes 30 days at 90/10 and 272 days at 99/1, because the precision of the comparison is capped by the smaller arm. This article computes the exact factor for every split, shows the same test reaching opposite conclusions purely because of how traffic was divided, explains why a ramp up is the right answer to the fear that motivates 90/10, and why the SRM check needs to know about a deliberate unequal split before you turn the test on. It is part of our complete guide to A/B testing and it pairs with how many visitors an A/B test needs.
The traffic allocation arithmetic nobody runs
The precision of a comparison between two groups depends on both of them. The variance of the difference between two rates is proportional to 1 over the number of people in control plus 1 over the number in the variant. With a fixed total split so that the variant gets a share p, that sum equals 1 divided by p(1 minus p) times the total. It bottoms out exactly at p equal to 0.5, and climbs fast on either side.
Kohavi, Longbotham, Sommerfield and Henne give the rule a section title that leaves no room for interpretation, “Assign 50% of users to treatment”, and supply the practical approximation: the multiplicative increase in running time relative to 50/50 is 1 divided by 4p(1 minus p), where the treatment receives portion p of the traffic. They finish with the example that usually wakes the room up: an experiment run at 99 against 1 percent will have to run about 25 times longer.
We checked the approximation against the exact two-proportion sample size calculation, searching numerically for the total that delivers 80 percent power at each split. Kohavi and co-authors’ shortcut holds up comfortably:
| Split (control / variant) | Factor from the formula | Total needed | Measured factor | Days at 40k a week | Visitors in the variant |
|---|---|---|---|---|---|
| 50 / 50 | 1.0000 | 62,467 | 1.0000 | 11 | 31,234 |
| 60 / 40 | 1.0417 | 64,835 | 1.0379 | 12 | 25,934 |
| 70 / 30 | 1.1905 | 73,825 | 1.1818 | 13 | 22,148 |
| 80 / 20 | 1.5625 | 96,534 | 1.5454 | 17 | 19,307 |
| 90 / 10 | 2.7778 | 170,965 | 2.7369 | 30 | 17,097 |
| 95 / 5 | 5.2632 | 323,330 | 5.1760 | 57 | 16,167 |
| 98 / 2 | 12.7551 | 782,675 | 12.5294 | 137 | 15,654 |
| 99 / 1 | 25.2525 | 1,548,950 | 24.7963 | 272 | 15,490 |
The last column holds the intuition most discussions are missing: the variant needs roughly the same number of people whatever the split (between 15,000 and 31,000 here). What explodes is the total, because to hand 17,000 visitors to the variant at 90/10 you have to send 154,000 to control, and the calendar pays that bill.
The plan behind all those numbers
Every row above comes from a single fixed horizon plan. A 5 percent baseline rate, a minimum detectable effect of 10 percent relative, 95 percent confidence, 80 percent power, two-sided. The calculator returns 31,234 visitors per variant, or 62,468 in total, which is 11 days at 40,000 visitors a week.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Check it: baseline 5, MDE 10 percent relative, confidence 95, power 80. That 31,234 is what the table multiplies. The calculator, like almost every one on the market, assumes an even split; it has no allocation field because the right answer for that field is always 50. If you are going to run unequal anyway, take the 62,468 total and multiply by the factor on your row.
The same test, two splits, opposite conclusions
None of this is abstract. We simulated a real 10 percent relative effect (5.0 against 5.5 percent) and spent exactly the same total traffic, 62,467 visitors, in two different ways.
| Scenario | Control | Variant | Measured relative lift | p-value | Confidence interval | Verdict |
|---|---|---|---|---|---|---|
| 90 / 10 | 2,801 of 56,220 (4.9822%) | 337 of 6,247 (5.3946%) | 8.28% | 0.156893 | minus 0.1760 to 1.0007 pp | inconclusive |
| 50 / 50 | 1,554 of 31,234 (4.9754%) | 1,703 of 31,234 (5.4524%) | 9.59% | 0.007326 | 0.1284 to 0.8257 pp | variant wins |
Same underlying truth, same traffic budget, same number of days. The only difference is where the traffic went, and it decided whether the team found the win or shelved the hypothesis. Paste both rows into the significance calculator to see the two verdicts side by side.
The effect on detection shows up cleanly when you fix the traffic budget and ask what power is left:
| Split | Visitors in control | Visitors in variant | Analytical power | Measured power over 40,000 replications | Smallest detectable effect |
|---|---|---|---|---|---|
| 50 / 50 | 31,233 | 31,234 | 80.00% | 80.03% | 10.00% relative |
| 70 / 30 | 43,727 | 18,740 | not simulated | not simulated | 10.89% relative |
| 80 / 20 | 49,974 | 12,493 | 61.94% | 61.64% | 12.48% relative |
| 90 / 10 | 56,220 | 6,247 | 40.57% | 40.61% | 16.71% relative |
| 95 / 5 | 59,344 | 3,123 | not simulated | not simulated | 23.17% relative |
At 90/10 the power of the test fell from 80 to 40 percent without a single line of the plan changing. A test with 40 percent power misses more than half of the real winners, and the team reads that silence as “the hypothesis did not work”. It is the same mechanism we described in observed power after the test: a null result is not information about the product, it is information about the design.
What an unequal split does NOT ruin
Precision matters here, because this is a common confusion. An unequal split does not inflate false positives. We ran 40,000 replications under the null hypothesis, with both variants sitting at exactly 5 percent:
| Split | Measured false positive rate | Monte Carlo standard error |
|---|---|---|
| 50 / 50 | 5.11% | 0.11 pp |
| 90 / 10 | 4.83% | 0.11 pp |
| 98 / 2 | 5.19% | 0.11 pp |
All where they should be. The two-proportion test already handles unequal group sizes correctly, and the p-value still means what it promises. The damage from an unequal split lands entirely on Type II error, the failure to see what is actually there. It never shows up in an alert, only in the time that goes by.
Why 90/10 feels safe, and the right answer to that fear
The motivation is nearly always the same, and it is reasonable: if the variant is bad, the fewer people who see it the better. Kohavi and co-authors name this practice as common among novice experimenters and agree with the worry, while disagreeing with the fix. Their answer is treatment ramp up: start at 99.9 against 0.1 percent, then move to 0.5, then 2.5, then 10, then 50 percent, with each step running for a couple of hours and existing to confirm there are no egregious problems before exposing more users.
The difference between a ramp and a split is what each one is trying to answer. The ramp answers “is this broken?” and it is short. The split answers “does this work?” and it is the whole test. Conflating them means accepting a bad measurement design to solve a safety problem that already has its own answer.
Bing’s ramp is not your ramp
Here is a detail the big-company literature tends to hide. The 0.1 percent step for two hours was written for sites with enormous volume. On a site doing 40,000 visitors a week, that step exposes 0.5 users. It detects nothing.
| Ramp step | Hours to expose 500 users at 40k a week | Hours to expose 500 users at 4M a week |
|---|---|---|
| 0.1% | 2,100 h (87.5 days) | 21 h |
| 0.5% | 420 h (17.5 days) | 4.2 h |
| 2.5% | 84 h (3.5 days) | 0.84 h |
| 10% | 21 h (0.9 days) | 0.21 h |
| 50% | 4.2 h | 0.04 h |
The practical conclusion for a mid-traffic site is that a useful ramp has two or three steps, not five, and they live between 10 and 50 percent. Below 2.5 percent, a step is not a safety check, it is a decorative pause that only burns calendar. And it burns little of the plan: three days at 10 percent hand 1,714 visitors to the variant, or 5.49 percent of the N it needs, which is acceptable as the price of safety.
The small arm can fall below the normality floor
There is an extra problem that only shows up on skewed metrics, and it is worse than lost power. The normal approximation underpinning the p-value needs a minimum number of observations to hold, and that minimum depends on the skewness of the metric. Kohavi, Deng, Longbotham and Xu propose a rule of thumb of 355 times the square of the skewness coefficient per variant, recommended when the absolute skewness exceeds 1, and publish the Bing table:
| Metric | Skewness | Minimum sample per variant |
|---|---|---|
| Revenue per user | 17.9 | 114,000 |
| Revenue per user (capped) | 5.2 | 9,700 |
| Sessions per user | 3.6 | 4,700 |
| Time to success | 2.1 | 1,550 |
In our 90/10 example the variant received 6,247 visitors. For conversion rate, a binary metric, that is nowhere near a problem. For uncapped revenue per user, with a floor of 114,000, the small arm is not merely underpowered: it sits below the level at which the p-value for that metric means anything. If revenue is your primary metric, an unequal split moves from the “slower” category into the “uninterpretable” one, and outlier capping becomes a prerequisite rather than a refinement.
Changing the split mid-test is a different design, not a tweak
A common temptation is to start at 90/10 “just to check things are fine” and move to 50/50 once confidence builds, treating it as an operational detail. It is not. FDA guidance classifies allocation adaptations as their own family of adaptive design and separates two cases. When the ratio shifts based on baseline characteristics, in pursuit of balance between the groups, it does not directly increase the Type I error probability provided the analysis uses the appropriate methodology. When the ratio shifts based on the outcomes already observed, what the agency calls response-adaptive randomization, you have left fixed horizon A/B testing and entered the territory of algorithms that reallocate traffic toward the winning arm, the subject of bandits versus A/B testing.
The practical rule is simple: changing the split for calendar or safety reasons, with the criterion written in advance, is fine, and the only consequence is how long the test takes. Changing the split because the variant is doing well turns the final reading into something else, and the two-proportion p-value stops being the right instrument.
Declare the split to the SRM check
One last operational detail, and it bites on the first morning. The sample ratio mismatch check compares the observed split against the expected one. If the tool assumes 50/50 by default and you turned on a 90/10 test, it will scream every day:
| Weights given to the checker | Chi-square | p-value | Flags a problem? |
|---|---|---|---|
| 50 and 50 (the default) | 39,977.9 | below 0.0001 | yes, and it is a false alarm |
| 0.9 and 0.1 (declared) | 0.0000 | 0.9968 | no |
The false alarm is worse than it looks, because it trains the team to ignore the guardrail that exists precisely to catch the real problem. A deliberate unequal split is only acceptable when the expected ratio is recorded alongside the experiment, so the checker tests against it.
Make this automatic with Donnu
An unequal split is rarely a decision, it is usually a default inherited from the tool or a caution nobody converted into days. In Donnu the experiment split appears next to its cost: moving control from 50 to 90 percent immediately updates the total needed, the projected end date and the power you have left with the traffic you actually get. Ramp up is a phase of its own with an exit criterion, and the SRM check reads the declared ratio from the experiment instead of assuming an even split, so a deliberate unequal allocation does not turn into a daily false alarm.
Frequently asked questions
The short answers live in the FAQ section of this page, built from the same calculations presented here.
References
- Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne. Controlled experiments on the web: survey and practical guide, Data Mining and Knowledge Discovery, 2009. Section 6.2.4 (“Assign 50% of users to treatment”) is the source of the recommendation to give 50 percent to each variant in order to maximise power and minimise running time, and of the 1 divided by 4p(1 minus p) approximation for the multiplicative increase in running time, with the example of 99 against 1 percent needing about 25 times longer. Section 3.5.1 is the source of the ramp up from 99.9/0.1 through 0.5, 2.5, 10 and 50 percent with steps of a couple of hours, and section 6.2.5 supplies the example of a 5 day test that becomes four weeks at 95/5 plus the warning against running at 99/1 because of the excessive duration and contamination from cookie churn.
- Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu. Seven Rules of Thumb for Web Site Experimenters, KDD 2014. Rule 7 is the source of the rule of thumb of 355 times the square of the skewness coefficient per variant, recommended when absolute skewness exceeds 1, and of the Bing table with revenue per user at skewness 17.9 and a minimum sample of 114,000, capped revenue at 5.2 and 9,700, sessions per user at 3.6 and 4,700, and time to success at 2.1 and 1,550.
- U.S. Food and Drug Administration. Adaptive Designs for Clinical Trials of Drugs and Biologics: Guidance for Industry, November 2019. Section V.E (“Adaptations to Patient Allocation”) is the source of the distinction between covariate-adaptive assignment, which pursues balance between groups and does not directly increase the Type I error probability when analysed with the appropriate methodology, and response-adaptive randomization, in which the chance of a newly enrolled subject landing in each arm changes over the course of the trial based on outcomes already observed, including the note that this second family can in some circumstances minimise the variance of the test statistics and shorten the trial.
Read next
Frequently asked questions
- What is the best traffic split for an A/B test?
- Half to each variant. Kohavi and co-authors recommend explicitly that 50 percent of users see each variant, because that split maximises power and minimises running time. The precision of the comparison is capped by the smaller arm, so moving traffic from one side to the other always makes the total worse, never better.
- How much does a 90/10 split cost compared to 50/50?
- Roughly 2.78 times more traffic for the same power. The factor is 1 divided by 4p(1 minus p), where p is the variant share: 1.04x at 60/40, 1.56x at 80/20, 2.78x at 90/10, 5.26x at 95/5 and 25.25x at 99/1. In the plan we work through here, an 11 day test at 50/50 becomes a 30 day test at 90/10 and a 272 day test at 99/1.
- Does an unequal split break the statistics of the test?
- No. The test stays valid: across 40,000 replications under the null hypothesis, the false positive rate came in at 5.11 percent for 50/50, 4.83 percent for 90/10 and 5.19 percent for 98/2, all within simulation error. What an unequal split ruins is power, not alpha. You will not see more false winners, you will fail to see real ones.
- If I am afraid of exposing the new variant, should I not run 90/10?
- The fear is legitimate, the fix is different: ramp up. Kohavi and co-authors describe starting at 99.9 against 0.1 percent, then stepping up through 0.5, 2.5, 10 and finally 50 percent, with each step running for a couple of hours and existing to catch egregious problems before more users are exposed. The ramp is a short safety phase, not the measurement design. Once it clears, the test runs at 50/50, which is where it measures.
- Will the SRM check flag a deliberate unequal split?
- It will, unless you declare the split to it. In our example, a 90/10 test with 56,220 and 6,247 visitors returns a chi-square of 39,977.9 and a p-value indistinguishable from zero when the checker assumes 50/50. Declaring weights of 0.9 and 0.1, the same test returns a chi-square of 0.0000 and a p-value of 0.9968. A deliberate unequal split means recording the expected ratio alongside the experiment, otherwise the guardrail turns into daily noise.
- Is there ever a good reason to run unequal?
- Two of them. The first is the ramp phase, which is short and is not the measurement. The second is when the variant carries a real per-user cost, such as an expensive API call or a discount, and you knowingly accept paying in calendar time to keep that cost down. Outside those, an unequal split is almost always a comfort decision that nobody translated into days.