Sample Ratio Mismatch (SRM) in A/B Testing
What sample ratio mismatch is, how the chi-square check works, how small an imbalance already invalidates a test, and how to find the root cause.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
A sample ratio mismatch is a statistically significant gap between the traffic split you configured and the split you actually got, and it is the single check that decides whether the rest of your analysis means anything. If a 50/50 test finishes 50.6 versus 49.4, that is not a rounding detail. It is evidence that something decided which visitors landed where, and once something other than the randomizer is choosing, the two groups are no longer comparable and the conversion difference between them is not the effect of your change. This guide covers what the check is, how small a gap already fails it, how to read the chi-square result, where mismatches come from, and what to do when you find one. It is part of our complete guide to A/B testing and sits alongside A/B testing statistical significance as the other half of a trustworthy result.
What sample ratio mismatch actually means
Randomization is the whole argument of an A/B test. It is what lets you claim that the only systematic difference between control and variation is the change you shipped. The sample ratio is the cheapest observable evidence that randomization worked, because you know the answer in advance: you configured 50/50, so you expect close to 50/50.
“Close to” is doing real work in that sentence, and the chi-square goodness-of-fit test is what makes it precise. It asks a single question: if assignment really were fair, how often would chance alone produce a gap at least this large? When that probability drops below the alarm threshold, chance stops being a credible explanation.
The consequence is severe and worth stating plainly. Fabijan and colleagues, writing about experimentation at Microsoft, Booking.com, Outreach.io and Online Dialog, describe SRM as a condition that in most cases completely invalidates experiment results. It is not a warning to note in the appendix. It is a stop sign.
How small a gap already fails
The uncomfortable part of SRM is that the tolerance shrinks as your test grows. The absolute gap that trips the alarm grows roughly with the square root of the total sample, so as a percentage of traffic it keeps getting tighter. A split you would describe as “basically even” is often already a mismatch.
| Total visitors | Smallest gap that trips the check | Resulting split |
|---|---|---|
| 2,000 | 116 visitors | 52.90 / 47.10 |
| 10,000 | 258 visitors | 51.29 / 48.71 |
| 20,000 | 366 visitors | 50.91 / 49.09 |
| 50,000 | 576 visitors | 50.58 / 49.42 |
| 100,000 | 816 visitors | 50.41 / 49.59 |
| 200,000 | 1,152 visitors | 50.29 / 49.71 |
Figures computed with the same chi-square goodness-of-fit test used by the calculator below, at a 1 percent alarm threshold and an expected 50/50 split. The gap column is the difference between the two arms.
Read the bottom row again. At 200,000 visitors, a split of 50.29 versus 49.71 is a mismatch. No human looking at a dashboard would flag that. This is the entire reason the check has to be automatic and numeric rather than a glance.
Fill in at least two variations. Expected allocation is the relative split weight (1 and 1 = 50/50, 9 and 1 = 90/10).
| Variation | Observed visitors | Expected | Actual % |
|---|
Chi-square goodness-of-fit test between the observed and the expected split. It flags a mismatch when the p-value drops below 0.01: with this split, a gap this large is too rare to be chance. SRM invalidates the test, so fix the assignment engine before you read conversion.
Reading the chi-square result without the ceremony
The statistic is simple enough to compute by hand, which is worth doing once so the number stops feeling like a black box. For each group you take the squared distance between observed and expected, divided by expected, and add them up.
With 48,005 total visitors and a configured 50/50 split, each group is expected to receive 24,002.5. Control got 24,301, the variation got 23,704, so each is 298.5 away from its expectation.
chi-square = (24,301 - 24,002.5)^2 / 24,002.5
+ (23,704 - 24,002.5)^2 / 24,002.5
= 89,102.25 / 24,002.5 x 2
= 7.42 (1 degree of freedom)
p-value = 0.0064Degrees of freedom is the number of groups minus one, so a two-arm test has one. The p-value answers the only question that matters: a fair coin would produce a gap this large or larger about 6 times in 1,000. That is not impossible, but it is a bad bet when the alternative explanation is a bug you can go find.
Two notes that save arguments. First, the threshold for SRM is deliberately stricter than the 0.05 you use for conversion, because every test you run is another chance to flag a false alarm and the cost of investigating one is a few hours while the cost of missing one is a wrong decision. A 1 percent threshold is a common working default and it is what the calculator above uses. Second, an SRM p-value is not a measure of how bad the bias is. It only tells you how confident you can be that the imbalance is real. A tiny, real, systematic bias in a huge test produces a terrifying p-value and may barely move your metric, while a large bias in a small test may squeak past. Use it as a trigger to investigate, never as a severity score.
Worked example: the winner that was not there
An ecommerce team tests a rebuilt checkout. The test is configured 50/50 and runs to 48,005 visitors. Here is the scoreboard they bring to the review.
- Control: 24,301 visitors, 1,094 purchases, 4.502 percent
- Variation: 23,704 visitors, 1,156 purchases, 4.877 percent
Run those numbers and you get z = 1.943, p-value = 0.0520, a relative improvement of +8.33 percent, and a confidence interval on the difference of -0.003 to +0.753 percentage points. Just outside significance, positive direction, large enough to matter commercially. The room’s instinct is to run it two more days.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
The instinct is wrong, and not because of the p-value. The split is 50.62 / 49.38, which as we computed above is a mismatch at p = 0.0064. The 597 visitors that never reached the variation are the story, and nobody in the room knows who they were.
Here is why rebalancing cannot rescue this. Suppose we ask what the result would have been if those 597 missing visitors had shown up. Two honest bounding scenarios:
If they behaved exactly like control and converted at 4.502 percent, the variation would show 24,301 visitors and about 1,183 purchases. Result: p = 0.0561, lift +8.14 percent, interval -0.009 to +0.742 percentage points.
If they were visitors who could not complete and converted at zero, the variation would show 24,301 visitors and 1,156 purchases, a rate of 4.757 percent. Result: p = 0.1808, lift +5.67 percent, interval -0.118 to +0.629 percentage points.
The observed result sits between a near miss and clearly nothing, and the data cannot tell you which. That is the definition of an invalid experiment. In this case the root cause turned out to be a redirect that timed out on slow mobile connections, which means the second scenario is closer to the truth: the missing visitors were disproportionately the ones least likely to buy, and their absence inflated the variation’s rate. After the redirect was fixed and the test rerun, the split came back at 24,102 / 24,088 (chi-square 0.004, p = 0.949) and the result was 4.502 percent against 4.583 percent: p = 0.6675, lift +1.81 percent, interval -0.290 to +0.453 percentage points. Inconclusive, and honest.
Where mismatches come from
The KDD 2019 taxonomy is useful precisely because it sorts causes by the stage where they appear, which is also how you decide who owns the fix. The paper identifies 25 distinct causes grouped into five stages.
| Stage | What goes wrong | Typical symptom |
|---|---|---|
| Assignment | The randomizer is not independent: shared seed, a carried-over bucket from a previous test, an exclusion rule that hits one arm | Mismatch present from hour one, stable in size, visible in an A/A test |
| Execution | A variant loads slower, a redirect fails, a flag arrives late, telemetry never fires for some users | Mismatch appears only on certain devices or connections, and performance metrics degrade on the same side |
| Log processing | Bot filtering, deduplication or a join that drops rows unevenly across variants | Mismatch appears in the warehouse but not in the raw assignment logs |
| Analysis | A trigger or filter applied to one group, a segment defined post hoc, a date range that clips one arm | Mismatch present in the filtered scorecard but absent in the full one |
| Interference | Another running experiment overlaps and its targeting correlates with yours | Mismatch appears only while the other test is live |
The distinction between the analysis stage and the rest is the one that saves the most time. If the split is fine on the full population and broken only in your filtered view, the experiment is probably healthy and your query is not.
The diagnostic order that finds it fastest
Fabijan and colleagues publish ten rules of thumb for narrowing down a cause. Condensed into the order that resolves most cases quickly:
- Compare the filtered scorecard against the full one. If the mismatch only exists in the filtered view, the trigger condition is wrong, not the experiment. Relax the filter step by step until it disappears.
- Segment by user attributes. A mismatch confined to one browser, one app version or one country is a localized cause, usually a capability the variant assumes and that segment lacks.
- Segment by time. Evidence that is strongest on day one and fades later points to caching, a delayed variant rollout or a staggered start, not to a broken randomizer.
- Look at performance metrics on the mismatched side. A variant that increased load time will lose telemetry from the slowest users, which produces a mismatch and a degraded metric from the same root cause.
- Look at engagement. If average engagement per user is higher in the arm that lost users, the cause is hitting less engaged users hardest, and vice versa.
- Count how many experiments are affected. Several unrelated tests flagging at once means the cause is systemic and lives in the platform, not in any one test.
- Check an A/A test. A mismatch in an A/A run is a platform problem by definition, and the A/A test calculator is the cheapest way to keep that baseline honest.
Rule 7 deserves emphasis because it inverts the usual mood in the room. When a test flags SRM, the argument is always about whether this particular test is broken. An A/A run settles it without any of the politics, because there is no result anyone is attached to.
What to do when you find one
The rules here are short, and one of them is unpopular.
Do not rebalance. Trimming the larger arm or reweighting the counts assumes the extra or missing visitors were a random draw. The mismatch is evidence that they were not. Whatever you do downstream inherits the bias.
Do not report the result with a caveat. A flagged test is not a weaker result, it is an unknown one, and a number in a slide deck outlives every asterisk attached to it.
Do find the stage first, then the cause. The table above turns a vague hunt into three or four targeted checks, and most mismatches are resolved by the first two rules in the diagnostic list.
Do keep the failed test in your records. A mismatch you diagnosed is a permanent improvement to the platform and belongs in the experiment repository exactly like a shipped winner. Programs that delete their invalid tests rediscover the same bug every quarter.
Do check both early and late. Running the check on day one catches assignment and rollout bugs while a restart is cheap. Running it again at the end catches log processing and analysis problems that only appear once the data is aggregated. Neither check is peeking, because you are inspecting the traffic split and not the outcome metric. The peeking problem applies to looking at the result and deciding whether to stop, which is a different act entirely.
Common mistakes with SRM
| Mistake | What it produces |
|---|---|
| Never running the check | Every bias of this kind ships silently, and the program cannot tell a real win from a broken split |
| Eyeballing the split instead of testing it | At scale, a mismatch that matters looks perfectly even on a dashboard |
| Treating the SRM p-value as a severity score | Huge tests flag tiny real biases with alarming p-values; small tests hide large ones |
| Rebalancing or reweighting the arms | Preserves the bias while removing the evidence of it |
| Checking only the filtered scorecard | Confuses a broken analysis query with a broken experiment |
| Restarting without finding the cause | The same mismatch returns, usually on the test that mattered most |
| Applying a 0.05 threshold | Too many false alarms across a busy program, which trains the team to ignore the check |
Automate this with Donnu
An SRM check only protects a program if it runs on every test without anyone remembering to ask. Donnu A/B compares the observed split against the configured one continuously, flags the mismatch before the conversion numbers are presented rather than after, and holds the verdict back instead of showing a lift that the split has already invalidated. The split, the expected counts and the chi-square result sit next to the result, so the first question in the review is answered before anyone opens the debate.
Start a free 14-day trial and run the check against your own traffic split.
References
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. and Dmitriev, P. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. KDD 2019. Source of the five-stage taxonomy, the ten rules of thumb, the approximately 6 percent SRM rate at Microsoft and the LinkedIn figure on triggered analyses. exp-platform.com.
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. On data quality checks and why sustained effects, not first-look numbers, are the object of an experiment. exp-platform.com.
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. On automated data quality monitoring in a large experimentation platform. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
Read also: What is A/B testing · A/B testing statistical significance · The peeking problem · Common A/B testing mistakes · Free SRM checker · Leia em português
Frequently asked questions
- What is a sample ratio mismatch (SRM)?
- A sample ratio mismatch is a statistically significant gap between the traffic split you configured and the split you actually observed. If you set 50/50 and finish with 50.6 percent of visitors in control, that gap is either chance or a bug, and a chi-square goodness-of-fit test tells you which. When the test flags a mismatch, the two groups are no longer comparable populations, so the conversion difference between them cannot be read as the effect of the change. Fabijan and colleagues (KDD 2019) describe SRM as a condition that in most cases completely invalidates experiment results.
- How small an imbalance already counts as SRM?
- Smaller than most people expect, and it gets stricter as the test grows. At a 50/50 split with 10,000 visitors total, a gap of about 258 visitors between the arms (51.29 versus 48.71 percent) is already enough to cross a 1 percent p-value threshold. At 100,000 visitors the gap only needs to reach about 816 visitors, which is a split of 50.41 versus 49.59 percent. The absolute gap grows roughly with the square root of the sample, so the percentage tolerance keeps shrinking. This is why eyeballing the split never works: at scale, a split that looks perfectly balanced can be a clear mismatch.
- Can I just rebalance the groups and analyze anyway?
- No, and this is the most expensive mistake in the whole topic. Trimming the larger group or reweighting the numbers assumes the missing or extra visitors were a random sample, and that assumption is exactly what the mismatch disproves. The visitors that went missing are usually missing for a reason connected to the change itself, such as a slower page, a failed redirect or a bot filter that behaves differently across variants. Rebalancing hides the symptom and keeps the bias. The only safe path is to find the root cause, fix it and rerun.
- How common is SRM in real programs?
- More common than teams assume. Fabijan, Gupchup, Gupta, Omhover, Qin, Vermeer and Dmitriev report that approximately 6 percent of experiments at Microsoft exhibited an SRM during the period they studied, and note that a product running ten thousand experiments a year can expect to see at least one SRM per day. The same paper cites work at LinkedIn finding that about 10 percent of triggered analyses had an SRM. If your program has never flagged one, the most likely explanation is that nobody is running the check.
- Does SRM apply to splits that are not 50/50?
- Yes, and uneven splits deserve more attention, not less. The chi-square goodness-of-fit test compares each group against its own expected count, so it works for 90/10, 80/20 or any number of arms. Uneven designs are also more fragile: a 90/10 ramp that lands at 44,600 and 5,400 users out of 50,000 looks close to target but produces a chi-square of about 35.6 and a p-value near 2.5 in a billion, because the small arm is where a few hundred misrouted users hurt most.
- What causes sample ratio mismatch most often?
- The KDD 2019 taxonomy groups the causes by the stage where they appear: assignment, execution, log processing, analysis and interference between experiments. In practice the recurring offenders are a redirect that loses traffic on one side, a variant that loads slower and drops telemetry from the slowest users, a bot filter that treats variants differently, an analysis filter applied to only one group, and a second experiment overlapping with yours. The stage matters because it tells you which team owns the fix.
- Should I check SRM before or after the test ends?
- Both, and the earlier check is the one that saves money. Run it on day one to catch assignment and rollout bugs while the test is cheap to restart, then again at the end before you read any conversion number. Checking mid-flight for data quality is not peeking: you are inspecting the split, not the outcome metric, so it does not inflate the false positive rate of the conversion test. Peeking is a problem when you look at the result and decide whether to stop.