SRM Checker (Sample Ratio Mismatch)
Before you read any winner, confirm the traffic split arrived the way you configured it. Paste the visitors of each variation and the checker tells you, by chi-square, whether there is a sample ratio mismatch. Free, no signup, with the math explained.
This is the first health check of your test, run before conversion. SRM (sample ratio mismatch) is the traffic split arriving crooked: you asked for 50/50 and got 52/48. When that happens at volume, it is not chance, it is a defect in the assignment engine that biases everything. Run this checker before you look at who won; if it flags, the conversion result does not hold. For the conversion verdict itself, use the statistical significance calculator afterward.
Fill in at least two variations. Expected allocation is the relative split weight (1 and 1 = 50/50, 9 and 1 = 90/10).
| Variation | Observed visitors | Expected | Actual % |
|---|
Chi-square goodness-of-fit test between the observed and the expected split. It flags a mismatch when the p-value drops below 0.01: with this split, a gap this large is too rare to be chance. SRM invalidates the test, so fix the assignment engine before you read conversion.
How to use it
- Enter the observed visitors of each variation (the real number that landed in each arm, not the conversions).
- Adjust the expected allocation with the weight of your split: 1 and 1 for 50/50, 9 and 1 for 90/10.
- Need more than two variations? Click add variation and enter the allocation of each.
- Read the verdict: split within the expected range, or SRM detected.
- Check the chi-square, the degrees of freedom and the p-value, plus the table with the expected count and actual share of each arm.
How it works: the formula
The checker uses the chi-square goodness-of-fit test, the same method serious experimentation tools apply for the SRM guardrail. It compares how many visitors arrived (observed) with how many should have arrived by your split (expected):
Where Oi is the observed count of each variation, Ei is the expected count (the total times the relative weight of that variation) and the sum runs over all variations. The degrees of freedom are the number of variations minus 1. The p-value comes from the chi-square distribution: it is the probability of seeing a gap as large as yours by pure chance, if the split were correct.
Worked example (reproduces the default result)
With the values already loaded: A with 5,200 visitors and B with 4,800, expected allocation 50/50. The total is 10,000, so the expected count is 5,000 per variation. The math:
χ² = (5200 − 5000)² / 5000 + (4800 − 5000)² / 5000 = 40000/5000 + 40000/5000 = 8 + 8 = 16.
With 2 variations, that is 1 degree of freedom. A χ² of 16 with 1 degree of freedom gives a p-value of about 0.00006, far below the 0.01 threshold. Verdict: SRM detected. A 52/48 split looks harmless to the eye, but with 10k visitors it is too rare to be chance. That is exactly what the tool shows above when you open the page.
How to read it and where it fools you
The verdict is binary on purpose: healthy split or SRM. A p-value above 0.01 means the gap fits within chance, your split is fine and you can move on to the conversion read. A p-value below 0.01 is a red flag: stop, the test is biased. Note that SRM measures no conversion at all, it only looks at the count of who entered each arm. A test can have SRM and still show a variation winning, and that is exactly where SRM saves you from celebrating a false positive.
Limits to keep in mind: the test needs volume to be sensitive, with a few hundred visitors it almost never flags anything. The expected allocation has to reflect what you actually configured, not what you assume. And SRM tells you a problem exists, not which one: the cause (redirect, cache, bots, tag) you investigate by hand. Once you have ruled out SRM, move to the significance calculator and review the traps in the guide on common A/B testing mistakes.
Best practices with the SRM guardrail
SRM is a routine check, not an exam you run only when you suspect something. Treat it as the first item of every test read.
- Run the checker before you look at conversion, every time. It is the guardrail that separates a real result from collection bias.
- If it flags SRM, do not fix the data: fix the cause, discard the biased period and restart the test.
- Compare the visitor count, not the conversion count. SRM is about who entered each arm.
- Suspect redirects and cache first: they are the most common way to lose visitors from a variation.
Frequently asked questions
- What is SRM (sample ratio mismatch)?
- SRM is when the split of visitors across your variations does not match the split you configured. If you asked for 50/50 and got 52/48 with a lot of traffic, something in the assignment engine is wrong (a redirect, cache, bots, a tag firing at the wrong time). The checker measures that gap with a chi-square test: if it is too large to be chance, it is SRM and the test is compromised.
- Why does SRM invalidate my A/B test?
- Because it breaks the core assumption of the experiment: that the groups are identical except for the change you tested. If the split is off, the groups probably differ in something else (device, source, timing), and any conversion difference could be that bias rather than your variation. Reading a winner with SRM active is reading noise. Fix the cause and rerun.
- Which p-value indicates SRM?
- The market convention is to flag it when the p-value drops below 0.01 (1%). The threshold is stricter than the 5% of a deliberate conversion test: SRM is a guardrail check you run every time, so a low threshold avoids false alarms. A p-value above 0.01 means the observed gap fits within chance and the split is healthy.
- Can I use the checker with a split other than 50/50?
- Yes. Adjust the expected allocation of each variation with the relative weight of your split: 1 and 1 for 50/50, 9 and 1 for a 90/10 ramp, 1 and 1 and 1 for 33/33/33. The test compares what you configured against what actually arrived, whatever the target proportion is.
- Does it work for tests with more than two variations?
- It does. Add one variation per arm of the test (A/B/n) and enter the expected allocation of each. The chi-square gains more degrees of freedom and the verdict stays the same: split within the expected range, or SRM detected.
- What do I do when the checker flags SRM?
- Do not read conversion yet. Investigate the usual causes: a redirect losing part of one variation, a tag or snippet loading at different times, cache serving a fixed version, bots concentrated in one arm, or an asymmetric analytics filter. Fix the root cause, discard the biased data and restart the test with a healthy split.
Keep going
Healthy split? Move to the conversion verdict in the statistical significance calculator and plan the next test with the sample size calculator. The full context is in the guide what is A/B testing.