A/A Test: Validate the Setup Before You Trust a Result
What an A/A test is, what a passing A/A actually proves, how to size one, and why the split check catches more real bugs than the p-value does.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
An A/A test runs your full experimentation pipeline while showing both groups the same experience, so every difference it reports is either noise or a defect. It is the cheapest way to find out whether your results mean anything, and the most commonly misread, because a passing A/A is a failure detector that came up empty, not a certificate that the system is sound. This guide covers what an A/A actually validates, why one significant A/A is expected rather than alarming, how to size one so a clean result carries information, and the split check that catches more real bugs than the conversion reading does. It is part of our complete guide to A/B testing and pairs with sample ratio mismatch, which is the failure mode an A/A most often surfaces.
What an A/A test is, and the two jobs it does
Kohavi, Longbotham, Sommerfield and Henne, in their 2009 survey of controlled experiments on the web, define the A/A test, which they note is sometimes called a null test, as follows: instead of an A/B test, you exercise the experimentation system, assigning users to one of two groups, but expose them to exactly the same experience. They name two uses for it, and the two are genuinely different jobs.
The first is measurement: collecting data to assess its variability for power calculations. Your own metric’s variance is an input to every sample size you will ever compute, and an A/A gives it to you on the exact population, instrumentation and window you will be testing on, rather than from a textbook assumption.
The second is validation: testing the experimentation system, where, in their words, the null hypothesis should be rejected about 5 percent of the time when a 95 percent confidence level is used. This is the job most teams mean when they say A/A test, and it is the one they most often misread, because that sentence contains the whole difficulty. The correct behaviour of a correct system is to produce a significant result now and then.
Why one significant A/A is the expected outcome, not an alarm
A test at a 95 percent confidence level is built to reject a true null 5 percent of the time. That is the definition of the threshold, not a flaw in it. So the question “did our A/A come back significant?” is the wrong question. The right one is “does our rejection rate sit near 5 percent across many runs?”
Run enough A/A tests and a significant one becomes not just possible but likely:
| A/A tests run | Chance at least one comes back significant |
|---|---|
| 1 | 5.0 percent |
| 2 | 9.8 percent |
| 3 | 14.3 percent |
| 5 | 22.6 percent |
| 10 | 40.1 percent |
| 20 | 64.2 percent |
| 40 | 87.1 percent |
A team that runs a weekly A/A for a year and treats each significant week as an incident will open roughly two or three investigations into a system that is working perfectly. A team that runs one A/A, sees it pass and declares the platform validated has learned almost nothing. The diagnostic signal is the rate, not the run. Which is also why the far more useful A/A is a permanent one, read as a long-run rejection rate, rather than a one-off ceremony before a big test.
The direction of the error matters too. A rejection rate materially above 5 percent means something is inflating your differences, and the 2009 survey names a specific culprit: robots. They report cases where robots caused many metrics to be significant when they should not have been, with much more than 5 percent false positives in an A/A test, and note that for some websites robots are thought to provide up to half the page views. A rejection rate materially below 5 percent is not reassuring either, since it usually means variance is being overestimated, which quietly costs you power in every real test you run.
Worked example: three A/A runs and what each one says
Run the numbers below in the calculator as you read. Each is an A/A on the same product at a 3 percent baseline conversion rate:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Run 1, the clean one. Control receives 40,000 users and 1,200 conversions (3.000 percent); the second arm receives 40,130 users and 1,236 conversions (3.080 percent). The difference is +0.08 percentage points, z = 0.66, p-value 0.5096, confidence interval -0.16 to +0.32 percentage points. The split, 49.919 percent against 50.081 percent, gives chi-square 0.211, p-value 0.646. Nothing fires. This is what a healthy A/A looks like, and note how wide that interval still is: it is consistent with a real bias anywhere from about 5 percent relative against the second arm to about 11 percent relative in its favour.
Run 2, the one that “wins”. Both arms receive 40,000 users; conversions land at 1,200 and 1,300. That is +8.33 percent relative, z = 2.03, p-value 0.0422, confidence interval +0.01 to +0.49 percentage points. Formally significant. The split is exactly 50/50, chi-square 0.000. This is the single most misread output in the topic. There is no treatment, so the effect is not real. One arm got luckier. The correct response is to record it and keep going, because the rate table above says this happens about 1 run in 20 in a system with nothing wrong with it. The wrong responses are to go hunting for a bug that is not there, or, worse, to conclude that the platform “detected a difference” and therefore works.
Run 3, the one that matters. Control receives 40,000 users and 1,200 conversions; the second arm receives 38,800 users and 1,210 conversions. The conversion reading is quiet: +3.95 percent relative, z = 0.97, p-value 0.3339, interval -0.12 to +0.36 percentage points. Anyone reading only the p-value passes this test. The split is not quiet: 50.761 percent against 49.239 percent gives chi-square 18.27, p-value 0.0000191, far past the 0.01 threshold conventionally used for a split check. Some 1,200 users who should have been in the second arm are not there.
The split check is the one that earns its keep
The reason run 3 is the interesting one is that the two checks are not redundant. The conversion reading asks whether the metric differs. The split check asks whether the machinery delivered the allocation you requested, which is a question about plumbing that no metric comparison can answer.
Fabijan and colleagues, studying sample ratio mismatch at Microsoft for KDD 2019, found that approximately 6 percent of experiments at Microsoft exhibit an SRM, and observe that a product running ten thousand experiments in a year can expect to see at least one per day. Their MSN.com case is an A/A story specifically: a team observed a mismatch in an A/A test, and the investigation found a bug in the assignment service where the control variant was assigned one bucket fewer than required out of a thousand, so a 50/50 test was actually set up as roughly 49.9/50.1, a skew they describe, correctly, as not necessarily an obvious issue.
That case is also a lesson in sizing, because a 0.1 point skew is very hard to see:
| Total users in the A/A | Observed split at 49.9/50.1 | Chi-square | p-value | Flagged at 0.01? |
|---|---|---|---|---|
| 80,000 | 39,920 / 40,080 | 0.32 | 0.572 | no |
| 200,000 | 99,800 / 100,200 | 0.80 | 0.371 | no |
| 500,000 | 249,500 / 250,500 | 2.00 | 0.157 | no |
| 1,000,000 | 499,000 / 501,000 | 4.00 | 0.046 | no |
| 2,000,000 | 998,000 / 1,002,000 | 8.00 | 0.0047 | yes |
| 5,000,000 | 2,495,000 / 2,505,000 | 20.00 | 0.0000077 | yes |
A defect of that size needs on the order of two million users before a routine check flags it. Which is the honest reading of every clean A/A: it rules out the biases your sample could have seen, and says nothing about the ones it could not. The mechanics of the check, its thresholds and the usual root causes are covered in sample ratio mismatch, and you can run your own numbers through the free A/A test checker.
The 2019 paper is also useful for what it says about scope. It lists three requirements for an assignment service: users must be equally likely to see each variant, repeat assignments of one user must be consistent, and when multiple experiments run there must be no correlation between them. Violating any of the three can cause a mismatch, and a defect in the assignment service usually shows up across many experiments at once rather than in one. So a split failure in an A/A is rarely a local problem, which is precisely what makes it worth catching there.
How to size an A/A so a clean result means something
Sizing an A/A is the same arithmetic as sizing any test, with one change of framing: the effect you are powering for is not a hoped-for improvement, it is the largest bias you are willing to leave undetected. Write that number down first, then size for it.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
At a 3 percent baseline, here is what a clean A/A actually rules out at different sample sizes:
| Users per arm | Smallest bias detectable at 80 percent power | In relative terms |
|---|---|---|
| 10,000 | 0.68 percentage points | 22.5 percent |
| 25,000 | 0.43 percentage points | 14.2 percent |
| 50,000 | 0.30 percentage points | 10.1 percent |
| 100,000 | 0.21 percentage points | 7.1 percent |
| 200,000 | 0.15 percentage points | 5.0 percent |
| 400,000 | 0.11 percentage points | 3.6 percent |
Read that table against run 1 above. With 40,000 users per arm, that A/A had roughly 68.0 percent power against a 10 percent relative bias and about 23.2 percent power against a 5 percent one. In other words, if the platform had been quietly favouring one arm by 5 percent, that run would have missed it more than three times in four. Getting to 80 percent power for a 5 percent relative bias needs about 207,938 users per arm. The same logic applies to reading any inconclusive result, and it is spelled out in minimum detectable effect.
Two practical constraints on top of the arithmetic. Run across at least one full weekly cycle, so weekday and weekend traffic are both inside the window rather than split across the boundary. And do not stop the A/A early because it looks clean, for the same reason you would not stop a real test early: repeatedly checking a running experiment inflates the false positive rate well above the nominal 5 percent, which is exactly the number an A/A exists to measure. That mechanism is covered in the peeking problem.
When to run one, and how to read the result
| Situation | Run an A/A? | What you are looking for |
|---|---|---|
| New experimentation tool, or a new integration of an existing one | Yes, before the first real test | Split correctness, event loss, rejection rate near nominal |
| Migrated to a new analytics or data warehouse pipeline | Yes | Metric definitions and joins that changed silently |
| Randomization moved from client side to server side, or vice versa | Yes | Assignment consistency for repeat visits |
| A real test produced a result nobody believes | Yes, but as a diagnostic, not a veto | Whether the machinery, rather than the result, is the problem |
| Routine, before every single experiment | No | Traffic spent here is traffic not spent deciding something |
| Continuously, in the background | Ideal, if you can afford the allocation | Long-run rejection rate and split stability over time |
And the reading rules, which matter more than the running:
- A split failure is an incident. Stop, find the root cause, and treat every concurrent experiment as suspect until you know the scope.
- A single significant conversion reading with a clean split is not an incident. Log it. Investigate only if the running rate drifts well away from 5 percent.
- A clean run is bounded by its sample. Report it as “no bias above X detected at this sample”, not as “validated”.
- Never report an A/A with the vocabulary of a winner. No arm won. If the write-up contains the word “lift”, the framing has already gone wrong.
Common mistakes with A/A tests
| Mistake | What it produces |
|---|---|
| Treating one significant A/A as proof of a broken system | Weeks spent debugging a platform that is behaving exactly as designed |
| Treating a clean A/A as proof of a sound system | False confidence in a pipeline that a small bias would pass straight through |
| Checking only the conversion p-value, never the split | The one failure mode most likely to be real goes unnoticed |
| Sizing the A/A by convenience rather than by tolerable bias | A clean result that rules out nothing anybody cared about |
| Stopping the A/A as soon as it looks clean | Peeking inflates the very false positive rate the test exists to measure |
| Running the A/A on a seed shared with a live experiment | The A/A samples that experiment’s population instead of sampling freshly |
| Reporting an A/A arm as a winner | A number with no cause behind it enters the decision record |
| Running one A/A ever, at platform launch | The pipeline that mattered is the one you have now, not the one you had then |
Automate this with Donnu
An A/A only pays off if the split check runs every time and the long-run rejection rate is something you can actually see. Donnu A/B runs the sample ratio check on every experiment, A/A or not, and surfaces a failed split as a data quality flag that blocks the readout rather than as a footnote under the result. Assignment is hashed per experiment so a validation run never inherits another test’s population, and an inconclusive reading reports its confidence interval, which is what tells you how much bias the run was actually able to rule out.
Start a free 14-day trial and check the split on your next experiment before you read the result.
References
- Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R. M. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 2009. Source of the A/A test definition and its alternative name of null test, of the two stated uses (assessing variability for power calculations and testing the experimentation system), of the expectation that the null is rejected about 5 percent of the time at 95 percent confidence, and of the finding that robots can drive an A/A well past 5 percent false positives. exp-platform.com.
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. and Dmitriev, P. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. KDD 2019. Source of the finding that approximately 6 percent of experiments at Microsoft exhibit an SRM, of the MSN.com assignment-service bug that turned a 50/50 test into roughly 49.9/50.1, and of the three requirements an assignment service has to satisfy. exp-platform.com.
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. On automated data quality monitoring and alerting inside a platform running hundreds of concurrent experiments. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on trust, data quality and the instrumentation of an experimentation platform. Companion material at experimentguide.com.
Read also: Sample ratio mismatch · A/B testing statistical significance · Minimum detectable effect · The peeking problem · Free A/A test checker · Leia em português
Frequently asked questions
- What is an A/A test?
- An A/A test runs your experimentation machinery at full speed while showing both groups exactly the same experience. Kohavi and colleagues (Data Mining and Knowledge Discovery, 2009), who also call it a null test, describe two uses: collecting data to assess variability for power calculations, and testing the experimentation system itself, where the null hypothesis should be rejected about 5 percent of the time at a 95 percent confidence level. Any difference it reports is by construction noise or a defect, because there is no treatment to cause a real one.
- Does a significant A/A test mean the system is broken?
- Not from a single run. At a 95 percent confidence level a correct system produces a significant A/A about 1 time in 20 by design, so one significant result is the expected behaviour rather than evidence of a bug. What matters is the rate across many runs. Ten A/A tests give roughly a 40.1 percent chance that at least one comes back significant, and twenty give about 64.2 percent. Judge the system by whether the long-run rejection rate sits near 5 percent, and investigate a single flagged run only if the split check also fails.
- How long should an A/A test run?
- Long enough to detect the size of bias you actually care about, which is usually far longer than teams expect. At a 3 percent baseline with 40,000 users per arm, the test has about 68 percent power against a 10 percent relative bias but only about 23.2 percent power against a 5 percent one, so a clean result at that size rules out very little. Reaching 80 percent power for a 5 percent relative bias needs about 207,938 users per arm. Run it across at least one full weekly cycle so day-of-week effects are inside the window rather than split across it.
- What does a passing A/A test actually prove?
- Much less than most teams assume. It is a failure detector, not a certificate. A clean A/A says the pipeline did not produce a detectable bias at the sample size you ran, which leaves plenty of room for a bias smaller than that sample can resolve. The MSN.com case documented by Fabijan and colleagues (KDD 2019) is the clearest illustration: a bug assigned control one bucket fewer than required, turning a 50/50 test into roughly 49.9/50.1, and a split that skewed needs on the order of two million users before a chi-square check flags it at the usual 0.01 threshold.
- Should I check the traffic split or the conversion rate?
- Both, and the split check is the one that earns its keep. A conversion reading in an A/A only fires at its nominal false positive rate and tells you little when it does. The split check tests something the conversion reading cannot see: whether randomization and logging delivered the allocation you asked for. Fabijan and colleagues (KDD 2019) found that roughly 6 percent of experiments at Microsoft exhibit a sample ratio mismatch, and an assignment-service defect usually shows up across many experiments at once rather than in one.
- Can I run an A/A test at the same time as real experiments?
- Yes, and a permanently running A/A is the more useful arrangement, provided the assignment for it is randomized independently of every other experiment. Kohavi and colleagues (2009) showed that some hashing schemes fail exactly that independence requirement, so an A/A sharing a seed with a live test can inherit that test population instead of sampling freshly. Run it continuously, alert on the long-run rejection rate rather than on any single window, and treat a split failure as a platform incident.
- Is an A/A test the same thing as an A/B test with no change?
- Mechanically yes, and that is the point: the value comes from putting the whole pipeline under load with a known correct answer. The difference is in how you read it. In an A/B test a significant result is a candidate finding; in an A/A test it can only be noise or a defect. That inversion is what makes the A/A a diagnostic, and it is also why an A/A must never be reported with the language of a winner.