A/A Test Calculator (Randomization Check)

This is the calculator that validates your tool, not your test. An A/A test runs two copies of the same control: since nothing changes between the groups, you expect a balanced split and indistinguishable conversion. If it flags a winner, the problem is in the randomization or the tracking, not the page. It is different from the significance calculator, which exists to find a real winner in an A/B, and more complete than the SRM checker, which only looks at the split: here we check both at once, split and conversion.

How to use it

1. Set up two identical variations in your tool (the same control on both), with the split you use in real tests (usually 50/50).
2. Let the test run until it gathers a volume close to your A/B tests, in the thousands of visitors per group.
3. Paste the visitors and the conversions of each group into the fields above.
4. Read the verdict: passed (equivalent groups) or flag (different conversion, unbalanced split, or both).
5. Check the two separate reads: the conversion one (p-value and z) and the split one (χ² and the sample ratio mismatch p-value).

How it works: the two calculations

The A/A test combines two independent checks. The first, on conversion, is a two-proportion z-test, the same one behind the significance calculator. In an A/A you want the p-value to be HIGH (above 0.05): that confirms the difference between the groups is consistent with chance.

The second, on the split, is a chi-square goodness-of-fit test between the visitors who landed in each group and the expected split (50/50). Here too you want the p-value HIGH (above 0.01): a low one is a sample ratio mismatch, a sign that the split engine is broken.

Where p̄ is the pooled rate of both groups, O_i is the observed count in each group and E_i is the count expected by the split. The A/A only passes when BOTH reads come back clean.

Worked example (reproduces the default result)

With the values already loaded: A1 with 20,000 visitors and 1,000 conversions (5.00%) and A2 with 20,000 visitors and 1,040 conversions (5.20%). The pooled rate is 2,040/40,000 = 5.10%. The pooled standard error is √(0.051·0.949·(1/20000 + 1/20000)) ≈ 0.0022, so:

z = (0.052 − 0.050) / 0.0022 ≈ 0.91, which maps to a p-value of about 0.36. Well above 0.05, so no significant difference. On the split, 20,000 against 20,000 gives an expected 20,000 per group, so χ² = 0 and p-value 1: perfectly balanced. Both reads pass, and the verdict is passed the A/A test. That is exactly what the tool shows when you open the page.

How to read it and where it fools you

The green verdict means your tool behaved as it should: it split traffic fairly and measured conversion with no visible bias. A red verdict is the valuable alert. If the flag is on conversion, identical groups converted differently, which points to unfair randomization or biased tracking. If it is on the split, the split engine did not deliver the 50/50. If it is both, something is quite wrong in the instrumentation.

The most important limit: an A/A that passes does NOT prove the tool is perfect, only that it did not fail at that volume. And remember the false positive: even with everything right, about 5% of A/A tests will flag a conversion difference by pure chance at 95% confidence. So a single flag deserves a second A/A before you conclude there is a bug. Once the tool is calibrated, go to the significance calculator to read your real A/B test, and get the context in the guide what is A/B testing.

Best practices with the A/A test

The A/A is a routine exam of your tool, not something you run only when you are suspicious. Use these rules to get value out of it.

• Run an A/A when you adopt a new tool, change the snippet install or switch the source of your conversion data.
• Use real volume: an A/A with few visitors passes easily and proves nothing.
• Treat a single flag as a suspicion, not a sentence. Confirm with a second A/A before hunting the bug.
• Do not leave the A/A running forever competing with real tests: it burns traffic that would earn more in an A/B.

Frequently asked questions

What is an A/A test?

It is a test where both variations are exactly the same: the two groups get the same control, with no change at all. Since nothing differs between them, you expect the traffic split to be balanced and the conversion of both to be statistically indistinguishable. It validates that your testing tool splits traffic fairly and measures conversion without bias, before you trust any A/B test.

Why run an A/A before an A/B?

To calibrate the tool. If an A/A test flags a winner where there should be none, you catch a randomization, tracking or instrumentation bug before making decisions with money on the line. Running an A/A now and then, or whenever you switch testing tools, is the cheapest way to make sure your A/B results are trustworthy.

My A/A test came back significant. What happened?

In an A/A the conversion difference should not be significant. If it is, there are three suspects: the randomization is unfair (the split engine sends different profiles to each group), the tracking is biased (one variation fires the pixel at a different time, or drops events), or it is just the false positive that happens in about 5% of A/A tests at 95% confidence. Rerun with more data: if it persists, it is a bug, not chance.

How is this different from the significance calculator?

The significance calculator exists to find a winner in an A/B test: you want the difference to be real. This is the opposite: the two versions are identical and you want to confirm there is NO difference. On top of that, the A/A test also checks the split balance (sample ratio mismatch), which the significance calculator does not. Think of it as the sanity check of your tool, not the verdict of your test.

How many visitors do I need for an A/A test?

The more the better: with a small sample the test almost never flags anything, so a tiny A/A passes easily without proving the tool is sound. Use a volume close to your real A/B tests, in the thousands of visitors per group, to have the power to detect an actual problem. If you do not have that traffic, check the guide on how many visitors a test needs.

Keep going

Tool calibrated? Read your A/B test in the statistical significance calculator and check the split balance of any test with the SRM checker. To size the next test, use the sample size calculator.