Statistical Power Calculator for A/B Testing

This is the retrospective calculator: did the test you already ran have enough power? It answers "with the sample I had, what was my chance of catching the effect?". It differs from the sample size calculator, which plans how many visitors to collect BEFORE you start, and from the MDE calculator, which tells you the smallest effect your traffic can catch. Use this one to diagnose a test that came back "not significant" and understand whether it was a real tie or just too little sample.

How to use it

1. Enter the control conversion rate (A's, in %).
2. Choose the effect you wanted to detect: the smallest win that would still change your decision, relative (%) or absolute (points).
3. Enter the sample per variation the test actually had.
4. Leave confidence 95% and two-sided (the defaults) or match what you used.
5. Read the achieved power and compare it with the sample that would give 80% power for the same effect.

How it works: the formula

Power is the exact inverse of the sample size math. With N per variation fixed, we isolate the power term in the same two-proportion normal approximation and pass it through the normal distribution function:

Where n is the sample per variation, δ the absolute difference between the rates (p₂ − p₁), p₁ the base rate, p₂ the target rate (base plus the effect), p̄ the average of the two, z_α the confidence critical value (1.96 for 95% two-sided) and Φ the standard normal distribution function. By construction, if you feed in the N the sample size calculator asked for, power comes back to 80%.

Worked example (reproduces the default result)

With the values already filled in: base rate 5%, target effect +10% relative (target of 5.5%), 20,000 visitors per variation, 95% two-sided confidence. The absolute difference is 0.5 point (δ = 0.005). The noise term under the null, z_α·√(2·p̄·(1−p̄)), is about 0.6182. The signal, √20,000 · 0.005, is 0.7071. The standard deviation under the alternative, √(p₁·(1−p₁) + p₂·(1−p₂)), is 0.3154.

So z_β = (0.7071 − 0.6182) / 0.3154 = 0.2819, and power is Φ(0.2819) = 61.1%. In other words: this test had only a 61% chance of detecting a +10% win. To reach 80% power for the same effect you would need 31,234 visitors per variation, not 20,000. That is exactly what the tool shows when you open the page, and it is the classic underpowered test.

How to read it and where it fools you

Power changes everything about reading a non-significant result. With high power (80% or more), a "not significant" is real evidence that the effect, if any, is too small to be worth the switch. With low power (60%, 50%), the same "not significant" almost always means a lack of sample: you did not tie, you went blind. Reading both cases the same way is the mistake that makes teams discard good variations too early.

The key limit: avoid so-called "observed power", which computes power from the effect measured in the test itself. That is circular and tells you nothing new. Here you enter the target effect you defined up front (the same one that would go into the sample size calculator), which keeps the number honest and useful for diagnosis. And remember: high power does not fix collection bias, SRM or peeking; it only covers the false-negative risk.

Best practices with power

Power is the forgotten twin of significance. To avoid deciding wrong, check these points.

• Set the target power (80% or 90%) and the minimum effect BEFORE you start, never after seeing the data.
• If the test came back "not significant", compute the power here before concluding it was a tie.
• Low power with a small effect signals missing traffic: gather more sample or chase a bigger effect.
• Do not confuse power with significance: they are opposite risks and you need to control both.

Frequently asked questions

What is statistical power, in one sentence?

It is the probability that your test detects a real effect when one truly exists. Power of 80% means that, if variation B really is better by the size you defined, the test has an 80% chance of catching it with significance. The remaining 20% is the risk of missing a true win (a false negative).

Is power the same as confidence?

No, they are two different risks. Confidence (1 minus alpha) protects against the false positive: calling B a winner when the two actually tied. Power (1 minus beta) protects against the false negative: missing a win that exists. A test can have 95% confidence and still only 50% power if the sample is too small for the effect you are chasing.

Why is 80% power the default?

It is the market convention, the same level the sample size calculator uses by default. It balances the cost of collecting more sample against the risk of missing real effects. Some teams raise it to 90% for expensive decisions. Below 80% the test starts going blind: you risk ending it without seeing a win that was actually there.

Does a non-significant result mean there is no difference?

Only if power was high. If the test had 60% power, a non-significant result is almost always a lack of sample, not proof of a tie. That is why power matters so much when reading results: with low power, "not significant" concludes nothing. With high power, a non-significant result is real evidence that the effect, if any, is small.

Is this the same as post-hoc power?

This calculator answers the useful planning question: given the effect size you wanted to detect and the sample you had, what was the power of the design? That is legitimate and helps diagnose an underpowered test. What statisticians criticize is "observed power", computed from the effect measured in the test itself, which is circular. Here you enter the target effect, not the observed one.

Keep going

Found out the test was underpowered? Plan the next one with the sample size calculator and see the smallest effect your traffic can catch with the MDE calculator. For the verdict of a test that already finished, use the significance calculator.