P-Value Calculator for A/B Testing

This is the teaching p-value calculator: to understand the number, not just read a verdict. It shows the z, the p-value in one or two tails, and how the same result is accepted or rejected depending on the threshold you pick (0.10, 0.05 or 0.01). If what you want is the direct business decision, with who won, relative lift and confidence interval, use the statistical significance calculator: it answers "can I switch the page?". This one answers "what does this p-value actually mean?".

How to use it

1. Enter visitors and conversions for the control (A).
2. Enter the same two numbers for the variation (B).
3. Choose the sidedness: two-sided (default) or one-sided.
4. Read the highlighted p-value, the z statistic and the rate of each version.
5. Check the decision table: if the p-value falls below 0.10, 0.05 or 0.01, the result rejects the null hypothesis at that level.

How it works: the two-proportion z-test

The calculator compares the two rates with a pooled standard error and turns the distance between them into a z score. The p-value comes from that z:

Where p₁ and p₂ are the rates of A and B, n₁ and n₂ the visitors of each, and p̄ the pooled rate (all conversions over all visitors). The two-sided p-value is 2·(1 − Φ(|z|)) and the one-sided is 1 − Φ(z), where Φ is the standard normal distribution function. The one-sided value is exactly half the two-sided one for the same z.

Worked example (reproduces the default result)

With the values already filled in: A with 2,000 visitors and 100 conversions (5.00%) and B with 2,000 visitors and 130 conversions (6.50%). The pooled rate is 230/4,000 = 5.75%. The standard error is about 0.007362, so z = 0.015 / 0.007362 = 2.04. The two-sided p-value is 2·(1 − Φ(2.04)) = 0.0416 and the one-sided is half of that, 0.0208.

In the decision table, that 0.0416 falls below 0.10 and 0.05 (rejects the null at 90% and 95% confidence) but above 0.01 (does not reject at 99%). This is exactly what the tool shows when you open the page, and it is the best example of why the threshold matters: the same data "wins" or "does not win" depending on the risk you accepted up front.

How to read it and where it misleads

The p-value measures one thing only: how surprising your data is if A and B were equal. It does not give the chance that B is better, does not give the size of the win, and does not prove the null is true when it comes out high. A p-value of 0.20 almost never means "the versions tied": it almost always means there was not enough sample to separate signal from noise. Before concluding there is no effect, check the planned size with the sample size calculator.

The other costly mistake is peeking: watching the p-value and stopping the moment it crosses 0.05 badly inflates the false positive rate. Fix the sample before you start, run to the end, and only then read the number here. And remember that a tiny p-value with a +0.2% lift may not pay for the switch: significance is not effect size.

Good practice when reading the p-value

The p-value is a risk ruler, not a sentence. To avoid deciding wrong with the right number, check these points.

• Pick the threshold (0.10, 0.05 or 0.01) by the cost of being wrong, before you see the data. Never after.
• Use two-sided by default. Only go one-sided if the direction was set before running.
• Translate the lift into revenue before celebrating. A low p-value with a tiny effect can be worth little.
• Confirm the planned sample was reached. Reading before that is peeking and it misleads.

Frequently asked questions

What is a p-value, in one sentence?

It is the probability of seeing a difference as large as yours (or larger) by chance alone, assuming A and B are actually equal. A p-value of 0.04 means a 4% chance of getting this result in a world where there is no difference at all. The smaller it is, the harder it is to explain the result as luck.

What is a p-value NOT?

It is not the probability that the null hypothesis is true, not the probability that your variation is better, and not the size of the effect. A p-value of 0.04 does not mean "96% chance B wins". It only measures how surprising your data is under the assumption of no difference. Mixing this up is the single most common reading mistake.

What is the difference between one-sided and two-sided?

The two-sided test asks "are A and B different?" and splits the risk across both tails. The one-sided test asks "is B greater than A?" and puts all the risk on one side, which makes the p-value exactly half the two-sided one for the same z. Use two-sided by default: it is more honest and it is what the market expects. Only pick one-sided if you fix the direction BEFORE running the test.

Is p = 0.05 a sacred rule?

No. The 0.05 threshold is a historical convention, not a law of nature. A p-value of 0.049 and one of 0.051 are practically the same evidence. That is why the calculator shows the decision at three thresholds (0.10, 0.05 and 0.01): it makes clear that "significant" is a risk choice, not a property of the result. Pick the threshold by the cost of being wrong, before you see the data.

Which test does this calculator run?

It compares two conversion rates (proportions): visitors and conversions for A and B, the classic A/B test case. It uses the two-proportion z-test. For continuous means (revenue per visitor, time on page) the correct test is a different one. If you want the ready business verdict, with lift and confidence interval, use the significance calculator.

Keep going

Understood the p-value? The next step is the business verdict, with lift and confidence interval, in the significance calculator. To plan the test up front, size it with the sample size calculator. The concept behind it is in the guide A/B testing statistical significance.