Permutation Tests in A/B Testing: P-value Without a Formula

A permutation test computes the p-value by shuffling the control and variant labels thousands of times over your own data, instead of assuming the difference between means follows a normal distribution. It is not universally more accurate: on a 50/50 split with a well-behaved metric it returns essentially the same number as the z test. It rescues the result exactly where the normal approximation breaks, and the simulation in this article shows how badly it breaks: on a 1 percent exposure ramp with revenue per user, the z test rejected 16.44 percent of A/A tests at a nominal 5 percent alpha, while the permutation test stayed at 4.78 percent. This guide covers how the arithmetic works, where it matters, how to run it, and what it does not fix. It is part of our complete guide to A/B testing and extends what we have already written about statistical significance.

The permutation test idea, in one sentence

You have 800 visitors in control with 24 conversions and 800 in the variant with 36. The question the test asks is: if the label “control” or “variant” made no difference at all, how often would a random split of those 1,600 visitors into two groups of 800 produce a difference as large as the one you saw?

The z test answers that with a formula that approximates the distribution with a normal curve. The permutation test answers the same question literally: it does the draw, many times, and counts.

The formal justification is the same one that supports the experiment itself. Guo, Lee and Toulis note that the approach dates back to Fisher in 1935, and that procedures of this kind are finite-sample valid for arbitrary data distributions, because they leverage the known experimental variation in assigning treatments rather than an assumption about the population. You are the one who randomized the traffic; the test just repeats that randomization.

Where the closed-form formula breaks

The honest answer is: less often than the folklore suggests. A typical A/B test, 50/50 split and hundreds of thousands of users, does not need permutation. We measured this.

The simulated experiment: revenue per user, 4 percent of users purchase, the purchase amount is lognormal, and a total of 20,000 users is split in different proportions between control and variant. The metric is violently skewed (the skewness coefficient of the pooled sample sits around 15, and the rule of thumb from Kohavi and coauthors would ask for roughly 81,000 observations per arm just for the mean to be approximately normal). We ran thousands of A/A tests, meaning tests where both arms come from exactly the same distribution and any significant result is a false positive.

control/variant split	users in control	users in variant	z test: “variant worse” tail	z test: “variant better” tail	z test: total	permutation: total	replications
50/50	10,000	10,000	2.40%	2.45%	4.85%	5.00%	4,000
80/20	16,000	4,000	3.45%	1.27%	4.72%	4.13%	4,000
90/10	18,000	2,000	5.43%	0.75%	6.18%	5.12%	6,000
99/1	19,800	200	16.35%	0.09%	16.44%	4.78%	8,000

The nominal rate is 5 percent in total and 2.5 percent in each tail. Read the table top to bottom:

At a 50/50 split, the z test is right. 4.85 percent against a 5 percent target, with balanced tails, even with a skewness of 15 and a sample seven times smaller than the rule of thumb would ask for. That is not luck: Kohavi and coauthors make exactly this caveat in the paper where they publish the rule. With an equally sized split, the distribution of the delta is approximately symmetric, and it is perfectly symmetric with zero skewness under the null hypothesis, so the rule stops providing a useful lower bound. The skewness of the metric cancels between the two arms.

As the split becomes unbalanced, the tails become unbalanced with it. At 90/10 the total still looks almost acceptable (6.18 percent), but it hides a 5.43 percent tail against a 0.75 percent one. In other words: the test errs in the same direction almost every time, declaring the small arm to be worse.

At 99/1 the result is indefensible. One in six A/A tests produces a significant result, and 16.35 of the 16.44 points come from the “the 1 percent arm is worse” tail. The permutation test, on the same data, landed at 4.78 percent, inside the Monte Carlo error of the simulation (with 8,000 replications, the standard error around 5 percent is 0.24 percentage points).

The practical conclusion is specific, and it is not “abandon the z test”. It is: during an exposure ramp, which is exactly when you run 1 percent or 5 percent of traffic through a new variant, reading a revenue metric with a normal approximation is the worst case available, and it errs systematically against the variant. It is the same care we already asked for when writing about SRM and unequal traffic splits, except that here the problem is not the randomization, it is the reading.

The worked example: a low-traffic test

The second place where the closed-form formula errs is the most common of all: small counts. Take a low-volume test, 800 visitors per variant, 24 conversions in control (3.000 percent) and 36 in the variant (4.500 percent), an apparent relative gain of 50 percent.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Pasting those numbers into the calculator above, the z test returns z of 1.5791, a p-value of 0.11432 and a 95 percent interval on the difference from minus 0.3604 to plus 3.3604 percentage points. Not significant.

Running the permutation test on the same data, with 100,000 shuffles, 14,667 of them produced an absolute difference as large as the one observed. The p-value comes out at 0.14668.

Here the arithmetic has a strong check available: in the binary case with fixed margins, the permutation test converges to Fisher’s exact test, which is computable analytically from the hypergeometric distribution. The exact value is 0.147193. Our Monte Carlo approximation landed 0.00051 away from it, and the expected Monte Carlo standard error with 100,000 shuffles is 0.00112. The permutation hit the target within half a standard error.

method	p-value	how it was obtained
z test (normal approximation)	0.11432	closed-form formula, the same one in the calculator above
chi-square with Yates continuity correction	0.14776	closed-form formula with a continuity correction
Monte Carlo permutation, 100,000 draws	0.14668	shuffling the labels
Fisher’s exact test	0.14719	hypergeometric, no simulation

The reading: the normal approximation returns a p-value 22 percent lower than the exact one, and it is always in that direction at counts of this size. In a test that is not significant anyway, that is irrelevant. In a test that came out at 0.048 by formula and 0.061 exactly, that is the difference between shipping a variant and not shipping it. Note as well that the Yates correction, which is closed form and runs in one line, reached 0.14776, practically the exact value: on a 2 by 2 table it is a cheap and very good substitute for permutation. That matters most for anyone doing CRO on a low-traffic site, where dozens of conversions is the normal case rather than the exception.

How to run it, in 15 lines

The algorithm fits in one function. This is the code that produced the numbers above, and it depends on nothing:

function permutationP(control, variant, B, rand) {
  const bucket = control.concat(variant);
  const n = bucket.length, nA = control.length;
  const mean = (a) => a.reduce((s, x) => s + x, 0) / a.length;
  const observed = Math.abs(mean(variant) - mean(control));
  let extreme = 0;
  for (let k = 0; k < B; k++) {
    for (let i = n - 1; i > 0; i--) {          // shuffle the bucket
      const j = Math.floor(rand() * (i + 1));
      [bucket[i], bucket[j]] = [bucket[j], bucket[i]];
    }
    let sumA = 0; for (let i = 0; i < nA; i++) sumA += bucket[i];
    let sumB = 0; for (let i = nA; i < n; i++) sumB += bucket[i];
    const diff = Math.abs(sumB / (n - nA) - sumA / nA);
    if (diff >= observed - 1e-12) extreme++;
  }
  return (extreme + 1) / (B + 1);             // Phipson and Smyth, 2010
}

Three details that separate a correct implementation from a wrong one:

The plus 1 on both numerator and denominator is not a hack, it is the correct calculation. Phipson and Smyth showed that the naive version, dividing the count by B, understates the p-value by about 1 over B and raises the type I error rate of the test. Their argument is that permutation does not estimate the tail probability of a continuous distribution: it generates an exact discrete null distribution, and the observed statistic is itself one of the possible outcomes. Adding 1 to both sides also removes the absurdity of reporting a p-value of exactly zero.

The statistic has to be recomputed on every draw, including the denominator. If you freeze the variance estimated on the original data and only shuffle the means, you lose the property that makes the method work.

Use greater than or equal, not strictly greater. The comparison against the observed statistic must include ties, otherwise the test becomes anticonservative on discrete metrics.

How many shuffles

Monte Carlo permutation has an error of its own, and it is easy to size: the number of extreme draws is binomial, so the standard error of a p-value estimated at p is the square root of p times (1 minus p) over B.

shuffles (B)	standard error near p of 0.05	half width of the 95% interval	smallest possible p-value
999	0.00690	0.01351	0.001000
9,999	0.00218	0.00427	0.000100
99,999	0.00069	0.00135	0.000010
999,999	0.00022	0.00043	0.000001

With a thousand shuffles, a p-value estimated at 0.05 comes with an interval running from 0.036 to 0.064: you do not know which side of the threshold you are on. Ten thousand already settles the binary significance decision. A hundred thousand is worth it when you are going to publish the number, and it only makes a real difference when the p-value is small. And note the last column: the smallest p-value you can report is 1 over (B plus 1), so “p below 0.001” requires at least 999 shuffles by definition, no matter how convincing the data is.

What permutation does not fix

This is the part enthusiastic write-ups tend to skip. A permutation test corrects one thing: the approximation of the null distribution. It corrects nothing that comes before it.

Concretely:

• It requires exchangeability under the null. If the randomization broke, and you have a sample ratio mismatch, the observations are not interchangeable across labels and the test returns a correctly computed p-value for a broken experiment.
• It does not protect against dependence. If the unit of analysis is the pageview but the randomization unit is the user, shuffling individual pageviews destroys the dependence structure and produces an optimistic p-value, exactly the way an i.i.d. formula would. The shuffling has to happen at the same unit at which randomization happened, a topic we cover in randomization unit.
• It does not produce a confidence interval. Permutation answers a testing question. For uncertainty around the effect, the instrument is the bootstrap, which resamples with replacement inside each group instead of shuffling across groups.
• It does not fix peeking. Running a permutation every hour and stopping when it drops below 0.05 inflates type I error exactly the way running a z test every hour does, for the reason we describe in the peeking problem.
• It does not fix skewness itself. The rule of thumb from Kohavi, Deng, Longbotham and Xu says the minimum number of independent observations for the mean to be approximately normal is 355 times the square of the skewness coefficient, and it is worth applying when the absolute skewness exceeds 1. In the Bing metrics the authors publish, revenue per user has skewness 17.9 and asks for 114 thousand observations; capped revenue per user has 5.2 and asks for 9.7 thousand; sessions per user has 3.6 and asks for 4.70 thousand. Permutation does not reduce that number, it merely stops needing it in order to compute the p-value. The sample size calculation is unchanged, and capping the metric remains the intervention that buys the most sensitivity.

An application checklist

1. 50/50 split and a binary metric with hundreds of conversions per arm: use the formula. Permutation will not move the third decimal place.
2. Fewer than roughly 100 conversions per arm: check the p-value against Fisher’s exact test or chi-square with Yates before deciding. Both are closed form and the difference is already material.
3. A continuous or count metric on an unequal split (a 1, 5 or 10 percent ramp): permutation, no debate. That is where the z test errs most, and it always errs against the small arm.
4. A statistic with no closed-form formula (difference in medians, difference in quantiles, ratio of ratios): permutation for the p-value, bootstrap for the interval.
5. Shuffle at the randomization unit. If you randomized users, permute whole users, with all of their events together.
6. Fix the random seed and publish B. A Monte Carlo p-value without the number of shuffles is a number with no declared precision.
7. Use (b plus 1) over (B plus 1). Always.

Make this automatic with Donnu

The reason almost nobody runs permutation tests is not theoretical, it is operational: it requires storing observations at the level of the randomization unit, not just aggregated totals. Anyone who only has “visitors and conversions” in a spreadsheet cannot permute anything, because the individual information was already thrown away in the aggregation.

Donnu stores the event at the level of the randomized user, so the empirical null distribution is computable over any metric in the experiment, not only the conversion rate. In practice that means being able to read a 1 percent ramp without the 16 percent false positive bias this article’s table shows, and being able to test a difference in medians without inventing a formula for it. If you are starting out, start with the cheapest step: run the significance calculator and, if your counts are low, check the result with Yates before deciding.

References

• Guo, W., Lee, J. and Toulis, P. ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments. arXiv 2501.07722, January 2025. Source of the framing that randomization inference dates back to Fisher (1935), that Fisher randomization test procedures are finite-sample valid for arbitrary data distributions because they leverage the known experimental variation in assigning treatments, and that such tests are typically implemented as permutation tests in online experimentation. arxiv.org.
• Phipson, B. and Smyth, G. K. Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology, volume 9, issue 1, article 39, 2010. Source of the demonstration that the naive p-value is understated by about 1 over the number of permutations and that substituting an unbiased estimator for the exact p-value raises the type I error rate; of the argument that permutation should be viewed as generating an exact discrete null distribution rather than estimating a tail probability; and of the historical record that full enumeration is due to Fisher (1935), drawing a random subset of permutations to Dwass (1957) and the Monte Carlo test to Barnard (1963). arxiv.org.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the rule of thumb of 355 times the square of the skewness coefficient as the minimum number of independent observations for the mean to be approximately normally distributed, with the recommendation to apply it when absolute skewness exceeds 1; of the table of Bing metrics (revenue per user with skewness 17.9 and 114 thousand observations, capped revenue per user with 5.2 and 9.7 thousand, sessions per user with 3.6 and 4.70 thousand, time to success with 2.1 and 1.55 thousand); of the observation that capping revenue per user dropped skewness from 18 to 5.3 and allowed detection of a change 30 percent smaller at the same sample size; and of the caveat that with an equally sized split the distribution of the delta is approximately symmetric and the rule stops providing a useful lower bound. exp-platform.com.
• Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Source of the context for why experimentation platforms at scale look for analytical variance estimators rather than resampling, and why the computational cost of the bootstrap is the practical obstacle to adopting it in production. arxiv.org.

Read next: A/B testing statistical significance · Outliers and metric capping · Randomization unit · CRO for low-traffic sites · Sample ratio mismatch · Significance calculator · Leia em português