Equivalence Testing: Proving a Variant Did Not Hurt

A non-significant A/B test does not prove the two versions tied. It proves only that the test could not tell them apart. To claim a variant did not hurt you need an equivalence test: in the worked example below, two tests with the same non-significant verdict (p-values of 0.5946 and 0.8395) reach opposite conclusions, because one establishes equivalence within half a percentage point and the other leaves a loss of more than half a point wide open. This guide covers how to set the margin, which interval to read, and what a tie costs in traffic. It is part of our complete guide to A/B testing and the direct companion to guardrail metrics, which only work when there is a rigorous way to say “it did not get worse”.

The burden of proof is on the wrong hypothesis

The traditional A/B test was designed to answer one question: is there a difference? The null hypothesis is that there is none, and the experiment only rejects it when the evidence for a difference is strong enough. Walker and Nowacki describe the structure bluntly: the burden of proof rests on the research hypothesis, which is established only when there is strong enough evidence in its favour. When that evidence does not materialise, the correct conclusion is merely that equality cannot be ruled out.

The trouble starts when the business question runs the other way. Nobody asks “does the new checkout convert differently?” when the goal is to replace an expensive checkout with a cheap one. The real question is “does the new checkout convert worse?”, and the answer everyone wants is no. At that point the traditional test is answering a different question, and the high p-value gets read as proof of a tie, which it never was.

study type	null hypothesis	research hypothesis
traditional comparative	there is no difference between the options	there is a difference between the options
equivalence	the options are not equivalent	the new option is equivalent to the current one
non-inferiority	the new option is inferior to the current one	the new option is not inferior to the current one

The table is a direct rendering of the hypothesis framework in Walker and Nowacki. Note that equivalence is not the traditional comparison under a new name: it is that comparison turned upside down. The null becomes non-equivalence, and it is the null that the data have to knock down.

Two readings of the same “no difference”

The worked example compares two versions of a page with a 4 percent baseline conversion rate. The first test ran large, the second ran small, and both ended without statistical significance.

Large test. 60,000 visitors per variant, 2,400 conversions in control (4.0000 percent) and 2,364 in the variant (3.9400 percent). The two-proportion z test returns z of minus 0.5322 and a p-value of 0.5946. The 95 percent interval for the difference runs from minus 0.2809 to plus 0.1609 percentage points.

Small test. 8,000 visitors per variant, 320 conversions in control (4.0000 percent) and 315 in the variant (3.9375 percent). The z is minus 0.2025 and the p-value is 0.8395. The 95 percent interval runs from minus 0.6675 to plus 0.5425 percentage points.

Under the traditional reading, the two cases are identical: no difference detected. Notice, though, that the small test has the larger p-value, the one habit reads as “more alike”. It is the opposite. Its interval is more than three times wider, and it leaves open the possibility that the variant is losing two thirds of a percentage point, which on a 4 percent baseline is a relative drop of almost 17 percent.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Paste either pair above into the calculator to reproduce the p-values and the 95 percent intervals. It is the same implementation behind the numbers in this article, so the check is direct.

Equivalence testing by the TOST procedure

The most common route to an equivalence claim is the two one-sided tests procedure, TOST, proposed by Schuirmann. The operating rule fits in a sentence, in Walker and Nowacki’s formulation: equivalence is established at the alpha level when the 1 minus 2 alpha confidence interval for the difference is contained entirely within the interval from minus delta to plus delta.

The detail almost everyone gets wrong is the level of the interval. With alpha of 0.05, the correct interval is the 90 percent one, not the 95 percent one. The reason is that the procedure amounts to running two one-sided 5 percent tests, one against each edge of the margin, and two one-sided 5 percent tests describe exactly a 90 percent interval. Hoenig and Heisey report the same result from Schuirmann, and add that the approach is always somewhat on the conservative side.

Since every significance calculator reports the 95 percent interval, the conversion is trivial: divide the half-width of the 95 percent interval by 1.96 to get the standard error, then multiply by 1.645 to get the half-width of the 90 percent interval.

In the small test, the half-width of the 95 percent interval is 0.6050 percentage points. Divided by 1.96 that gives a standard error of 0.3087 points. Multiplied by 1.645 it gives 0.5077, so the 90 percent interval runs from minus 0.5702 to plus 0.4452 percentage points. It does not fit inside minus 0.5 to plus 0.5: equivalence is not established.

In the large test, the same path gives a standard error of 0.1127 points and a 90 percent interval from minus 0.2454 to plus 0.1254 percentage points, entirely inside the margin. Equivalence is established at the 5 percent level.

Anyone who prefers a p-value to an interval has an option, also described by Walker and Nowacki: since the test amounts to two one-sided tests, the equivalence p-value is the larger of the two one-sided p-values. In the large test the two z values against the edges are 3.9031 and 4.9676, and the TOST p-value is 0.000047. In the small test the z values are 1.4173 and 1.8223, and the TOST p-value is 0.0782, above 0.05: nothing established. The authors recommend reporting both, interval and p-value.

The margin is the decision that matters

Walker and Nowacki call determining the margin the most critical step in the procedure, and the reason is arithmetic before it is conceptual: the margin enters squared in the denominator of the sample size. A smaller margin means a narrower equivalence zone, harder to establish and far more expensive.

Two of their rules apply verbatim to product experimentation. First: the margin has to be fixed before the data are collected, or type I error control is gone. Second: the margin is the maximum loss you are willing to accept in return for the non-efficacy advantages. In clinical trials those advantages are fewer side effects or lower cost. In product they are the same thing under other names: a faster page, a cheaper stack to maintain, one less dependency.

The authors also record an uncomfortable fact about practice: in a review of 162 reports of equivalence and non-inferiority trials published in 2003 and 2004, about 80 percent did not justify their choice of margin and 28 percent did not take the margin into account in the sample size calculation. The translation to our context is direct: writing “we want to make sure it does not get worse” in the planning doc is not setting a margin. Setting a margin means writing down the number and the reason for it.

What proving a tie costs

The sample size formula for TOST with a true difference of zero follows the standard shape, with the margin in place of the minimum detectable effect and the 1 minus beta over 2 percentile in place of the 1 minus beta one:

// N per variant to establish equivalence (true difference zero)
// zA = percentile of 1 - alpha; zB = percentile of 1 - beta/2
function nEquiv(p, delta, alpha = 0.05, power = 0.8) {
  const zA = normInv(1 - alpha);           // 1.6449 for alpha 0.05
  const zB = normInv(1 - (1 - power) / 2); // 1.2816 for power 0.8
  return Math.ceil((2 * p * (1 - p) * (zA + zB) ** 2) / delta ** 2);
}

On a 4 percent baseline, alpha of 0.05 and 80 percent power, the table looks like this:

equivalence margin	N per variant	total traffic	days at 60,000 visitors per week
0.10 pp	657,704	1,315,408	154
0.20 pp	164,426	328,852	39
0.30 pp	73,079	146,158	18
0.50 pp	26,309	52,618	7
0.75 pp	11,693	23,386	3
1.00 pp	6,578	13,156	2

The non-linearity is the point to take away: because the margin enters squared, halving the margin multiplies the sample by four. Going from 1 point to 0.1 point multiplies it by a hundred. That is why the margin conversation has to be honest from the start: too tight a margin does not make the test more rigorous, it makes the test impossible.

The small test in the example had 8,000 visitors per variant. Establishing equivalence within half a percentage point would have taken 26,309, more than three times as many. It did not fail through bad luck: it was born too small for the question being asked of it.

A check against the literature

It is worth showing that this arithmetic is not homemade. Walker and Nowacki publish a table of sample sizes computed in the PASS software for a scenario with true efficacies of 28 and 33 percent, 0.80 power and alpha of 0.05. Running the same closed-form expression, with the true 5 point difference entering the calculation, the numbers line up:

margin	PASS (Walker and Nowacki)	closed form in this article
0.06	26,185	26,134
0.07	6,547	6,534
0.08	2,910	2,904
0.09	1,637	1,634
0.10	1,048	1,046
0.11	728	726
0.12	535	534

The gap stays under 0.3 percent on every row, which is what you expect from slightly different variance conventions. The most important thing the table shows is not the numerical agreement, it is the shape of the curve: between a 6 point margin and a 12 point margin, the required sample falls by a factor of almost 50.

Non-inferiority: when only one side matters

Most product cases do not need two-sided equivalence. If the variant lifts conversion, nobody complains. What must not happen is a drop. That is the non-inferiority structure, and its rule is also one line: non-inferiority is established at the alpha level when the lower bound of the 1 minus 2 alpha interval sits above minus delta.

In the small test, the lower bound of the 90 percent interval is minus 0.5702 points, below minus 0.5: not even non-inferiority holds. In the large test, the lower bound is minus 0.2454, comfortably above the margin, so the variant is non-inferior to the control within half a percentage point.

business question	correct test	criterion with margin delta
does the variant change conversion?	traditional comparison	the 95 percent interval excludes zero
is the variant practically the same?	equivalence (TOST)	the 90 percent interval fits between -delta and +delta
does the variant at least not hurt?	non-inferiority	the lower bound of the 90 percent interval sits above -delta

This is the reading missing from most experimentation programmes: every guardrail metric is, in practice, a non-inferiority test in disguise. When a team declares that “the guardrail did not flag anything” merely because the p-value was high, it is using the test with the burden of proof reversed, which is exactly the error Walker and Nowacki describe.

How this fits into test design

Three decisions belong in the pre-registered analysis plan, before any data:

1. Which metric is superiority and which is non-inferiority. The primary metric is usually superiority; guardrails are non-inferiority. Mixing the two readings after the result is picking the ruler that flatters the conclusion you wanted.
2. The margin for each guardrail, with a justification. “We accept up to 0.3 percentage points of checkout conversion loss in exchange for 400 milliseconds less load time” is a margin. “It cannot get worse” is not.
3. The sample size computed from the margin. If the planned sample does not support the chosen margin, one of the two has to change before the test starts, not after.

The minimum detectable effect and the equivalence margin are cousins: both translate a business decision into a number that enters squared in the denominator of the sample size. The difference is the direction of the question. MDE is the smallest gain worth detecting; the margin is the largest loss you agree to ignore.

Common mistakes

• Reading a high p-value as proof of a tie. This is the central error, and it gets worse the smaller the test, because small tests produce high p-values effortlessly.
• Using the 95 percent interval for TOST. With alpha of 0.05 the correct interval is 90 percent. Using 95 makes the test more conservative than declared and kills true equivalences.
• Choosing the margin after seeing the result. A margin set after the data always fits the observed interval. That is not analysis, it is decoration.
• Applying one margin to metrics of different weight. Half a percentage point off checkout conversion and half a point off the click rate of a secondary banner do not cost the same.
• Confusing equivalence with non-inferiority on guardrails. Demanding two-sided equivalence on a guardrail spends extra traffic proving something nobody needed proved: that the variant did not improve things too much either.
• Forgetting the margin is a promise. Declaring equivalence within half a percentage point is asserting that losing half a point is agreed. If leadership would not sign that sentence, the margin is wrong.

Make this automatic with Donnu

The question “did it get worse?” comes up in every test that swaps infrastructure, drops a form field or simplifies a page, and it is systematically answered with the wrong tool: the p-value of the superiority test.

At Donnu, every experiment carries its primary metric and its guardrail metrics declared separately, with each guardrail’s margin set at design time rather than at reading time, and the report shows the interval for the difference next to the margin instead of collapsing everything into a significance stamp. To check any figure by hand, the significance calculator returns the 95 percent interval for any pair of groups (the conversion described above gets you to the 90 percent one), the confidence interval calculator works the precision of a single rate, and the sample size calculator sizes the test before it starts.

References

• Walker, E. and Nowacki, A. S. Understanding Equivalence and Noninferiority Testing. Journal of General Internal Medicine, volume 26, number 2, pages 192 to 196, 2011. Source of the hypothesis framework for the three study types, of the definition of the equivalence margin as the maximum clinically acceptable difference accepted in return for secondary benefits, of the TOST rule (equivalence established at alpha when the 1 minus 2 alpha interval is contained in minus delta to plus delta), of the non-inferiority rule based on the lower bound of that interval, of the TOST p-value as the larger of the two one-sided p-values, of the requirement to fix the margin before data collection, of the PASS sample size table for efficacies of 0.28 and 0.33, and of the Le Henanff review of 162 reports in which about 80 percent did not justify the margin and 28 percent did not consider it in the sample size calculation. pmc.ncbi.nlm.nih.gov.
• Hoenig, J. M. and Heisey, D. M. The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, volume 55, number 1, pages 19 to 24, February 2001. Source of section 4 on equivalence testing: the reversal of the burden of proof (failing to show a difference is not sufficient, one must be fairly certain a large difference does not exist), the formulation of the null as the absolute effect being at least delta, Schuirmann’s result that a 1 minus 2 alpha interval lying entirely between minus delta and plus delta rejects non-equivalence at level alpha, and the observation that the approach is always a bit on the conservative side. stat-help.com.
• Lakens, D. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. Social Psychological and Personality Science, volume 8, number 4, pages 355 to 362, 2017. Source of the claim that it is statistically impossible to support the hypothesis that a true effect size is exactly zero, and of what is possible instead: rejecting effects large enough to be considered worthwhile, with the bounds set from the smallest effect size of interest. pmc.ncbi.nlm.nih.gov.
• Schuirmann, D. J. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, volume 15, pages 657 to 680, 1987. The original TOST paper, cited here through the accounts of Walker and Nowacki and of Hoenig and Heisey, who describe its content and its results. link.springer.com.

Read next: Guardrail metrics · Minimum detectable effect · Pre-registered analysis plan · A/B testing statistical significance · Significance calculator · Leia em português