Region of Practical Equivalence (ROPE) in A/B Testing

The region of practical equivalence (ROPE) is a Bayesian decision rule, proposed by psychologist John Kruschke, that declares a result “practically equivalent” to no effect whenever its entire credible interval falls inside a band of values too small to matter, instead of only asking whether that interval excludes zero. It answers a question that a plain significance check cannot: not “is this probably a real difference”, but “is this difference big enough to be worth acting on.” This guide explains what a ROPE is, how to set its width for an A/B test, and walks through a full numeric example, computed with this blog’s own Bayesian engine, of a result that is statistically real and simultaneously practically worthless.

This piece is the third leg of a small cluster on reading a Bayesian A/B test honestly. Our complete guide to Bayesian A/B testing covers the mechanics of priors and posteriors; our guide to credible interval vs confidence interval covers what that interval actually means; our guide to expected loss turns the posterior into a single risk number. ROPE sits next to all three: it takes the same credible interval and asks one more question that none of the others ask on their own, is the whole plausible range close enough to zero to ignore.

The question a plain significance check cannot answer

Every method for reading an A/B test, frequentist or Bayesian, is built to answer some version of “is this difference real, or could it be noise.” A p-value below 0.05 answers it. A 95% credible interval that excludes zero answers it. A high P(B beats A) answers it. All three are, at their core, a direction check: is the effect probably positive, probably negative, or could it plausibly be zero.

None of them, on their own, answer a second and completely different question: how big is the effect, and is that size worth anything. A one-visitor-in-a-million shift can be “real” in the direction sense and simultaneously invisible in every way that matters to a business. This is exactly the gap that the industry has started calling practical significance versus statistical significance, and VWO carries the distinction into the test objective itself: its campaign duration documentation describes an “Improvement” mode, where you test for a strict improvement in the primary metric, and an “Improvement or Equivalence” mode, where you test that a variation is not worse than the baseline, with the ROPE serving as the range of differences treated as basically the same.

Kruschke made the same point formally in a 2018 paper for Advances in Methods and Practices in Psychological Science, arguing that Bayesian estimation should shift away from asking whether a parameter equals a null value exactly and toward asking whether the parameter falls inside a band of values that are, for practical purposes, equivalent to the null. The motivation behind that shift is documented by the bayestestR package that implements the method: for a continuous parameter, the probability of any exact point value is zero, so testing the point null itself was never the useful question.

How the ROPE decision rule works

A ROPE is just an interval you choose in advance, usually centered on zero (or on “no lift”), sized to whatever margin your business genuinely does not care about. Kruschke and Liddell’s decision rule then compares that band against the posterior’s highest density interval (HDI), the shortest interval that holds a given amount of credible mass, typically 95%:

• If the entire HDI falls outside the ROPE, reject the null: the effect is real and large enough to matter.
• If the entire HDI falls inside the ROPE, accept the null: whatever the true effect is, it is practically equivalent to no effect.
• If the HDI overlaps the ROPE boundary on either side, stay undecided: keep collecting data, the current evidence cannot settle the size question yet.

For the roughly symmetric posteriors that show up in most A/B test conversion-rate comparisons, the HDI and the plain equal-tailed credible interval this blog’s calculator reports land close enough together that the same rule applies to either one in practice.

Choosing a ROPE width for an A/B test

Kruschke’s own generic starting point, meant for standardized effects across many fields of research, is roughly plus or minus 0.1 standard deviations, a value he ties to Cohen’s conventional threshold for a “negligible” effect size, with a wider plus or minus 0.18 suggested for log-odds ratios in logistic-style models, according to the documentation of the bayestestR R package that implements his method. That default was never built with conversion-rate A/B tests in mind, and it does not translate directly onto a “percent lift” scale.

The A/B testing industry has converged on a more direct convention instead: define the ROPE as a band of relative lift. VWO’s own documentation states plainly that the platform uses “a conservative ROPE value of plus or minus one percent as a good default to start with,” and shows a worked example: with a baseline rate of 40%, a business decision that anything between 38% and 42% is “the same as baseline” turns into a ROPE of plus or minus 5% relative. The number is not fixed; it is a genuine business input, same as picking a minimum detectable effect before calculating a sample size.

A few practical anchors for setting that number on your own tests:

Situation	A reasonable starting ROPE (relative lift)	Why
High-traffic checkout or pricing page, revenue-critical	About plus or minus 1%	VWO’s own documented default; small enough that a “win” below it would not move revenue meaningfully
Expensive change to implement (new backend flow, redesign)	Wider, about plus or minus 2 to 3%	The engineering cost only pays for itself above a bigger minimum win
Cheap, reversible UI tweak (copy, color, spacing)	Narrower, about plus or minus 0.5%	Little cost to ship, so a smaller genuine improvement is still worth taking
Generic standardized effect outside A/B testing context	About plus or minus 0.1 standard deviations	Kruschke’s own default, tied to Cohen’s negligible-effect convention, not calibrated to lift percentages

The common thread: a ROPE should reflect the smallest lift that would actually change what your team does next. If a plus 0.3% relative lift would not justify shipping the change, it belongs inside the ROPE, whatever the credible interval says about zero.

A worked example: real, and still not worth shipping

Here is a case built with this blog’s own Beta-Binomial engine (the same one behind the calculator below), where a naive “does it exclude zero” check and a ROPE check disagree.

Picture a checkout page at a very high-traffic retailer, running a test long enough, and to enough visitors, to collect 2,000,000 visitors per variant. Control A converts at exactly 20.00% (400,000 conversions). Variation B converts at 20.10% (402,000 conversions), a relative lift of 0.50%.

Feeding those two arms into a Beta(1,1) posterior (a uniform prior, the same one this blog’s calculator uses) gives a posterior mean conversion rate of about 20.00% for A and 20.10% for B, with a posterior standard deviation for each arm of roughly 0.028 percentage points, tight enough that at two million visitors per arm the prior has essentially no influence left. Combining the two arms’ posterior variance gives a standard error for the difference of about 0.040 percentage points. That is small enough, relative to the 0.10 percentage-point gap between the two rates, that the 95% credible interval on the lift itself (using the standard normal approximation to the difference of two Beta posteriors, accurate here because both posteriors have hundreds of thousands of pseudo-observations behind them) works out to approximately 0.11% to 0.89% relative lift. It excludes zero by a comfortable margin, corresponding to roughly a 99.4% posterior probability that B genuinely beats A.

Type those same numbers into the calculator, 2,000,000 visitors and 400,000 conversions for A, 2,000,000 visitors and 402,000 conversions for B, and you can watch it live:

A (control)

VisitorsConversions

B (variation)

VisitorsConversions

-probability that B beats A

Rate of A (posterior)-

Rate of B (posterior)-

Probability A wins-

Risk of choosing B (expected loss)-

Relative lift (means)-

Beta-Binomial model with a uniform Beta(1,1) prior and a 95% credible interval. Deterministic calculation, updates live.

You should see something close to a 99% probability that B beats A, a relative lift near 0.5%, and an expected loss from choosing B that rounds to essentially zero, all consistent with the numbers above. A naive rule that only checks “does the interval exclude zero, and is P(B beats A) high” would call this an obvious, confident win, ship immediately.

Now apply a plus or minus 1% ROPE, the same conservative default VWO documents. The entire credible interval, roughly 0.11% to 0.89%, sits inside that band. Every plausible value for the true lift, from the low end to the high end of what the data supports, is a change nobody would notice in the business. The honest verdict is not “ship it”, it is “this is probably real, and it is too small to act on.” That is a different decision from the one the naive rule reached, off the exact same data.

This is not a quirk of one made-up scenario. It is the mechanical consequence of running a test with enough traffic to resolve an effect far smaller than anyone actually cares about, which mostly happens at the scale of very large retailers and platforms, exactly the audience VWO is writing its ROPE documentation for. Smaller sites will rarely reach a sample size where this gap shows up; when they do get a “significant” 0.5% relative lift, it is usually because the interval is still wide enough that the upper end promises something real, not because the whole range is trapped this close to zero.

Where ROPE fits next to expected loss and the credible interval

The three ideas in this cluster are not competing, they read the same posterior for different purposes:

Tool	Question it answers	Output shape
Credible interval	What range of true lift is plausible, given the data	A range, with a stated probability of containing the truth
Expected loss	If I ship the wrong variant, how much would that cost on average	A single number, in the metric that matters
ROPE (this article)	Is the plausible range small enough to treat as no effect at all	A categorical verdict: real, equivalent, or undecided

A team that only checks P(B beats A) can be fooled by a large-sample, small-effect result exactly like the one above. A team that also checks expected loss gets a genuine risk number, but even a near-zero expected loss does not, by itself, say whether the winning side of the bet was worth the shipping cost in the first place. Layering a ROPE on top of both closes that gap: it is the one tool of the three built specifically to say “yes, this is probably real, and no, it still does not matter.”

Automate This on Donnu

The hardest part of a Bayesian result is not computing the posterior, it is deciding, cleanly, when a “yes it’s probably a win” reading is actually too small to bother shipping. Donnu’s Bayesian reporting already surfaces the probability that a variant wins and the expected loss of choosing it side by side, the same two numbers this article started from, so you are never left guessing at a bare P(B beats A) with no sense of the effect’s real size.

Start a free 14-day trial and read your next test’s result with both the direction and the size of the effect in view. For the full mechanics behind these posteriors, see our complete guide to Bayesian A/B testing, for what the interval itself is allowed to mean, see credible interval vs confidence interval, and for turning the same posterior into a single risk number, see expected loss in Bayesian A/B testing.

References

• Kruschke, J. K. Rejecting or Accepting Parameter Values in Bayesian Estimation. Advances in Methods and Practices in Psychological Science, 1(2), 270-280, 2018. doi.org/10.1177/2515245918771304.
• VWO. What is the Region of Practical Equivalence (ROPE)? The conservative plus or minus 1% default and the 40% baseline example with a 38% to 42% band. help.wingify.com.
• VWO. Estimate Your Campaign Duration. The “Improvement” and “Improvement or Equivalence” modes and the ROPE as the range of differences treated as basically the same. help.wingify.com.
• Makowski, D., Ben-Shachar, M.S. & Lüdecke, D. Region of Practical Equivalence (ROPE), bayestestR documentation (implements Kruschke’s decision rule). easystats.github.io/bayestestR.