Guardrail Metrics: The Checks That Stop a Bad Ship
What guardrail metrics are, how to set a degradation threshold before launch, and why a non-significant guardrail is not a cleared guardrail.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
Guardrail metrics are the measures a change is not allowed to harm, even when the metric the test was built to move goes up. They exist because a winning primary metric is not the same thing as a safe ship, and the most common way teams get this wrong is to read a non-significant guardrail as a cleared one. This guide covers what a guardrail is and how it differs from a secondary metric, how to set the threshold that gives it teeth, the power arithmetic that quietly disarms most guardrails, and a starter set for ecommerce and SaaS. It is part of our complete guide to A/B testing and pairs with the design arithmetic in minimum detectable effect.
What guardrail metrics are, and what they are not
Dmitriev and colleagues, writing about metric interpretation at Microsoft in 2017, define guardrail metrics as measures that are not clearly indicative of the success of the feature being tested, but which the team does not want to significantly harm when making a ship decision. Page load time on a site like Bing or MSN is their canonical example: nobody launches a feature in order to make the page slower, and yet almost any feature added to a page increases load time a little. Small degradations are expected. Large ones are not allowed.
That definition draws a line that many experiment records blur. Four roles are worth keeping separate, because each one gets read differently.
The distinction that matters most in practice is guardrail versus secondary metric. A secondary metric is read for direction and explanation. A guardrail carries a veto: it can block a ship the primary metric won. Because that is a real power, it has to be assigned in advance. A metric promoted to guardrail status after results arrive is being used to justify a decision already taken, and one demoted after results arrive is worse.
Deng and Shi, describing metric development at Bing in 2016, add a second guardrail role worth knowing: guardrails also stand in for the goal metric when the goal metric does not apply. Their example is a search organization whose goal metric counts long dwell time clicks, which is meaningless for the team building instant answers whose whole job is to satisfy the user without a click. That team needs guardrails that capture the experience the shared goal metric cannot see.
The threshold, not the p-value, is what gives a guardrail teeth
Here is the failure mode that costs the most, and it hides inside a sentence that sounds responsible: “the guardrails came back not significant, so we shipped.”
A test is sized for its primary metric. Guardrails are usually rarer events with much lower baselines, and sample size scales with roughly one over the square of the effect you want to see, so a lower baseline buys much less resolution at the same visitor count. Take a checkout test sized to detect a 10 percent relative move on a 4.0 percent purchase rate: that needs 39,475 visitors per variation, 78,950 in total, about 14 days at 40,000 eligible visitors a week. At that exact sample, here is what each metric on the dashboard can actually resolve.
| Metric | Baseline | Smallest relative move detectable at 39,475 per variation | In percentage points |
|---|---|---|---|
| Add to cart | 22.0 percent | about 3.8 percent | 0.83 pp |
| Checkout started | 9.0 percent | about 6.4 percent | 0.58 pp |
| Purchase (the primary metric) | 4.0 percent | 10.0 percent | 0.40 pp |
| Refund rate | 1.2 percent | about 18.9 percent | 0.23 pp |
| Support contact rate | 0.8 percent | about 23.5 percent | 0.19 pp |
| Unsubscribe rate | 0.3 percent | about 39.8 percent | 0.12 pp |
Read the last three rows slowly. In a test that is perfectly well powered for its purpose, a refund rate that comes back “not significant” is consistent with anything from a real improvement to a 19 percent degradation, and an unsubscribe rate that comes back quiet is consistent with a 40 percent degradation. Nothing was cleared. The question was never asked with enough data to have an answer.
Dmitriev and colleagues document exactly this at MSN.com. Total page views per user rose 0.5 percent in one experiment with a non-significant p-value. For a business at that scale, a 0.5 percent move in page views is commercially meaningful. The confidence interval turned out to span roughly plus or minus 5 percent, and the design could only detect a change of 7.8 percent or larger at 80 percent power. Their conclusion is the sentence to hang above the dashboard: we cannot assume that we did not impact the underpowered metric.
Set your own baselines and see what your traffic resolves:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
The fix is not statistical, it is procedural. Declare a threshold for each guardrail before launch, then compare the upper bound of the confidence interval against that threshold instead of comparing the p-value against 0.05.
Worked example: the primary metric wins and the guardrail says nothing
A checkout redesign runs at the sample above: 39,475 visitors per variation, purchase baseline 4.0 percent, refund rate baseline 1.2 percent. The threshold declared before launch is that refunds must not rise by more than 10 percent relative, which at this baseline is plus 0.12 percentage points. Run these numbers yourself:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
The primary metric. Control finishes with 1,579 purchases on 39,475 visitors (4.000 percent), the variation with 1,745 (4.421 percent). That is z = 2.94, p-value = 0.0033, relative improvement +10.5 percent, confidence interval on the difference of +0.14 to +0.70 percentage points. A clean win, exactly the size the test was designed to find.
The guardrail. Refunds finish at 474 in control (1.201 percent) and 512 in the variation (1.297 percent). That is a +8.0 percent relative move in the wrong direction, with z = 1.22 and a p-value of 0.2233. The naive read is that the guardrail is clear.
It is not. The confidence interval on that difference runs from -0.059 to +0.251 percentage points, and the upper bound of +0.251 pp is +20.9 percent relative on a 1.2 percent baseline. The declared threshold was +0.12 pp. The interval is roughly twice as wide as the number the business said it could tolerate, so this test simply cannot say whether the redesign stayed inside the limit. To clear a 10 percent relative degradation properly, the refund rate would need about 135,624 visitors per variation, roughly 48 days of traffic instead of 14.
For contrast, here is what a guardrail alarm actually looks like at the same sample: refunds at 474 against 592 is +24.9 percent relative, z = 3.64, p-value = 0.00027, interval +0.138 to +0.460 percentage points. Entirely above the threshold, and no reading of that result ships without a conversation.
The honest write-up of the first case is three lines: primary metric up 10.5 percent and significant; refund rate directionally up 8.0 percent, interval -0.06 to +0.25 pp against a +0.12 pp threshold, not cleared at this sample; recommended action is to ship with refund rate monitored in a staged rollout for four weeks. That is a different decision from “guardrails clear, shipping”, and it is the decision the data supports.
How many guardrails, and the false alarm arithmetic
Guardrails are cheap to add and expensive to ignore, which is how teams end up with a list of fifteen and a habit of waving them all away. The arithmetic is unforgiving: every independent guardrail read at a 5 percent threshold adds its own chance of alarming by pure luck.
| Guardrails watched | Chance at least one alarms by chance alone |
|---|---|
| 1 | 5.0 percent |
| 2 | 9.8 percent |
| 3 | 14.3 percent |
| 4 | 18.5 percent |
| 5 | 22.6 percent |
| 6 | 26.5 percent |
| 8 | 33.7 percent |
| 10 | 40.1 percent |
At ten guardrails, two tests in five will show a red cell that means nothing. The team learns, correctly, that red cells are usually noise, and then misses the one that was not. Three to six standing guardrails, each with a written threshold, beats fifteen watched loosely. Add a test-specific guardrail only when the change plausibly threatens something the standing set does not cover, and treat a breach as a trigger for a confirmation run rather than as a verdict, the same discipline that applies to a borderline win on the primary metric. The correction methods for deciding on several metrics at once are covered in testing multiple variants.
A starter set of guardrail metrics for ecommerce and SaaS
Guardrails are business-specific, but the categories repeat. The set below is a starting point, not a standard, and the threshold column is the part your team has to fill in with real numbers.
| Category | Ecommerce guardrail | SaaS guardrail | Why it earns veto power |
|---|---|---|---|
| Speed | Page load time at the 75th percentile | Time to interactive on the core app view | The 2013 Bing slowdown experiment found every 100 milliseconds of improvement was worth about 0.6 percent of revenue |
| Regret | Refund and return rate | Cancellation within the first 30 days | A conversion win paid for by buyers who did not want the product is not a win |
| Friction cost | Support contacts per 1,000 orders | Support tickets per 100 new accounts | Moves cost from the funnel onto a team that does not appear in the test result |
| Reachability | Email unsubscribe rate | Notification opt-out rate | Spends a durable asset to buy a one-off conversion |
| Quality of the win | Average order value | Plan mix, share landing on the entry tier | Detects a lift built entirely on discounting or downgrading |
| Reliability | Client error rate on checkout | Crash-free session rate | The failure mode that never shows up in a conversion rate at all |
Two notes on that table. First, the speed row is not a soft guardrail. The 2013 account of experimentation at Bing describes a deliberate slowdown experiment, 10 percent of users slowed by 100 milliseconds and another 10 percent by 250 milliseconds for two weeks, and reports that every 100 milliseconds of improvement was worth roughly 0.6 percent in revenue. That result is specific to Bing at that time and should not be transplanted as a number, but it establishes that latency is a real commercial variable, not a hygiene concern.
Second, the same paper contains a warning that belongs in this article. Its authors flag a published claim from another company that a 200 millisecond delay did not matter, and argue the more likely explanation is that the experiment lacked the statistical power to detect the difference rather than that the difference did not exist. That is the underpowered guardrail failure appearing in public, in a widely read post, from a competent team. It is not an exotic mistake.
Common guardrail mistakes
| Mistake | What it produces |
|---|---|
| Reading a non-significant guardrail as cleared | Real degradations ship, and the program cannot tell a safe launch from an unexamined one |
| Choosing guardrails after seeing the results | The list becomes whatever supports the decision already taken |
| No written threshold | Every guardrail conversation restarts from scratch and lands wherever the loudest person wants |
| Fifteen guardrails on the dashboard | About two tests in five show a red cell by chance, and the team learns to ignore red cells |
| Treating any breach as an automatic block | Guardrails get quietly removed instead of being reasoned about, because small degradations are normal |
| No owner per guardrail | A breach becomes a group discussion with no decision and the launch proceeds by default |
| Guardrails only in the test, never after ship | The rare and slow harms, churn and refunds especially, are exactly the ones a two-week test cannot see |
If the recurring problem is that your guardrails are never powered enough to say anything, the structural answers are variance reduction, covered in our guide to CUPED, or accepting the guardrail as post-ship monitoring in a staged rollout rather than pretending the experiment answered it.
Automate this with Donnu
A guardrail only works if its threshold exists before the test starts and its interval is visible next to the verdict, not buried two clicks away. Donnu A/B keeps declared guardrails and their thresholds attached to the experiment, reports the confidence interval alongside every guardrail rather than a lone significance flag, and marks a guardrail as unmeasured when the sample cannot resolve the declared threshold instead of showing a reassuring green tick. When a change wins the primary metric and leaves a guardrail unanswered, that shows up as the open question it is.
Start a free 14-day trial and set thresholds for your own guardrails before the next launch.
References
- Dmitriev, P., Gupta, S., Kim, D. W. and Vaz, G. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments. KDD 2017. Source of the guardrail metric definition, the metric taxonomy used above, the MSN.com underpowered metric case with its 7.8 percent detectable threshold, and the recommendation of a priori power analysis at 80 percent power for OEC and guardrail metrics. exp-platform.com.
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the Bing slowdown experiment (100 and 250 millisecond delays over two weeks, every 100 milliseconds worth about 0.6 percent of revenue) and of the argument that a published null result on a 200 millisecond delay was most likely underpowered. exp-platform.com.
- Deng, A. and Shi, X. Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned. KDD 2016. Source of the two guardrail roles, standing in for an inapplicable goal metric and covering dimensions the goal metric cannot measure, and of the requirement that a guardrail have a clear directional interpretation. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on metrics, the OEC and organizational guardrails. Companion material at experimentguide.com.
Read also: Minimum detectable effect · A/B testing statistical significance · Sample ratio mismatch · Common A/B testing mistakes · Free statistical power calculator · Leia em português
Frequently asked questions
- What are guardrail metrics in A/B testing?
- Guardrail metrics are measures you do not want a shipped change to harm, even though improving them is not the point of the test. Dmitriev and colleagues (KDD 2017) define them as metrics that are not clearly indicative of the success of the feature being tested, but that the team does not want to significantly harm when making a ship decision. Page load time is the canonical example: no feature is launched to make the page faster, but a feature that makes it much slower should not ship. The primary metric decides whether the change is good; the guardrails decide whether it is safe.
- Is a guardrail with a non-significant p-value cleared?
- No, and this is the single most expensive misreading in the topic. A test powered for a 4 percent primary metric collects enough sample to resolve a 10 percent relative move there, but a refund rate sitting at a 1.2 percent baseline only resolves about a 19 percent relative move at that same sample. A non-significant guardrail in that test is consistent with anything from a real improvement to a 19 percent degradation. Dmitriev and colleagues (KDD 2017) document exactly this case at MSN.com, where a metric moved 0.5 percent with a non-significant p-value while the design could only detect 7.8 percent or larger. Read the confidence interval, not the verdict.
- How do I set a guardrail threshold?
- Name the largest degradation the business would accept in exchange for the win the test is chasing, write it down before launch, and then compare the upper bound of the confidence interval against that number rather than comparing the p-value against 0.05. If the interval upper bound sits below the threshold, the guardrail is genuinely cleared at this sample. If it sits above, the test simply did not answer the question, whatever the p-value says. Setting the threshold after seeing the data turns the guardrail into a formality.
- How many guardrail metrics should a test have?
- Few enough that each one is monitored seriously, which in practice means three to six for most teams. Guardrails multiply false alarms: at a 5 percent threshold, the chance that at least one of five independent guardrails crosses the line by pure chance is about 22.6 percent, and with eight it is about 33.7 percent. A long guardrail list trains the team to wave alarms away, which is worse than having no guardrails at all. Keep the standing set small and add a test-specific guardrail only when the change plausibly threatens something the standing set does not cover.
- What is the difference between a guardrail metric and a secondary metric?
- A secondary metric helps explain the primary result and is read for direction and diagnosis. A guardrail carries veto power: it can block a ship even when the primary metric wins. That difference has to be declared before launch, because a metric promoted to guardrail status after the results arrive is being used to justify a decision already made. Deng and Shi (KDD 2016) also describe a second guardrail role, standing in for the goal metric in cases where the goal metric does not apply to the team or surface being tested.
- Do guardrails need the same statistical power as the primary metric?
- Ideally yes, and in practice almost never, which is why the threshold framing matters. Dmitriev and colleagues (KDD 2017) recommend a priori power analysis for at least the OEC and the guardrail metrics, and at least 80 percent power to detect changes small enough to matter commercially. When the traffic to power a rare guardrail does not exist, the honest options are to accept it as directional monitoring with the detectable range stated openly, to run a longer dedicated safety test after ship, or to watch the guardrail in a staged rollout instead of in the experiment.
- Should a guardrail alarm always stop the launch?
- Not automatically, but it should always force an explicit decision by a named person rather than a silent override. Small degradations in guardrails are expected: Dmitriev and colleagues note that almost any added feature slightly increases page load time. The useful rule is that a guardrail inside its declared threshold ships without discussion, a guardrail past its threshold ships only with a written trade-off accepted by whoever owns that metric, and a guardrail whose interval is too wide to tell gets recorded as unmeasured rather than as passed.