Instrumentation Bias: When the Measurement Is the Bug
When a variant changes how data is collected, your A/B test measures the instrument. How to recognise, isolate and correct instrumentation bias.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
An A/B test compares two experiences, but it only ever sees what the telemetry managed to record. When the variant changes how data is collected rather than what users did, the experiment starts measuring the instrument. The output is a perfectly significant difference, with a small p-value and a tight interval, corresponding to a true effect of zero. This guide covers why instrumentation bias sails through every statistical check, a worked example where an apparent 4.26 percent improvement comes entirely from signal loss, the four failure modes that show up in practice, and the four-arm design that separates effect from measurement. It is part of our complete guide to A/B testing and pairs with common A/B testing mistakes.
Measurement is not neutral
There is a hidden assumption in every experiment dashboard: that the pipe carrying behaviour into a number is the same in both arms. Almost always it is. When it is not, nothing in the statistical apparatus tells you.
The reason is simple. A p-value measures how unlikely the observed data are under the null hypothesis, given that the data measure what you think they measure. It carries no information whatsoever about whether the measurement is valid. A telemetry defect that inflates treatment produces a small p-value with total confidence, and a tight confidence interval around a number that does not describe the world.
Worked example: plus 4.26 percent of nothing
A team tests opening a destination link in a new window. The true click rate is 12.000 percent in both arms, because the change does not affect the decision to click, only what happens after the click. There are 60,000 users per variant.
Clicks are recorded by an asynchronous request for a 1 by 1 pixel image, the industry standard mechanism and a well known lossy one: when navigation leaves the page before the request completes, the event simply never arrives. In control, navigating in place kills 6 percent of the signals. In treatment, the new window keeps the original page alive and the loss falls to 2 percent.
Paste the numbers into the calculator:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
| What the database recorded | Users | Clicks measured | Rate measured |
|---|---|---|---|
| A, control, loses 6 percent | 60,000 | 6,768 | 11.280 percent |
| B, new window, loses 2 percent | 60,000 | 7,056 | 11.760 percent |
The calculator returns plus 0.480 percentage points, plus 4.26 percent relative, z = 2.604, p-value 0.00921, with a 95 percent interval of plus 0.119 to plus 0.841 percentage points. Significant at 99 percent confidence.
And here is what actually happened:
| What actually happened | Users | Real clicks | Real rate |
|---|---|---|---|
| A, control | 60,000 | 7,200 | 12.000 percent |
| B, new window | 60,000 | 7,200 | 12.000 percent |
The calculator returns plus 0.000 percentage points, p-value 1.00000, with an interval of minus 0.368 to plus 0.368 percentage points. True effect exactly zero, measured result significant at 99 percent. No amount of extra sample would fix this: doubling the test would only tighten the interval around a 4.26 percent that does not exist.
This is not a hypothetical. Kohavi reports the real case in his paper on unexpected results in SIGKDD Explorations. The alarm was not the headline result, it was a metric that should not have moved: the percentage of users who clicked the link came out significantly higher in treatment. Since clicking the link is the experiment’s triggering point, and before that point both arms are identical, that difference was impossible by construction. The team sliced by browser, found highly significant variation in the treatment effect by browser family, and reached the explanation: opening the destination in a new window significantly improved signal reliability in non-Internet Explorer browsers. After the correction the feature was still positive, just far less positive than the initial result claimed.
The four failure modes of instrumentation bias
| Mode | What happens | Signal that gives it away |
|---|---|---|
| Unequal signal loss | One arm records better than the other | Pre-trigger metric differs across arms |
| Concentrated robot | Automated traffic always lands in one variant | Per user event counts with an absurd tail |
| Asymmetric penalty | One arm pays a redirect, latency or extra step | Page load time differs across arms |
| Broken exposure control | Different populations enter each arm | Country, browser or device mix differs |
The last two deserve detail, because they do not look like measurement problems at first glance.
Robots: when noise becomes bias
The common intuition is that robots always contaminate an experiment. The reality is more precise and more useful. Crook, Frasca, Kohavi and Longbotham put the rule this way in their KDD 2009 paper: for experimentation, the concern is removing robots that cause a bias. If traffic from a robot is distributed across the variants of an experiment in an unbiased way, then the presence of the robot adds noise to the data and reduces the power of the experiment, but does not invalidate the results. Robots seen as multiple unique users, because they reset cookies or run from multiple machines, also do not introduce bias.
The dangerous case is a robot that acts like a single user and consistently generates traffic for one variant. In the authors’ words, if a robot consistently assigned to variant A generates an excessive number of clicks, it may cause A to have a statistically significantly higher click-through rate than B even if B is preferred by human users.
The scale of this is surprising. In the same paper, the authors report an MSN portal experiment where a small change localised to a single module produced statistically significant click-through differences in several unrelated areas of the page. The cause was robots that accepted cookies and executed JavaScript, firing onclick events at rates of about 100 per minute for durations of 2.5 hours. That is 15,000 clicks from a single source. Against an arm of 60,000 users carrying roughly 7,000 legitimate clicks, it triples the metric.
The authors add a detail that complicates cleanup: when a robot runs from a machine also used by a human, both typically share the same cookie. If user identity is stored in a cookie, which is very common, the user appears schizophrenic, acting like a human at certain times and like a robot at others.
Asymmetric penalty and exposure
Redirects are the canonical example of a penalty falling on one arm only. Kohavi lists three reasons the redirected version underperforms: treatment users suffer an extra redirect, which may look fast in the lab but can cost hundreds of milliseconds in the field, and slowdowns on that scale have significant impact on metrics; different robots handle redirects differently, introducing subtle biases that make even A/A tests fail; and redirects are asymmetric, because the destination page can be bookmarked, passed to a friend or crawled, and in most implementations the treatment page never checks that the user should really have been randomized into treatment, which produces contamination.
Exposure control is the close relative. Kohavi reports two cases. The MSN US home page redirected visitors from certain countries to their local versions, and the new version under test had not yet implemented that reverse IP lookup, so the population of users from non-US addresses was much higher in treatment than in control. In another case, a misconfiguration at Bing caused all Microsoft users to always see control. In neither case does the problem surface as an error. It surfaces as a result.
The four-arm design that isolates instrumentation
When the asymmetry cannot be removed, it can be measured. Kohavi describes the design: run A, B, A prime and B prime, where A prime and B prime both pass through the same redirect or the same measurement path.
- A prime against B prime is the fair comparison of the product effect, because both paid the same penalty.
- A against A prime measures exactly what the mechanism cost in key metrics, since the content is identical and only the path differs.
The cost is traffic split four ways instead of two, which reduces power and stretches the timeline. It is worth it when the decision is expensive and the mechanism is suspect, and it is a natural extension of the discipline behind A/A tests and platform validation.
Checklist before trusting a result
- Do pre-trigger metrics match? Exposed user counts, trigger rate, and any metric preceding the point of the change. A difference here is a defect, not a finding.
- Does the traffic split match the plan? The sample ratio mismatch procedure catches a good share of these before any analysis.
- Is page load time equal across arms? If one arm is slower, part of the effect is speed rather than product.
- Does the population mix match? Slice by country, browser and device. A large composition difference points at broken exposure control.
- Does the effect concentrate in one browser or device? Product effects tend to be diffuse; instrumentation effects tend to concentrate exactly where the capture mechanism differs.
- Is there an absurd tail in per user counts? A user with hundreds of events in a short window is a robot or a monitor, not a customer.
- Does any automated system touch this flow? Kohavi reports a case where an availability monitor requested the page and simulated a click on the purchase button, and under the new design that click failed and the system retried many times, dragging down the treatment click-through rate. The tell was page views per user coming out significantly higher in treatment.
Common mistakes
- Treating an unbalanced pre-trigger metric as a curiosity. It is the cheapest evidence available that something is wrong, and it gets ignored regularly because it is not the headline metric.
- Excluding all robots as a precaution. Aggressive robot filtering also removes humans and can introduce the very bias it was meant to avoid. The target is the robot concentrated in one arm, not robot volume.
- Comparing speed only when the result disappoints. If load time enters the analysis only when a test loses, you have built a filter that corrects in one direction.
- Trusting a large win without checking instrumentation. The bigger the result, the higher the chance it is a defect. An exceptional win calls for a measurement check before a celebration, a principle that extends to guardrail metrics too.
Make this automatic with Donnu
None of this is hard to understand. It is hard to remember, every single time, with a good result already on screen and a meeting booked.
In Donnu, the checks that expose measurement problems run before the result appears rather than after somebody gets suspicious: pre-trigger metric comparison across arms, traffic ratio check, page load time per variant, and population mix by browser, device and source. When one of them fires, the report shows the warning above the headline result, because the order information appears in changes the decision that gets made. If you would rather check the numbers by hand, the p-value calculator takes the raw counts from any slice.
References
- Kohavi, R. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations, 12(2), 2010. Source of the new window link case, including the percentage of users clicking coming out significantly higher in treatment despite the click being the triggering point, the highly significant variation in treatment effect by browser family, the explanation that clicks are instrumented via an asynchronously requested 1 by 1 pixel image and that the mechanism is lossy, and the conclusion that the feature stayed positive but smaller after correction. Also the source of the three reasons redirects penalise one arm, the recommendation of a server-side mechanism and the redirected-arm design, and of the MSN and Bing exposure control cases and the monitoring system that simulated purchase clicks. kdd.org.
- Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source of the rule that robots distributed in an unbiased way add noise and reduce power without invalidating results while robots consistently assigned to one variant create significant bias, of the MSN portal experiment where robots accepting cookies and executing JavaScript fired onclick events at about 100 per minute for 2.5 hours, and of the observation that a robot and a human sharing a machine share a cookie. exp-platform.com.
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the principle that complex designs hide defects, with the eligibility check case whose bug removed users who had already seen the feature, and of the observation that page load time has significant impact on key metrics. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on experiment trustworthiness, telemetry and diagnosing suspicious results. Companion material at experimentguide.com.
Read also: Common A/B testing mistakes · A/A test validation · Sample ratio mismatch · Guardrail metrics · P-value calculator · Leia em português
Frequently asked questions
- What is instrumentation bias in A/B testing?
- It is when the variant changes not user behaviour but how that behaviour gets recorded. The result is a difference that is real in the data and absent in reality. Kohavi describes exactly this case: opening a link in a new window improved click signal reliability in non-Internet Explorer browsers, because clicks are instrumented with an asynchronous request for a 1 by 1 pixel image, a mechanism well known to be lossy. The feature stayed positive after the correction, but far less positive than the first number claimed.
- How do you tell instrumentation bias from a real effect?
- The strongest check is the trigger point. Before the point where the variants become different, both arms saw exactly the same thing, so no pre-trigger metric should differ to a statistically significant degree. When one does, that is impossible by construction and the cause is measurement, not behaviour. This is exactly how the new window case was caught: the percentage of users clicking the link came out significantly higher in treatment, which could not happen.
- Do robots always ruin a test result?
- No. Crook, Frasca, Kohavi and Longbotham draw the distinction carefully in their KDD 2009 paper: if a robot traffic is distributed across the variants in an unbiased way, it adds noise and reduces the power of the experiment but does not invalidate the results. A robot that resets cookies or runs from multiple machines appears as multiple unique users and also introduces no bias. The dangerous case is a robot that acts like a single user and consistently generates traffic for one variant, because it can make that variant statistically better without any human preferring anything.
- Why are redirects a problem in A/B testing?
- Because they penalise one arm only. Kohavi lists three reasons: treatment users suffer an extra redirect that can cost hundreds of milliseconds, different robots handle redirects differently and introduce subtle biases, and redirects are asymmetric, since the destination page can be bookmarked, shared or crawled without anyone checking that the user should have been in treatment at all. The recommendation is to prefer a server-side mechanism and, when that is impossible, to make sure both arms pay the same penalty.
- What is an A/A prime B prime test?
- It is the design that isolates instrumentation from effect. You run four arms: normal A and B, plus A prime and B prime which go through the same redirect or the same measurement path. Comparing A prime with B prime is fair because both paid the same penalty, and the gap between A and A prime measures exactly what the measurement mechanism cost in key metrics. Kohavi reports using this design when the redirect cannot be removed.