Bot Traffic in A/B Tests: Clean Before You Read

Bot traffic does not invalidate an A/B test by existing. It invalidates the test when it lands unevenly across variants. A robot distributed in an unbiased way only adds noise and costs you power; a robot that behaves like a single user and always feeds the same variant shifts that arm and can flip the winner. This guide covers the difference between noise and bias, why executing JavaScript does not separate humans from robots, a worked example where a crawler erased a real 13 per cent win, and the detection method the original authors of the problem proposed. It is part of our complete guide to A/B testing and the direct neighbour of instrumentation bias: there the instrument measures wrong, here the thing being measured is not a person.

Noise and bias are different problems

The common intuition is that bots are dirt and dirt ruins everything. The experimentation literature is more precise than that.

Crook, Frasca, Kohavi and Longbotham, in the KDD 2009 paper that catalogued seven pitfalls of controlled experiments on the web, put it this way: for experimentation the primary concern is removing robots that cause bias. If a robot traffic is distributed across the variants in an unbiased way, its presence adds noise to the data and reduces the power of the experiment, but it does not invalidate the results. Robots seen as multiple unique users, because they reset cookies or run from multiple machines, do not introduce bias. The case that does introduce bias is the robot acting like a single user and consistently generating traffic for a single variant.

This reorders the priorities. A site can carry a huge fraction of automated requests and still run trustworthy experiments, as long as randomisation throws those requests into both arms in the same proportion. And a site with almost no bot traffic can produce a completely wrong result because of one agent that keeps hitting a single URL.

robot mechanism	identity	how it splits	effect on the test
Crawler resetting cookies on every visit	many, ephemeral	tends to be symmetric	noise, lost power
Robot running from several machines	many	tends to be symmetric	noise, lost power
Robot with a stable cookie and extreme behaviour	one, persistent	stuck in one variant	bias, flipped winner
Crawler that only follows one URL pattern	any	one arm only in a split URL test	bias, flipped winner
Uptime monitor hitting a fixed route	one, persistent	one arm only	bias, flipped winner
Robot sharing a machine with a person	mixed	follows that person’s arm	contaminates that arm’s metric

The last row deserves its own paragraph. The same authors note that robots implemented by automating browsers such as Internet Explorer or Firefox support all the functionality of those browsers, including cookies and JavaScript, and that when such a robot runs from a machine also used by a human, both typically share the same cookies. If the user identity is stored in a cookie, which is very common, that identifier behaves like a person at certain times and like a robot at others. No identifier level filter resolves that case cleanly.

The MSN portal case

The most useful account in the literature on this topic is short and specific. In an experiment on the MSN portal, a small change was made to a single module of the page. The results came back with click through rates statistically significantly different in several areas of the page, including areas with no relationship to the change at all.

The investigation found the cause: robots that accept cookies and execute JavaScript. According to the authors, executing code in JavaScript is one of the most common characteristics used to separate humans from robots, and some web analytics vendors even claim that page tagging using JavaScript is so robust that no additional robot detection should be done. In that case, though, the robots were executing JavaScript onclick events, which fire on the MSN portal when users click a link, at extremely high rates of about 100 per minute for durations of 2.5 hours.

The detail that matters to anyone reading an A/B scorecard is not the number itself, it is the shape of the anomaly: a significant difference where nothing changed. That is the cheapest available signal that something non human is in the data, and it reads for free on any scorecard that shows secondary metrics next to the primary one.

Worked example: the crawler that erased a real win

Every number below came from the same significance calculator embedded further down, two sided, and you can reproduce each row by pasting the counts.

The scenario is a split URL test on a product page. Variant B lives on its own URL. A price comparison crawler discovers that URL and starts visiting it, across many sessions, never buying anything.

Reading A, the real human data. A got 40,000 visitors and 1,200 orders, a rate of 3.000 per cent. B got 40,000 visitors and 1,360 orders, 3.400 per cent. The calculator returns a relative lift of 13.33 per cent, z of 3.2141 and a p value of 0.001309, with a confidence interval of 0.156 to 0.644 percentage points on the absolute difference. B wins, comfortably.

Reading B, the data as it reaches the dashboard. The crawler added 6,000 visits to variant B, none of them with an order. B now shows 46,000 visitors against the same 1,360 orders, a rate of 2.957 per cent. The calculator returns a relative lift of minus 1.45 per cent, z of minus 0.3742 and a p value of 0.708243, with an interval of minus 0.271 to 0.184 percentage points. The result is not merely non significant: the sign flipped, and the surface reading is “the new variant lost”.

reading	visitors A	orders A	visitors B	orders B	rate B	relative lift	p value	verdict
Humans only	40,000	1,200	40,000	1,360	3.400%	+13.33%	0.001309	B wins
With 3,000 robot visits on B	40,000	1,200	43,000	1,360	3.163%	+5.43%	0.175281	inconclusive
With 6,000 robot visits on B	40,000	1,200	46,000	1,360	2.957%	minus 1.45%	0.708243	B loses

Look at the middle row. At half the contamination the experiment does not become a loss, it becomes a tie. A tie is the most expensive of the three outcomes, because nobody investigates a tie: the variant is quietly discarded and the 13 per cent gain is never shipped.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Note that the direction of the damage depends on which side of the fraction the robot touches. In the example above the robot inflated the denominator and sank the rate. A robot firing click events, as in the MSN case, inflates the numerator and lifts the rate of the variant it lives in, producing a winner that does not exist. Both failures share an origin and a remedy.

Why the known bot list is not enough

The most common layer of defence is the list of known agents. Google Analytics documentation states that traffic from known bots and spiders is automatically excluded, using a combination of Google research and the International Spiders and Bots List maintained by the Interactive Advertising Bureau. The same page records that, at this time, you cannot disable that exclusion and you cannot see how much known bot traffic was excluded.

That is useful and it is insufficient, for two structural reasons.

The first is the entry criterion for the list: it catalogues agents that declare themselves. A search crawler, a link checker, a social preview fetcher and an uptime monitor identify themselves because being identifiable is part of the job. An agent that does not want to be recognised simply does not show up under a catalogable name.

The second is the lack of visibility. When exclusion happens before the data reaches your report and the excluded volume is not exposed, you cannot answer the only question that matters for experimentation: did what was removed land evenly on both arms? A symmetric exclusion is harmless. An exclusion that removed more from one side than the other is itself a source of bias.

layer	what it catches	what it misses
Known agent lists (IAB and similar)	search crawlers, SEO tools, link previewers, monitors	any agent that does not declare itself
Requiring JavaScript execution	simple scripts, plain HTTP clients	automated browsers (the MSN case)
Events per session threshold	obviously extreme behaviour	a robot tuned to look human
A/A tests and post hoc re randomisation	allocation bias, whatever its origin	it does not name the robot, only proves asymmetry

That last row closes the problem, and it is the original proposal of the 2009 authors.

Detection through A/A tests and post hoc re randomisation

The authors acknowledge that identifying all robots is difficult in general and that there is no clear way to evaluate how well a robot detection algorithm performs on real data. Their way out is elegant: use the controlled experiment itself as the evaluation function, at least for the robots most critical to analysis, the ones able to skew results by accepting cookies and behaving like extreme users.

The scheme is the A/A test: users are split into control and treatment, but there is no systematic difference between the two versions they are exposed to. The null hypothesis in an A/A test should be rejected about 5 per cent of the time when a 95 per cent confidence level is used. If that does not hold true, there is bias introduced by extreme behaviour of users, most likely robots being assigned to a particular variant. The authors stress that multiple A/A tests must be run to have confidence about whether biased robots exist in the data.

The most valuable operational detail comes next: these tests do not have to be live. According to the authors it is sufficient to run tests post hoc, by re randomising users and assigning them to control and treatment and evaluating the hypothesis that they are the same. In practice that means you can audit robot bias on the historical data you already hold, today, without spending a week of traffic.

Practical signatures in your own data

Before investing in sophisticated detection, there are cheap queries that resolve most cases. They all rest on the same principle: the robot that biases is the robot that behaves like an extreme user.

1. Events per identifier tail. Sort identifiers by event count over the period. If the top of the list sits orders of magnitude above the median, inspect those identifiers one by one. The statistical treatment of the long tail is in outliers and capping, and note that capping treats the symptom without answering whether the extreme was a person.
2. Rhythm within the session. People produce irregular gaps between events. Near constant cadence, or dozens of events per minute sustained for hours, is a mechanical signature.
3. Split of the extreme identifiers across variants. This is the query that decides whether you have noise or bias: take the top 1 per cent of identifiers by volume and check how they distribute across arms. Even split is noise. Uneven split is your problem.
4. Overall traffic split. A deviation in the proportion of visitors across variants is the cheapest alarm in the arsenal, and it fires for robots, collection failures and bucketing bugs alike. The full procedure is in sample ratio mismatch and can be run in the SRM checker.
5. Metrics that should not have moved. This is the MSN signature. If the experiment touched one module and three unrelated areas moved, the robot hypothesis goes ahead of the effect hypothesis.
6. Symmetry of the filter itself. After applying any exclusion rule, compare how many records it removed from each arm. An asymmetric filter is a new source of bias.

Where to filter bot traffic: collection, ingestion or analysis

stage	advantage	risk
At collection (the event is never sent)	clean data from the source	what was dropped cannot be audited later
At ingestion (flagged and separated)	lets you compare raw against clean	requires storing the flagged volume
At analysis (excluded in the query)	reviewable, reproducible rule	the rule can be changed after seeing the outcome

The most defensible setup is usually the middle one: flag at ingestion, keep everything, exclude at analysis through a rule written into the pre registered analysis plan. That way you can report both readings, raw and clean, and show that the conclusion does not depend on the filter. When the two readings disagree, that is information, not embarrassment: it is exactly the finding that justifies investigating before deciding.

Checklist before trusting the number

1. Is the exclusion rule written down before the test starts? A filter chosen afterwards is a degree of freedom.
2. Can you see the filtered volume per variant? If not, you cannot claim the filter was symmetric.
3. Does the traffic split pass on the already cleaned data? Checking before filtering answers the wrong question.
4. Are the top identifiers by volume split evenly across arms?
5. Did any metric unrelated to the change move?
6. Is there a battery of A/A tests, live or post hoc, over this same traffic segment?
7. Does the conclusion survive both readings, raw and clean?

Common mistakes

• Treating bot volume as the problem. What breaks the experiment is asymmetry, not volume.
• Trusting that JavaScript execution separates humans from robots. The MSN case exists precisely to disprove that.
• Filtering after seeing who won. If the filter flips the winner and was chosen afterwards, the result is not defensible, even when the filter is technically correct.
• Excluding without counting what was excluded. An asymmetric filter swaps one bias for another.
• Running a single A/A test and concluding all is well. One A/A test rejects the null 5 per cent of the time by construction. The signal is in the distribution of a battery.
• Confusing robots with fraud. A large share of automated traffic is legitimate and declared. The goal here is to measure people, not to punish anyone.
• Applying capping and calling it done. Capping the tail reduces variance and does not answer whether the extreme was human.

Make this automatic in Donnu

The practical problem is rarely a lack of willingness to filter. It is that the bot decision happens far from the dashboard: somebody wrote a rule somewhere, months ago, and nobody can say whether it removed more from one arm than the other in this specific experiment.

In Donnu, the observed proportion of visitors per variant sits next to the result from day one of the test, not buried in a separate health report, because allocation asymmetry is the symptom shared by robots, collection failures and bucketing bugs. If you want to redo any calculation by hand, the significance calculator takes both raw and filtered counts so you can compare the two readings side by side, and the SRM checker tests the traffic split before you look at the winner.

References

• Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source of pitfall 5 (neglecting to filter robots); of the distinction between a robot distributed in an unbiased way, which adds noise and reduces power without invalidating results, and a robot acting as a single user feeding one variant, which creates significant bias; of the observation that robots seen as multiple unique users because they reset cookies or run from multiple machines do not introduce bias; of the MSN portal case, with statistically significant differences in unrelated areas of the page caused by robots that accept cookies and execute JavaScript, firing onclick events at about 100 per minute for 2.5 hours; of the note that some web analytics vendors claim JavaScript page tagging removes the need for additional robot detection; of the case of a robot sharing cookies with a human on the same machine; and of the A/A test evaluation scheme, including the post hoc variant based on re randomising users. exp-platform.com.
• Google. [GA4] Bot traffic. Google Analytics Help. Source of the statement that traffic from known bots and spiders is automatically excluded, that identification uses a combination of Google research and the International Spiders and Bots List maintained by the Interactive Advertising Bureau, and that you cannot disable this exclusion or see how much known bot traffic was excluded. support.google.com.
• Interactive Advertising Bureau. IAB/ABC International Spiders and Bots List. Source for the existence and scope of the industry reference list of declared automated agents used to exclude non human traffic. iab.com.
• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the scale context in which these checks run, with each experiment typically exposing several million users and over 200 experiments running concurrently, and of the description of the alerting system that automatically detects data quality events. exp-platform.com.

Read next: Instrumentation bias · Sample ratio mismatch · A/A testing: validate the setup · Outliers and metric capping · Pre registered analysis plan · Significance calculator · Leia em português