Tracking Loss in A/B Tests: The Data That Vanishes

Tracking loss does not bias an A/B test by being large, it biases the test by being differential. If both variants lose the same fraction of events, the experiment has less data, loses power and stays honest; if the new variant loses more than the old one, the number you read is wrong and no sample size fixes it. This guide covers the three layers erasing data today, with the official dates and limits, a worked example where 8 per cent of unlogged conversions erases a real 10 per cent win, and what changes when the loss hits the denominator instead of the numerator. It is part of our complete guide to A/B testing and the collection side sibling of instrumentation bias: there the instrument measures wrong, here the instrument does not measure at all.

Three layers erase data, for different reasons

Separating the layers matters because their remedies and their time horizons differ.

Layer 1, consent. Where the lawful basis requires prior consent for measurement, a visitor who declines must not be measured. That is not a technical failure to route around, it is the system working correctly. What matters to the experiment is the size of the hole and its symmetry.

Layer 2, blockers. Extensions and block lists prevent the script from loading or the event from being sent. Loss here is selective by domain, by file name and by request pattern, which makes it structurally asymmetric: a new script served from a new path is easier to block than one that has been in production for a year.

Layer 3, browser policy. This is the one that surprises experiment builders most, because it reaches first party cookies. WebKit announced on 21 February 2019, with Intelligent Tracking Prevention 2.1 in iOS 12.2 and Safari 12.1, that all persistent client side cookies, that is persistent cookies created through document.cookie, are capped to a seven day expiry. On 24 March 2020, with iOS and iPadOS 13.4 and Safari 13.1 on macOS, WebKit went further and began blocking cookies for cross site resources by default, plus deleting all of a website script writable storage (IndexedDB, LocalStorage, media keys, SessionStorage and Service Worker registrations) after seven days of Safari use without user interaction on the site. Mozilla, on 14 June 2022, began rolling out Total Cookie Protection by default to more Firefox users worldwide, across Windows, Mac, Linux and Android, creating a separate cookie jar for each website you visit.

layer	mechanism	what is lost	time horizon
Consent	lawful basis, banner	everything about that visitor	immediate
Content blocker	script or request blocked	events, sometimes the whole attribution	immediate
Script written cookie expiry	7 days in Safari since ITP 2.1	identity across visits	from day 8
Script writable storage cleanup	7 days without interaction, since March 2020	identity kept outside the cookie	from day 8
Per site isolation	cookie jar per site in Firefox	cross domain stitching	immediate

Rows three and four deserve special attention from anyone running A/B tests, because they do not erase an event, they erase identity. A visitor returning on day ten is not recognised as the same person, gets bucketed again and may land in the other variant. The practical effect is not only lost data: it is dilution of the measured effect, because part of the population assigned to treatment starts seeing control. The correct treatment of the randomisation unit is in randomisation unit.

On the third party cookie front, an important course correction: on 22 April 2025 Anthony Chavez, VP of Privacy Sandbox, wrote that Google had decided to maintain its current approach to offering users third party cookie choice in Chrome and would not be rolling out a new standalone prompt. For experimentation this changes little, because a properly built A/B test never depended on a third party cookie. What squeezes the experiment is the expiry cap on the first party cookie written by JavaScript, and that cap is still in force.

Symmetric and differential tracking loss

There is a common reasoning trap here. Someone observes that 30 per cent of events are lost, concludes the data is garbage and stops testing. If that loss is symmetric the conclusion is wrong: with 30 per cent less data you need more time, not a different decision. And there is the opposite trap, which costs more: someone observes only 5 per cent loss, calls it negligible and never checks the symmetry. Five per cent concentrated on one arm is enough to flip a result, as the example below shows.

Worked example: 8 per cent that erases a 10 per cent win

Every number came from the significance calculator embedded below, two sided. You can paste the counts and reproduce every row.

The scenario: variant B moves the buy button into a new component, and that variant conversion is logged by a new event served from a new path on your domain. That path sits on a popular block list. The control event, live for two years, does not.

The truth. A got 30,000 visitors and 1,500 orders, a rate of 5.000 per cent. B got 30,000 visitors and 1,650 orders, a rate of 5.500 per cent. The calculator returns a relative lift of 10.00 per cent, z of 2.7457 and a p value of 0.006039, with an interval of 0.143 to 0.857 percentage points. B wins.

What the dashboard shows. Eight per cent of B conversions were never logged. B appears with 1,518 orders against the same 30,000 visitors, a rate of 5.060 per cent. The calculator returns a relative lift of 1.20 per cent, z of 0.3362 and a p value of 0.736707, with an interval of minus 0.290 to 0.410 percentage points. Inconclusive.

reading	orders A	orders B logged	rate B	relative lift	z	p value	verdict
Nothing lost	1,500	1,650	5.500%	+10.00%	2.7457	0.006039	B wins
4% of B conversions lost	1,500	1,584	5.280%	+5.60%	1.5530	0.120415	inconclusive
8% of B conversions lost	1,500	1,518	5.060%	+1.20%	0.3362	0.736707	inconclusive

Notice that the dashboard never shouts. No error appears, no alarm fires, and variant B never actually “loses”: it simply stops winning. That is the most dangerous shape of a data failure, because the outcome is indistinguishable from a hypothesis that did not work. The winner’s curse covers the error in the opposite direction, overestimating the winner; here the error erases the winner.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

When the loss hits the denominator

The case above is numerator loss: the conversion happened and was not recorded. There is a case that is harder to diagnose in prose and easier to detect in practice: denominator loss, when the visitor exposure to the variant never reaches the server.

Denominator loss is easier to detect because it leaves an arithmetic mark: the observed proportion of visitors across arms stops matching the configured allocation. That is exactly what the sample ratio mismatch test measures, and the numbers below came from the SRM checker, a chi square goodness of fit test against the expected 50 per cent per side.

exposures in A	exposures in B	observed share of B	chi square	p value	passes the 0.01 guardrail?
30,000	29,700 (1% lost)	49.749%	1.51	0.2195	yes
30,000	29,400 (2% lost)	49.495%	6.06	0.0138	yes, barely
30,000	28,800 (4% lost)	48.980%	24.49	0.00000075	no
30,000	27,600 (8% lost)	47.917%	100.00	below 0.000001	no

The useful reading of that table is the second row. A 2 per cent loss in exposure logging produces a p value of 0.0138, which passes the conventional 0.01 threshold used as an SRM guardrail, and is still enough to move the observed rate. In other words: the mismatch alarm is a floor, not a ceiling. It catches the gross failure and lets through the failure that is small enough to be silent and large enough to matter.

Why the new variant almost always loses more

The asymmetry is not bad luck, it is a structural consequence of how tests get implemented. Keep this list handy as a review script before shipping the experiment.

1. New script on a new path. Freshly created domains and paths have no history and match generic block rules more easily.
2. New event with a new name. The control event survived two years of tracking fixes; the variant event was born yesterday.
3. An extra funnel step. If the variant inserts a screen, every extra screen is another chance for the visitor to leave before the event fires.
4. When the event fires. An event fired on page unload, or after a navigation, is lost more often than one fired on click.
5. Weight and load time. A heavier variant records less because some visitors abandon before the script finishes loading.
6. Changes to consent. Any variant that touches the banner changes the consent rate itself, and then loss is differential by construction.
7. A third party domain in the conversion. If the variant leads to a different payment provider, the conversion crosses one more domain boundary.

Points 1, 2 and 7 fall entirely inside the territory of first party data, and point 5 is one of the reasons why a variant with worse loading performance can look worse at converting without the visual change having anything to do with it.

What actually reduces the problem

In order of impact, for anyone choosing where to invest first:

1. Log exposure on the server. It stabilises the denominator and is the only move that attacks sample ratio mismatch at the source. The path is in server side A/B testing and the comparison between the two architectures is in client side versus server side.
2. Use a first party identifier with an expiry set by the server. A cookie set by the server does not fall under the seven day rule that hits cookies written through document.cookie.
3. Keep the variant collection identical to the control. Same script, same path, same firing moment. Only what appears on screen changes.
4. Measure the loss rate as a guardrail metric. Log the ratio of expected events to received events, per variant, next to the result. The role of those brakes is in guardrail metrics.
5. Pick a randomisation unit compatible with how long identity actually survives. If identity does not last seven days in that browser, a four week test measuring per user behaviour is not measuring what it promises.
6. Declare the consent policy in the analysis plan. Who counts and who does not is a design decision, not an analysis decision, and it belongs in the pre registered analysis plan.

Checklist before reading the result

1. Is the ratio of received to expected events measured per variant?
2. Does the observed traffic split match the configured one, and did you look at the p value rather than only the percentage gap?
3. Does the variant use the same script, the same path and the same firing moment as the control?
4. Does the variant touch the consent banner in any way?
5. Is the test duration compatible with how long identity survives in your audience dominant browser?
6. Is the consent decline rate similar across both arms?
7. Is there any server side metric, such as a paid order, to compare against the browser side metric?

Item 7 is the most underrated. Comparing your database order count against your tracker conversion count, per variant, is the cheapest audit available and resolves most cases in one query.

Common mistakes

• Concluding the data is useless because loss is high. Symmetric loss costs power, not the conclusion.
• Calling 5 per cent loss negligible without checking symmetry. Five per cent concentrated on one arm flips winners.
• Testing changes to the consent banner and reading the result from browser events. Loss there is differential by construction.
• Trusting the mismatch guardrail to catch everything. It passed comfortably on the 2 per cent loss in the table above.
• Treating Safari policy as a third party cookie topic. The seven day rule hits the first party cookie written by script.
• Assuming Chrome keeping third party cookies settled the question. For experimentation it was never about that.
• Comparing totals only and never the expected to received ratio per variant. The total hides precisely the asymmetry that matters.

Make this automatic in Donnu

The rare hole is technical. The common hole is organisational: the data loss is known to whoever owns tracking, the test result is read by whoever owns the product, and the two facts never appear on the same screen.

In Donnu, the observed split of visitors per variant sits next to the result from day one, with the goodness of fit p value, because an allocation deviation is the symptom shared by denominator loss, bucketing errors and bot traffic. Bucketing and exposure logging can run on the server, with no dependency on a script written cookie, and the same significance calculator accepts counts straight from your database so you can compare the browser reading against the server reading before deciding anything.

References

• WebKit. Intelligent Tracking Prevention 2.1. 21 February 2019. Source of the seven day cap: with ITP 2.1, in iOS 12.2 and Safari 12.1, all persistent client side cookies, that is persistent cookies created through document.cookie, are capped to a seven day expiry, a restriction that does not reach session cookies or authentication cookies marked Secure and HttpOnly. webkit.org.
• WebKit. Full Third-Party Cookie Blocking and More. 24 March 2020. Source of the default blocking of cookies for cross site resources in iOS and iPadOS 13.4 and Safari 13.1 on macOS, and of the extension of the seven day cleanup to all script writable storage, including IndexedDB, LocalStorage, media keys, SessionStorage and Service Worker registrations, after seven days of Safari use without user interaction on the site. webkit.org.
• Chavez, A. Next steps for Privacy Sandbox and tracking protections in Chrome. Privacy Sandbox blog, 22 April 2025. Source of the decision to maintain the current approach to offering users third party cookie choice in Chrome and not to roll out a new standalone prompt for third party cookies. privacysandbox.google.com.
• Mozilla. Firefox rolls out Total Cookie Protection by default to more users worldwide. 14 June 2022. Source of the isolation mechanism: Total Cookie Protection works by creating a separate cookie jar for each website you visit, and is being rolled out by default to more Firefox users worldwide, across Windows, Mac, Linux and Android. blog.mozilla.org.
• Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source of the collection audit guidance: validate the collection of user behaviour data, the assignment of users to variants and the calculation of metrics by comparing data received by the experimentation system against the existing system of record, record by record where possible, and of the authors observation that such audits found serious problems even in established systems of record. exp-platform.com.

Read next: Instrumentation bias · Sample ratio mismatch · Randomisation unit · Client side versus server side testing · First party data · SRM checker · Leia em português