First-Party Data and A/B Testing in a Post-Cookie World
First-party data A/B testing: what browser storage limits and consent really cost you, how identity loss inflates sample size, and what to fix first.
📚 This article is part of the guide GA4 and A/B Testing: The Complete Integration Guide.
A/B testing never depended on third-party cookies, so the privacy shift did not break the method; it broke the bookkeeping around it. The split happens on your own domain, with your own visitors, measured against your own conversions, which is first-party by construction. What degraded is the durability of the identity that carries the assignment from one visit to the next, and the completeness of the analytics that describe the visitors who were assigned. This article is a child of the complete guide to GA4 and A/B testing and answers one practical question: what a testing program has to change when a meaningful share of its visitors cannot be measured, or cannot be recognised on their second visit.
The short version: your required sample size does not move, your weekly denominator does, and almost every practical problem downstream comes from teams not redoing that second piece of arithmetic.
What changed, precisely
Three different things are usually bundled together under “the cookie apocalypse”, and they have very different consequences for an experiment.
| Change | What it affects | Effect on an A/B test |
|---|---|---|
| Third-party cookies blocked by default in Safari and Firefox | Cross-site tracking and ad measurement | Almost none: the split and the conversion are same site |
| Client-side first-party storage capped in lifetime | How long an assignment survives | Real: returning visitors get reassigned, diluting the measured effect |
| Consent required before measurement tags fire | Who appears in analytics at all | Real: the measured population becomes a self-selected subset |
The first row is the one that gets the headlines and the one that matters least here. Google announced in April 2025 that Chrome would not ship a standalone prompt for third-party cookies and would keep the user’s existing choice in place, while Safari and Firefox have blocked them by default for years. Either way, none of that touches a same-site experiment.
The second row is where the damage lives. Apple’s Intelligent Tracking Prevention caps the lifetime of cookies written by JavaScript on the client, documented by the WebKit team since ITP 2.1 in 2019, with a further cap on script-writable storage after cross-site navigations with link decoration. A test whose assignment is stored that way loses returning visitors on a schedule set by the browser, not by you.
The third row is a design decision you control, and the one most often applied inconsistently.
Identity loss dilutes the effect, it does not bias it
This distinction is worth being precise about, because the two failure modes call for different responses.
When a returning visitor is re-randomized, roughly half the time they land back on the same side and nothing happens, and roughly half the time they cross over. Crossovers do not systematically favour either variation, so they do not create a fake winner; they mix the two populations and pull the observed difference toward zero. A real 10% improvement measures as something smaller, and a test sized for 10% is now underpowered for what it can actually detect.
Bias, by contrast, comes from anything that treats the two branches differently: a consent flow that gates one side and not the other, a tag that fires before the render on one branch, a script blocker that hits the variation’s extra request and not the control’s. Those produce wrong answers rather than weak ones.
The arithmetic: same sample, longer calendar
Here is the part teams get wrong. The required sample per variation is a property of the statistics, not of your tracking, so it does not move when visibility drops. What moves is how fast you accumulate it.
Take a 4% baseline and a target of detecting a 10% relative improvement (4.0% to 4.4%), at 95% confidence and 80% power. The math returns 39,475 visitors per variation. Now vary only the measurable weekly traffic:
| Measurable visitors per week | What that represents | Days to fill both variations |
|---|---|---|
| 25,000 | Everyone measurable | about 23 days |
| 17,500 | 30% of visitors not measurable | about 32 days |
| 12,500 | Half not measurable | about 45 days |
Run your own baseline and traffic through the calculator before committing to a date:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
The dangerous version of this mistake is not the longer calendar, it is keeping the original one. A team that planned 23 days, lost 30% of measurable traffic and stopped on day 23 anyway ends up with roughly 28,750 per variation instead of 39,475. At that sample, the power to detect the same 10% relative effect falls to about 67%, meaning that if the improvement is real, the test misses it one time in three. The result is not a negative finding, it is no finding at all, reported as if it were one.
Where the assignment should live
Four places, in increasing order of durability and cost.
- JavaScript-written cookie or local storage. Simplest and most common, and the one most affected by client-side storage lifetime caps. Fine for short experiments on high-frequency traffic, weak for anything that depends on recognising a visitor weeks later.
- First-party cookie set by your own server, with the HttpOnly flag. Not written by a script, so it is not subject to script-storage caps, and not readable or removable by page-level blockers. This is the highest-value single change most teams can make.
- Assignment tied to a logged-in account. The most stable identity there is, and the only one that survives a device change. It only covers the logged-in part of the funnel, so tests that start before login still need one of the above.
- Assignment computed at the edge or on the server for each request. Removes the storage question almost entirely, at the cost of engineering effort and a caching strategy, since a cached page cannot vary per visitor without a cache key that includes the variation. The client-side versus server-side comparison covers that trade in full.
A caution about the second option, because it is often oversold: an HttpOnly first-party cookie makes the assignment durable, it does not make it consented. Whether you may set it before consent depends on how your jurisdiction and your own policy classify a cookie that exists to run an experiment rather than to deliver the service. Decide that with your legal owner, document the decision, and apply it identically to both branches.
Consent, applied symmetrically or not at all
The one rule that keeps a consented experiment honest is symmetry. Whatever the banner does, it must do to both branches at exactly the same point in the flow. If control renders immediately while the variation waits for a consent callback, you are no longer comparing two page designs, you are comparing a page against a page plus a delay, and the delay is usually worth more conversion points than the design change you were testing.
Two consequences follow that are worth writing into the test plan rather than discovering afterwards:
- State the population. If measurement starts after acceptance, the finding applies to consenting visitors. That is a legitimate population and an honest sentence in the report. Presenting it as “our visitors” is where it becomes misleading.
- Do not compare across a consent change. A test that ran before a banner redesign and one that ran after are measuring different populations. Treat the banner change as the boundary of a measurement era, the same way you would treat a tracking migration.
What first-party data actually buys you
The framing so far has been defensive, which undersells the opportunity. A first-party identity you control, especially a logged-in one, enables readings that third-party tracking never supported well.
- Longer outcome windows. A durable identity lets you attribute a conversion that happens three weeks after the exposure, which matters for any product with a considered purchase. Short-lived storage forces you to measure the fast, shallow outcome instead of the one you care about.
- Segmentation that survives. New versus returning, plan tier, lifecycle stage: all of these are yours, they do not depend on a third party, and they let you check whether an aggregate result holds inside the segments that matter, which is the practical defence against Simpson’s paradox.
- Guardrail metrics further down the funnel. With identity that persists, you can watch whether a lift in signups shows up as a lift in activation or gets absorbed by worse retention. Without it, you optimize the first step and hope.
A worked example, reading a test with reduced visibility
A pricing page converts at 4.00% among measurable visitors. The team runs the test to 20,000 visitors per variation and closes with 800 conversions in control and 880 in the variation (4.40%). Running those four numbers through the same two-proportion math used everywhere on this blog: z = 1.99, p-value about 0.046, an observed relative lift of +10.0%, and a 95% confidence interval on the difference from +0.007 to +0.793 percentage points.
Significant, and barely. The interval’s lower bound is seven thousandths of a percentage point above zero, which is another way of saying that a small amount of extra noise would have flipped the verdict. That is exactly the position a program with degraded identity keeps finding itself in: dilution shrinks the observed gap, the gap lands near the threshold, and the decision rests on the last few hundred visitors. Paste your own numbers below to see where your test sits:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
The right response is not to declare the win louder. It is to note that the estimate is compatible with anything from a negligible gain to a solid one, and to decide whether the change is cheap enough to ship on a weak positive or important enough to deserve a confirmation run with better identity persistence.
What not to do
- Fingerprinting to replace a cookie. A compliance risk and an unstable identity at the same time, since fingerprints drift with browser and device changes, which can move the same person between variations mid experiment.
- Stitching identities across sites you do not own. Whatever the vendor calls it, this is the thing the privacy shift exists to stop, and it is not needed for a same-site experiment.
- Blaming privacy for a flat result. Dilution weakens effects; it does not erase real ones. Before concluding that measurement is at fault, check the boring explanations first: was the change actually visible, was the sample actually reached, was the split actually even.
- Rebuilding a cross-device graph to run a button test. The identity you need should match the size of the question. Most tests do not need to recognise the same human on three devices.
Make this automatic with Donnu
The failure mode described in this article is rarely a missing feature; it is an arithmetic step nobody redid. Traffic visibility dropped, the sample requirement stayed the same, the stop date did not move, and a test that never had the power to answer the question was reported as an answer.
Donnu handles the web and client-side half of this: a first-party snippet served from your own domain, sample sizing computed from your real baseline rather than a default, and honest statistics that show the confidence interval next to the verdict instead of a single green number. It does not build you a cross-device identity graph and does not pretend to; what it does is refuse to call a diluted, underpowered result a winner. Start a free trial and see the interval, not just the arrow.
Read also: GA4 and A/B Testing: The Complete Integration Guide · Client-Side vs Server-Side A/B Testing · CRO for Low-Traffic Sites · Common A/B Testing Mistakes
References
- The Privacy Sandbox. A new path for Privacy Sandbox on the web. Announcement, April 2025, that Chrome would keep the user’s existing third-party cookie choice rather than ship a standalone prompt. privacysandbox.com/news/privacy-sandbox-next-steps.
- WebKit. Intelligent Tracking Prevention 2.1. Cap on the lifetime of client-side, script-written cookies. webkit.org/blog/8613/intelligent-tracking-prevention-2-1.
- WebKit. Intelligent Tracking Prevention 2.3. Further limits on script-writable storage after cross-site navigation with link decoration. webkit.org/blog/9521/intelligent-tracking-prevention-2-3.
- Mozilla. Enhanced Tracking Protection. Third-party tracking cookies blocked by default in Firefox. support.mozilla.org/kb/enhanced-tracking-protection-firefox-desktop.
- Google Analytics Help. Consent mode. How tag behaviour changes before and after consent. support.google.com/analytics/answer/9976101.
- MDN Web Docs. Set-Cookie: the HttpOnly attribute. Why a server-set cookie is not reachable from page scripts. developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on dilution and on triggering. Supporting material at experimentguide.com.
Frequently asked questions
- Did the end of third-party cookies break A/B testing?
- No, and the confusion is worth clearing up. A/B testing has always been a first-party activity: you split your own visitors on your own domain and measure your own conversions, so no third-party cookie was ever required for the split itself. What the privacy shift changed is the reliability of the identity that carries the assignment, and the completeness of the analytics that report on it. The experiment still works; the bookkeeping around it gets noisier.
- What actually breaks when browser storage is restricted?
- The persistence of the assignment, not the randomization. When a browser expires or clears the storage key that holds "this visitor is in variation B", the returning visitor is treated as new and gets randomized again, which can put the same person on both sides of the test across visits. That dilutes the measured difference toward zero and, in the worst case, breaks the split ratio. The statistics remain valid; what degrades is the assumption that one visitor equals one consistent exposure.
- Does a consent banner make my A/B test invalid?
- It makes it partial, which is different from invalid. If measurement only starts after consent, you are running the experiment on the subset of visitors who accept, and that subset is not a random sample of your traffic. The result is still valid for that population, and you should say so when you report it. What is genuinely invalid is applying consent asymmetrically, for example letting the variation branch run before the banner while the control branch waits, because then the banner itself becomes part of what you are measuring.
- Should I move my A/B tests server-side because of privacy changes?
- Only if the constraint you are hitting is identity persistence, and only with clear eyes about the cost. A server-side assignment written to an HttpOnly first-party cookie survives client-side storage restrictions better than a key written by JavaScript, and it cannot be removed by a script blocker. In exchange, you give up the fast iteration of a snippet, you need engineering for every change, and you now own the caching problem, since a cached page cannot vary per visitor without extra work at the edge.
- How much extra sample does identity loss cost me?
- Not extra sample, extra time, which is the trap. The required sample per variation does not change: with a 4% baseline and a target of detecting a 10% relative improvement at 95% confidence and 80% power, it is 39,475 per variation whether or not you can measure everybody. What changes is the weekly denominator that fills it. At 25,000 measurable visitors a week the test takes about 23 days; if 30% of them stop being measurable, the same test takes about 32 days; at half, about 45 days. Teams that do not redo this arithmetic end up stopping on the original date with a fraction of the sample they planned: about three quarters of it if 30% of traffic became unmeasurable, and about half of it if half did.
- Is fingerprinting a legitimate replacement for cookies in testing?
- No, on two independent grounds. Legally, identifying a device without consent is treated as personal data processing under the GDPR and under Brazilian law, and doing it specifically to evade a storage restriction is hard to defend as legitimate interest. Technically, it is unstable in exactly the way that matters here, because a fingerprint drifts with browser updates, screen changes and network conditions, so the same person can drift across variations mid experiment. A fragile identity that is also a compliance risk is a bad trade for an experiment.