Geo Experiments: Testing Without User Level Data

When an intervention cannot be delivered per user, the way out is to randomize whole regions: half get the campaign, half do not, and the gap between the two sets is the incremental effect. The price is that the sampling unit becomes the region rather than the person, and ignoring that trade produces p-values that are off by a large factor. This guide covers when a geo experiment is the only honest option, a worked example where a result with seven leading zeros in its p-value turns non significant once read on the right unit, the linear model Google published for estimating return on ad spend, and the design rule that cuts the confidence interval by 10 percent for free. It is part of our complete guide to A/B testing and it uses the same randomization unit reasoning as switchback experiments.

When you cannot randomize people

User level A/B testing depends on two things: being able to deliver different experiences to different people, and being able to tie each outcome back to the person who produced it. There is a whole family of marketing decisions where neither is possible.

• Media that reaches whoever is nearby. Television, radio, out of home, local sponsorships. Exposure is not chosen by you, it is geographic by nature.
• Budget and campaign structure changes. Raising bids, adding keywords, turning a campaign off. The side effects cross channels and do not respect a cookie split.
• Outcomes that happen off site. In store sales, calls to the contact center, dealership visits. No identifier links exposure to conversion.
• Regional pricing or operations. Shipping, delivery windows, inventory availability.

In all of these, the unit you can actually control is the region. Vaver and Koehler describe exactly this design in Google’s paper on measuring advertising effectiveness: non-overlapping geographic regions are randomly assigned to a control or treatment condition, and each region realizes its assigned condition through geo-targeted advertising. They note that it is not generally feasible to work with geos as small as postal codes, and that in the United States one possible set is the 210 Designated Market Areas defined by Nielsen, broadly used as a geo-targeting unit by advertising platforms.

Worked example: the p-value that evaporates

A retail chain runs a geo experiment across 210 regions, drawing 105 into treatment and 105 into control. Each region averages 40,000 reached customers, giving 4.2 million per arm. At the end of the period, the aggregate numbers look like this:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Naive reading, per customer	Customers	Conversions	Rate
Control, 105 regions	4,200,000	84,000	2.000 percent
Treatment, 105 regions	4,200,000	86,520	2.060 percent

The calculator returns plus 0.060 percentage points, plus 3.000 percent relative, z = 6.165, a p-value below 0.00001, and a 95 percent interval of plus 0.041 to plus 0.079 percentage points. It looks like a closed case.

It is not, and the reason is that the draw was not made among customers. It was made among 210 regions. Customers in the same region share weather, local holidays, a local competitor, the neighboring retailer campaign and regional seasonality, so observations within a region are not independent. The factor that corrects this is the design effect, whose form has been known since Kish: it equals 1 plus the intraclass correlation coefficient times the cluster size minus 1.

Assume a modest intraclass correlation of 0.0005, an illustrative and deliberately small value, and clusters of 40,000 customers:

Calculation	Result
Design effect: 1 plus 0.0005 times (40,000 minus 1)	21.00
Effective sample per arm: 4,200,000 divided by 21.00	200,005 customers
Standard error multiplied by the square root of 21.00	factor 4.583
Corrected z: 6.165 divided by 4.583	1.345
Corrected p-value	0.178

The result that looked certain with seven leading zeros is, read correctly, not significant. Nothing in the data changed. What changed was honoring the unit that was actually randomized.

Notice the size of the lever: an intraclass correlation of 0.0005 looks negligible, and yet, multiplied by clusters of 40,000 people, it erases 95 percent of the effective sample. That is why the number of regions, not the number of people, is what sets the power of a geo experiment.

The right calculation: the ROAS linear model

The two proportion test we just corrected by hand is not the instrument Google uses in this design. Vaver and Koehler analyze the result with a linear model at the region level, and its shape explains why it works.

The response of a region during the test period is modeled from three things: an intercept, the response of that same region during the pretest period, and the ad spend differential for that region. The parameter of interest is the coefficient on the spend differential, which is precisely the return on ad spend, that is, the incremental impact the spend had on the response metric.

Model term	Role
Intercept and pretest response coefficient	absorb seasonal differences between pretest and test, and the size of each region
Spend differential coefficient	the parameter of interest, return on ad spend
Weights equal to 1 over the pretest response	control the heteroscedasticity caused by differences in geo size

Three practical consequences follow:

1. The pretest does the heavy lifting. It enters as an explanatory variable, and that is what lets you compare a large region against a small one without the size difference dominating the result.
2. The spend differential is not the test period spend. When spend already existed during the pretest, the differential is estimated by a second model fitted using only the control geos, which serves as the counterfactual for what spend would have been without the experiment. For control geos the differential is zero by definition, since they continue operating at baseline.
3. The response metric is your choice. The authors list paid and organic clicks, online or offline sales, website visits, newsletter sign-ups and software downloads.

It is worth recording why this design exists in the paid media world: cost per click does not tell the whole story, because part of a paid click may be cannibalizing an organic click you would have gotten for free. The metric a geo experiment delivers, and direct observation does not, is cost per incremental click.

Design: stratify by size before randomizing

A simple draw across 210 regions can, by bad luck, put the largest ones all on one side. Since regions are wildly unequal in size, that costs precision. The correction Vaver and Koehler recommend is cheap and specific:

1. Rank the geos by the response metric observed in the pretest period.
2. Partition that ranked list into groups of size M, where the test fraction is 1 over M. For half in treatment, M equals 2, that is, pairs.
3. Randomly select one geo from each group for treatment.

They report that grouping geos by size prior to assignment reduces the confidence interval of the return on ad spend measurement by 10 percent or more, and that confidence interval estimates come out roughly 10 percent lower when the random assignment is constrained this way. That is statistical power for free, obtained before the experiment starts.

Duration, delay and the limit of prediction

Two calendar traps show up in this design.

The effect keeps happening after the spend stops. Vaver and Koehler are explicit: offline sales is an example of a response metric likely to carry a positive delay, because consumers need time to research, decide and visit a store. That is why the test period has to extend past the end of the spend change by the size of that delay, to fully capture the incremental sales. If you cut the period on the day you switch the campaign off, you underestimate the effect systematically.

Predicting the confidence interval works, up to a point. They describe a design procedure where you carve pseudo pretest and test periods out of historical data, matching the lengths of the planned periods, and estimate the half width of the confidence interval across many random assignments. The prediction is quite accurate. But they flag the limit: once the hypothesized period grows longer than the available history, pretest data has to be used multiple times to generate each estimate, and that reuse makes the estimates overly optimistic. Take it as a general warning: power simulations built on recycled data promise more than they deliver.

Geo experiment checklist

1. Is the intervention genuinely impossible per user? If you can randomize a person, randomize a person. A geo experiment trades a lot of power for validity.
2. Do the regions actually not overlap? Two regions sharing an audience through media spillover stop being independent units, and the problem becomes interference between variants.
3. Is there a clean pretest period? With no campaign structure difference across regions during it.
4. Was the draw stratified by size? Rank by pretest response and randomize within groups.
5. Does the analysis run at the region level? Or, if it runs per user, was the design effect applied. What you cannot do is both at once.
6. Does the test period cover the response delay? Especially for offline sales.
7. Was the power calculation done on the number of regions? Not on the number of people.

Common mistakes

• Running the two proportion test on user counts. It is the mistake this whole article is about, and it is attractive because it produces the prettiest p-value in the report.
• Hand picking the regions. Comparing the capitals against everywhere else is not an experiment, it is observation wearing an experiment costume.
• Ignoring media spillover between neighboring regions. A broadcaster crossing the region boundary contaminates control and shrinks the measured gap.
• Using too few regions. Ten regions per arm almost never gives power to detect a realistic marketing effect, no matter how many people live in them.
• Comparing before and after instead of treatment against control. Without a simultaneous control group, anything that happened nationwide becomes your result.

Make this automatic with Donnu

The expensive mistake in a geo experiment is not the model arithmetic, it is the temptation to switch units at analysis time, because the per user version always looks more conclusive.

In Donnu, an experiment declares its randomization unit in the design, and the report refuses to present a confidence interval computed on a unit different from the one that was randomized. When units are clustered, the dependence warning appears next to the result, with the design effect estimated from your own data rather than from a generic reference value. And if you want to redo any part of the arithmetic yourself, the p-value calculator and the confidence interval calculator accept raw counts.

References

• Vaver, J. and Koehler, J. Measuring Ad Effectiveness Using Geo Experiments. Google Inc., 2011. Source for the definition of a geo experiment with non-overlapping regions randomly assigned to control or treatment and realized through geo-targeted advertising, the observation that geos as small as postal codes are not generally feasible and that the 210 Nielsen Designated Market Areas are one possible set in the United States, the pretest and test period structure, the linear model with pretest response and spend differential fitted with weights equal to the inverse of the pretest response, the second model fitted only on control geos to estimate the counterfactual spend, the definitions of return on ad spend and cost per incremental click, the stratification procedure with groups of size M and the roughly 10 percent reduction in the confidence interval, the response delay for offline sales and the need to extend the test period, and the warning that reusing pretest data makes interval estimates overly optimistic. research.google.
• Tu, S., Li, C., Wester, C. W., De Schacht, C. and colleagues. Unified and Simple Sample Size Calculations for Individual or Cluster Randomized Trials with Skewed or Ordinal Outcomes. arXiv 2505.01640. Source for the form of the design effect as 1 plus the intraclass correlation times the cluster size minus 1, the attribution of the idea to Kish as a measure of the impact of a sampling design on the variance of an estimator, its application by Donner and colleagues as a sample size inflation factor, and the relationship between individual and cluster sample sizes. arxiv.org.
• Kastelman, D. and Ramesh, R. Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash. DoorDash, 2018. Reference for the general principle of matching the randomization unit to the analysis unit, applied there to time and region units rather than purely geographic ones. careersatdoordash.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. General reference on choosing the randomization unit and on the consequences of analyzing at a different unit than the one randomized. Companion material at experimentguide.com.

Read next: Switchback experiments · Interference between variants · Ratio metrics and the delta method · Attributing revenue to your A/B test winner · P-value calculator · Leia em português