Budget Experiments: Where Spend Stops Paying Off
A budget experiment measures a curve, not a winner. Marginal return versus average ROI, why two arms never find the optimum, and the cost in power.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
A budget experiment has no winner. It has a curve. The treatment is not a page variant or a creative, it is a spend level, and the question is not “which arm converted more”, it is “does the next dollar still pay for itself”. Three consequences follow, and they almost never make it into the test plan. First: average ROI and marginal return are different numbers, and the second one is what decides. In the worked example in this guide, the highest spend level shows an apparent return of 8.0100 while the last dollar invested returns 0.8100, which means it gives back less than it cost. Second: two arms never find an optimum, because the optimum lives between the steps. Third, and the expensive one: the adjacent step, the one you actually care about, is the smallest effect in the entire experiment, and the smallest effect demands the largest sample. This guide covers how to randomize a spend level, how to read a curve instead of a champion, what saturation costs you in power, and where the experiment ends and the mix model begins. This guide is part of our complete guide to A/B testing.
A budget experiment returns a curve, not a winner
In a page test the treatment is discrete: it either exists or it does not. The visitor sees the new button or the old one, and “which one wins” is a sensible question because there is nothing in between the two versions.
Budget does not behave that way. Between spending 60,000 dollars and spending 180,000 dollars over a window there are infinitely many intermediate levels, and the outcome moves continuously along that continuum. When you randomize two spend levels and declare a winner, here is what actually happened: you sampled two points of a function and crowned the higher one. With only two points, the higher one is almost always the bigger spend, because more media usually buys more conversions. Nobody asked whether it bought them cheaply.
Google’s Meridian documentation names the two quantities separately. Incremental outcome is defined as the portion of your outcome, such as sales or conversions, that was caused by a specific marketing activity. ROI is that incremental outcome divided by cost, and the documentation describes it in plain terms as a historical, channel-wide average. mROI, by contrast, is the return on a small increase in spend, for example the next dollar, computed with a small increment, which the documentation exemplifies as 1 percent.
The sentence that closes the argument sits on the same page: if the mROI is much lower than the ROI, the channel is beginning to saturate at its historical spend level. The documentation’s own illustration is a shoe store with an ROI of 2.50 dollars and an mROI of 0.80 dollars in the same channel. Comfortable average, negative margin.
Average ROI and marginal return are different numbers
This is the most expensive confusion in paid media, and it is not a statistics problem. It is an accounting problem.
Take the high level from the worked example: 8,010 conversions over a window that burned 180,000 dollars, with an average order value of 180 dollars. Multiply, then divide: 1,441,800 dollars of attributed revenue over 180,000 dollars of spend gives an apparent return of 8.0100. That is the number on the dashboard, that is the number that goes into the deck, and it is ten times the true marginal return at that spend level, which is 0.8100.
The gap is not rounding, it is attribution. The 8.0100 credits media with every conversion that happened while media was running, including the 7,200 conversions the low level was already delivering on a third of the spend, plus all the ones that would have shown up at zero spend. The 0.8100 asks something far narrower and far more useful: of the last 60,000 dollars you added, how much came back?
| number | what it measures | value in the example | what it can decide |
|---|---|---|---|
| apparent return at the level | attributed revenue divided by total spend | 8.0100 | almost nothing, because it includes demand media did not cause |
| average return on the increment | incremental revenue from L to H divided by the spend increase | 1.2150 | whether sitting at the high level beats sitting at the low one |
| marginal return on the last step | incremental revenue from M to H divided by that step’s $60,000 | 0.8100 | whether the next dollar pays, which is the real question |
Look at the middle row. The average return on the increment, 1.2150, blends one step that pays 1.6200 with one step that pays 0.8100. It lands above 1, which makes the move from low to high look like a good trade. Half of it was: the first half of the increase was excellent and the second half destroyed value. Averaging the two hides exactly the fact you needed.
Lewis and Rao explain why no amount of observational data cleans this up. Across their 25 field experiments with large US retailers and brokerages, 19 retailers and 6 financial services firms, together covering 2.8 million dollars of digital advertising spend, the standard deviation of individual-level sales is typically 10 times the mean over a typical campaign evaluation window. With that much noise in the outcome, any gap between exposed and unexposed people that did not come from a coin flip is dominated by selection rather than by effect.
How to randomize a spend level
Randomizing a page is easy: the visitor lands, you flip a coin, they see one version. Randomizing a spend level is harder, because budget is a property of the campaign, not of the person. Three units of randomization solve that in different ways.
Region. You group geographies into comparable pairs and randomize which pair gets which spend level. That is the design behind this guide’s worked example and the most accessible one for advertisers, because it needs no special access to the platform. The mechanics, the pairing and the read are covered in geo experiments, and the causal read of the result in incrementality testing. The cost is the unit: you have very few regions, and a handful of large units carries far more variance than a crowd of small ones.
User submarket. You split the user base into two disjoint sets of equal size and run one spend level in each. Hermle and Martini describe this design in detail, and it is available only to whoever operates the ad platform, because it requires isolating entire auction submarkets. It is the cleanest of the three and, for exactly that reason, the least available to advertisers.
Platform-side budget split. The platform divides the total budget across the experiment arms. Google Ads documentation is explicit about which mechanism applies where: for certain Search, App and Display experiments Google supports budget splits, while for others, such as Performance Max, Demand Gen and Video, it supports a traffic split. Under a budget split, Google divides the total budget across the arms; the page’s own example is a 70-30 split, where 70 percent of the budget goes to control and 30 percent to treatment. Under a traffic split, what gets divided is how many times the ads enter the auction.
| unit of randomization | who can use it | what it handles well | what it costs |
|---|---|---|---|
| region | any advertiser | isolates the auction between arms, allows account-wide spend levels | few units, high variance, needs careful pairing |
| user submarket | the platform only | person-level randomization with no auction cannibalization | requires control of the auction, out of reach for advertisers |
| platform-side budget split | advertisers, on supported campaign types | quick to set up, no infrastructure of your own | the mechanism varies by campaign type and the arms share one total |
Two operational warnings from the Google Ads documentation apply to that third case. First, the recommendation is to run the experiment for at least 4 to 6 weeks, or longer if your conversion delay is long, a mechanism covered in conversion lag. Second, the first 7 days of data are discarded to account for the experiment ramp-up time. In other words, the platform already bakes in a burn-in period, and your analysis plan needs to know that before you compute duration.
Worked example: three levels, one step that decides
The design: a geo experiment with three spend levels randomized across paired regions, 900,000 people per level, a 4 week window, and an average order value of 180 dollars. The numbers below came out of the same statistics engine that powers the calculators on this page.
| level | spend in the window | conversions | rate |
|---|---|---|---|
| low (L) | $60,000 | 7,200 | 0.8000% |
| medium (M) | $120,000 | 7,740 | 0.8600% |
| high (H) | $180,000 | 8,010 | 0.8900% |
Three levels produce three possible comparisons. All two-sided, at 95 percent:
| comparison | A | B | abs. diff. | rel. lift | z | p | 95% CI (pp) | verdict |
|---|---|---|---|---|---|---|---|---|
| M against L (doubles the spend) | 7,200/900,000 = 0.8000% | 7,740/900,000 = 0.8600% | +0.0600 pp | +7.5000% | 4.4364 | 0.000009 | +0.0335 to +0.0865 | M wins |
| H against M (adjacent step, +50% spend) | 7,740/900,000 = 0.8600% | 8,010/900,000 = 0.8900% | +0.0300 pp | +3.4884% | 2.1609 | 0.030704 | +0.0028 to +0.0572 | H wins |
| H against L (triples the spend) | 7,200/900,000 = 0.8000% | 8,010/900,000 = 0.8900% | +0.0900 pp | +11.2500% | 6.5957 | 0.000000 | +0.0633 to +0.1167 | H wins |
Read as an ordinary A/B test, the answer is trivial: the high level won both comparisons it appeared in, so spend more. Paste 7,740 conversions out of 900,000 against 8,010 out of 900,000 into the calculator below to reproduce the middle row: the screen shows p of 0.0307 and a lift of +3.5%, which are the 0.030704 and plus 3.4884 percent from the table, rounded.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Now read the same experiment as a curve. Each step cost the same 60,000 dollars:
- L to M: plus 540 conversions for plus $60,000, which works out to $111.11 per incremental conversion and a marginal return of 1.6200. Incremental revenue of $97,200.00.
- M to H: plus 270 conversions for the same plus $60,000, which works out to $222.22 per incremental conversion and a marginal return of 0.8100. Incremental revenue of $48,600.00.
- L to H: plus 810 conversions for plus $120,000, an average return on the increment of 1.2150.
Cost per incremental conversion doubled from one step to the next. Marginal return halved and crossed 1 on the way down. The high level won on significance and lost money doing it, and both statements hold at once because they answer different questions: the first asks whether the effect exists, the second asks whether it was worth the price.
The step you care about is the most expensive one to measure
Here is the arithmetic that makes saturation a genuinely hard measurement problem rather than a careless read.
Required sample size scales with the inverse square of the effect you want to detect. Near saturation, effects between neighboring levels are small by definition, because the curve is flattening out. So the closer you get to the point where the decision matters, the smaller the effect and the larger the sample. Both move the wrong way together.
On a 0.86 percent baseline, two-sided at 95 percent with 80 percent power:
| what to measure | baseline | relative MDE | N per level | duration with 3 levels at 900,000 people/week |
|---|---|---|---|---|
| step H against M | 0.86% | 3.4884% | 1,512,816 | 36 days |
| step M against L | 0.80% | 7.5000% | 358,916 | 9 days |
| +10% relative | 0.86% | 10.0000% | 189,927 | 5 days |
| +5% relative | 0.86% | 5.0000% | 741,784 | 18 days |
| +2% relative | 0.86% | 2.0000% | 4,568,901 | 107 days |
The step that decides, M to H, asks for 1,512,816 people per level, nearly double the 900,000 the experiment actually had. The step that no longer mattered, L to M, asks for 358,916. Reproduce the first row in the calculator below with a 0.86 percent baseline, a relative minimum detectable effect of 3.49 percent, 95 percent confidence and 80 percent power; with 3.49 the screen shows 1,511,417 per level, and the 1,512,816 in the table comes from the full fraction, 3.4884 percent.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
With the 900,000 people per level the experiment really had, effective power lands here:
- step H against M: 57.96 percent. The result came out significant, p equal to 0.030704, but it came from a test with power below 60 percent. When an underpowered test rejects the null, the estimated effect tends to come back inflated, because only the luckiest samples clear the threshold. Choosing a sensible threshold in the first place is the subject of minimum detectable effect, and the traffic arithmetic sits in how many visitors an A/B test needs.
- H against L: 100.00 percent. The comparison nobody needed is the only one with power to spare.
Lewis and Rao hit the same wall from another direction, with numbers worth quoting. In their retail experiments, the median confidence interval on return on investment is about 100 percentage points wide. The median standard error on ROI is 26.1 percent in the retail experiments and 115 percent in the brokerage ones. To reliably separate a highly profitable campaign, at plus 50 percent ROI, from one that merely breaks even at zero, their median campaign would have had to be 9 times larger. For a 10 percent tolerance on the ROI difference it would have to be 62 times larger, which the authors call nearly impossible for a campaign of any realistic size.
Two further figures from the same study help calibrate the intuition. In a stylized example of a highly profitable campaign, the authors compute that the treatment variable would explain an R-squared on the order of 0.0000054. And to earn plus 25 percent ROI, the median campaign had to cause a 0.35 dollar lift in average sales per person, against campaign costs running from 0.02 to 0.35 dollars per exposed user, most of them near 0.10 dollars. Every experiment in the study had more than 500,000 unique users and most exceeded 1,000,000; the authors calculate that informative experiments can easily require more than 10 million person-weeks.
Why the platform can do this and you cannot
Hermle and Martini, at LinkedIn, attack exactly this problem, and their solution explains why measuring the curve is easy on one side of the auction and hard on the other.
The starting point is endogeneity. Correlating budget with performance identifies nothing, and the bias can run either way. An advertiser who expects strong performance puts more budget behind the campaign, which biases the estimate upward. An advertiser chasing a target level puts more budget in precisely when they expect weak performance per unit of spend, which biases it downward, and the authors note that in extreme cases this produces measured incrementality that is negative while the true causal effect is positive.
Their answer is the asymmetric budget split, and the most important detail is what it does not do: it does not change the ad’s total budget. The procedure has three steps. First, split the user base into two disjoint sets of equal size, creating submarkets M1 and M2. Second, create two copies of the same ad and divide the total budget unevenly between them, with the high budget set to alpha times B and the low budget to one minus alpha times B, where alpha is greater than 0.5. Third, randomly draw which copy goes to which submarket, with probability 0.5 for each allocation.
The estimator uses the random component of budget, defined as the budget actually allocated minus 0.5 times the ad’s total budget, in other words the deviation from an even split. That deviation is what the coin flip created, and isolating it is what separates experimental variation from the natural variation in budget across ads.
Two relationships are worth recording. The symmetric budget split, the one used to handle interference between variants in a marketplace, is the special case where alpha equals 0.5. The two answer different questions and are compatible with each other. And the method, by the authors’ own account, generalizes to any setting where a divisible, exhaustible quantity drives the outcome, including inventory, notification quotas and impression caps.
In the production deployment on LinkedIn’s Jobs Marketplace, running continuously rather than as a one-off measurement, the asymmetry factor was fixed at alpha equal to 0.6, chosen through a power calculation that accounts for outcome variation in the marketplace. The published result axis is censored for confidentiality, but the direction is stated: incremental return per unit of budget is lower in countries with higher GDP per capita and lower for higher-seniority jobs, because in those cases the market price of a candidate is higher.
So what is left for the advertiser? Not submarket isolation, and not a free hand with their own budget either, because moving budget mid-campaign disturbs delivery pacing and the optimization systems running on top of it, mechanisms covered in smart bidding and budget pacing. Two things are left: a geo split you run yourself, and the platform-side budget split where the campaign type supports it, with the caps and the seven-day discard noted above.
The boundary with marketing mix modeling
It is worth drawing a sharp line between what belongs to marketing mix modeling calibrated by experiments and what belongs here, because both use the word “curve” for different objects.
Over there, the object is a model, and the experiment serves as an anchor. A response curve, in Meridian’s definition, is incremental outcome as a function of spend, preserving historical flighting patterns. It is fitted on historical data and exists across the whole range. Budget optimization scenarios then run on top of it, and the documentation declares three. Scenario 1, fixed budget, is the default method: find the optimal allocation across channels and tactics to maximize return, with a default spend constraint of plus or minus 0.3 per channel, meaning each channel may move between 70 and 130 percent of its allocated spend. Scenario 2, flexible budget with a target minimum ROI, asks how much you can spend with no budget cap until overall ROI hits the target; that constraint applies at the total marketing ROI level and not at channel level, with default bounds of 0 to 200 percent of allocated spend. Scenario 3, flexible budget with a target mROI, asks how much you can spend until each channel’s mROI hits the target; here the constraint applies at channel level and not at the total, with the same bounds.
Here, the object is the curve itself, measured by randomization across a narrow range. The experiment does not hand you the whole curve, it hands you two or three points on it, each step carrying a confidence interval. Far less sweeping, and far harder to argue with.
The two fit together, and the reason is stated in the Meridian documentation: extrapolation risk affects incremental outcome estimation for points on the response curve above historical spend, and the risk grows the further out you go. The experiment settles that inside the range it covered, and nowhere else. In practice the model proposes the direction and size of the move, the experiment confirms the next step, and the following model run inherits a measured point instead of an extrapolated one.
Guardrail metrics
A budget experiment fails in ways a page test never does, because the treatment reaches into the delivery system rather than only into what the visitor sees.
| signal | what to watch | what to do if it fires |
|---|---|---|
| realized spend per level | each arm’s actual spend against the level’s planned spend | if a level underspent, the treatment was never applied; do not read the curve yet |
| budget-limited arm | any level hitting its delivery cap before the day ends | that level is throttled, not treated; the step you measured is not the step you planned |
| people reached per level | divergence between levels that widens over time | check the randomization of units before reading conversions; see sample ratio mismatch |
| average order value per level | a gap between levels larger than historical noise | extra spend may be buying a different audience, which makes incremental revenue non-comparable across steps |
| conversion lag | median days from exposure to conversion against your read date | wait at least one full cycle past the end before locking a marginal return |
| marginal return on the last step | the adjacent step crossing 1 on the way down | stop climbing; the next step costs more sample and returns less |
One rule covers all six: before you read the curve, confirm that every level actually spent what the randomization told it to spend. A level that could not spend its assigned budget is not a point on your curve, it is a point on a different one.
Common mistakes
- Declaring a winner in a budget experiment. The highest level almost always “wins” on absolute conversions. That is arithmetic, not information. The information lives in the slope between neighboring levels.
- Using the apparent return at a level to decide the next dollar. In the worked example the apparent return is 8.0100 and the marginal return is 0.8100. Deciding on the first one leads you to raise budget forever.
- Running only two levels. Two points give you a line, and a line has no saturation point. You measure an average slope that can hide a value-destroying step, exactly the way 1.2150 hides 0.8100.
- Sizing the sample on the big step. The L to M step needs 358,916 people per level, the M to H step needs 1,512,816. Sizing on the first guarantees the second one, the one that decides, arrives underpowered.
- Treating a significant, underpowered result as a reliable estimate. The H against M step came in at p equal to 0.030704 with 57.96 percent power. When a weak test rejects, the estimated effect tends to come back inflated.
- Extrapolating past the randomized range. The experiment measured $60,000 through $180,000. Nothing in it speaks to $400,000, and extrapolation risk grows with distance from observed spend.
- Changing budget mid-window to “fix” a level. That does not fix anything, it creates a fourth spend level nobody randomized and no arm represents.
- Inferring the curve from account history. Budget is endogenous: the campaigns expected to do well got more money, and campaigns chasing a target got more money when performance per dollar was worse. History blends those two biases in opposite directions.
- Peeking at the result before the end with no sequential plan. Checking weekly and stopping when the step turns significant inflates false positives, a mechanism covered in the peeking problem.
Make this automatic with Donnu
The specific pain in this guide is a reading problem, not an execution problem: the number your media dashboard hands you is a channel average, and the decision in front of you is marginal. No tool fixes that on its own, because the gap between the two numbers depends on how much demand would have existed without the media, and only randomization answers that.
What can be automated is the statistical half: declaring the level, the window and the minimum effect before you start, then reading each step with its confidence interval beside it instead of a champion. With Donnu the experiment lives on your own site, with start date, burn-in period and read date declared before the test ships, and the report shows the absolute difference with the interval next to it, which is the format that will not be fooled by an underpowered step. If you only want the arithmetic, our sample size calculator and significance calculator are free and open.
Donnu is one option among several, and no on-page testing tool measures the budget curve for you: that part requires randomization in the media layer, by region or through the platform itself. What does not change from tool to tool is the principle: if you never randomized more than one spend level, the marginal return you are deciding on was not measured, it was assumed.
Frequently asked questions
The questions at the top of this page cover whether a budget experiment is the same thing as an A/B test, the difference between ROI and marginal return, why two spend levels are not enough, how many people it takes to measure saturation, whether the curve can be estimated without an experiment, and what a budget experiment does not answer.
References
- Lewis, R. and Rao, J. The Unfavorable Economics of Measuring the Returns to Advertising, Quarterly Journal of Economics 130(4), 2015, pp. 1941-1973. Source for the 25 field experiments with large US retailers and brokerages (19 retailers and 6 financial services firms) covering 2.8 million dollars of digital advertising spend, for the median confidence interval on ROI in retail being about 100 percentage points wide, for the median standard errors of 26.1 percent in retail and 115 percent in brokerage, for the standard deviation of individual-level sales being typically 10 times the mean, for the 9 times and 62 times campaign-size factors, for the R-squared of 0.0000054 in their stylized example of a highly profitable campaign, for the 0.35 dollar per person lift needed for plus 25 percent ROI, for the 10 million person-weeks and for campaign costs of 0.02 to 0.35 dollars per exposed user. Checked on September 25, 2026. gwern.net.
- Hermle, J. and Martini, G. (LinkedIn). Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split, arXiv:2207.00206, 2022. Source for the two-directional endogeneity argument about budget, for the three-step asymmetric split design with disjoint submarkets, for the estimator built on the deviation from an even split, for the relationship to the symmetric split as the special case where alpha equals 0.5, for the alpha equal to 0.6 used in production on LinkedIn’s Jobs Marketplace, and for the direction of the result by GDP per capita and job seniority. Checked on September 25, 2026. arxiv.org.
- Google for Developers, Meridian. Incremental Outcome, ROI, mROI & Response Curves. Source for the definition of incremental outcome, for the description of ROI as a historical, channel-wide average, for the definition of mROI as the return on a small increase in spend with an increment such as 1 percent, for the statement that an mROI much lower than ROI signals saturation at historical spend, for the shoe store example with an ROI of 2.50 and an mROI of 0.80 dollars, and for the warning about extrapolation risk above historical spend. Checked on September 25, 2026. developers.google.com.
- Google for Developers, Meridian. Budget optimization scenarios. Source for the three optimization scenarios, for the default plus or minus 0.3 per-channel constraint under fixed budget, for the 0 to 200 percent bounds in the flexible scenarios, and for the distinction that a target ROI applies at the total marketing level while a target mROI applies at channel level. Checked on September 25, 2026. developers.google.com.
- Google Ads Help. Experiments FAQs. Source for the distinction between budget splits, supported on certain Search, App and Display experiments, and traffic splits, supported on Performance Max, Demand Gen and Video, for the 70-30 example of dividing the total budget across arms, for the recommendation to run experiments for at least 4 to 6 weeks, and for discarding the first 7 days of data because of experiment ramp-up time. Checked on September 25, 2026. support.google.com.
Read also: Budget reallocation experiments · Budget pacing · Geo experiments · Incrementality testing · Marketing mix modeling · Minimum detectable effect · How many visitors · The peeking problem · Smart bidding · Leia em português
Frequently asked questions
- Is a budget experiment the same thing as an A/B test?
- It is an A/B test with a different kind of treatment. In a classic A/B test you randomize people between two versions and ask which one wins. In a budget experiment the treatment is a spend level, and the answer is not a winner, it is a curve: how much each additional dollar still produces. That is why these designs usually carry three or more arms, and why you read the slope between neighboring arms rather than which arm posted the biggest number.
- What is the difference between ROI and marginal return?
- ROI is a historical, channel-wide average; marginal return is what the next dollar buys. The Meridian documentation defines mROI as the return on a small increase in spend, with an increment such as 1 percent, and states that when mROI is much lower than ROI the channel is beginning to saturate at its historical spend level. In the worked example in this guide, the apparent return at the high spend level is 8.0100 while the marginal return on the last step is 0.8100. The second number is the one that decides the next dollar.
- Why are two spend levels not enough?
- Because two points only describe a straight line, and the question is about curvature. With two arms you measure the average slope between them and still do not know whether break-even sits before, inside or beyond that interval. In this guide, jumping from the low level straight to the high one returns an average of 1.2150, which looks comfortable, but that figure averages one step paying 1.6200 with another paying 0.8100. Three levels show where the curve crosses break-even; two levels hide it.
- How many people do I need to measure saturation?
- More than intuition suggests, because the step that decides is the smallest effect in the whole experiment. In this guide, detecting the step from the medium to the high level, plus 3.4884 percent relative on a 0.86 percent baseline, two-sided at 95 percent with 80 percent power, takes 1,512,816 people per level. The previous step, the one that no longer mattered, needs only 358,916. Sample size scales with the inverse square of the effect, so every move toward saturation multiplies the bill.
- Can I measure the curve without an experiment?
- You can estimate it, not identify it. Hermle and Martini show that correlating budget with performance identifies nothing, because budget is endogenous: advertisers who expect strong performance put more budget behind a campaign, and advertisers chasing a target level put more budget in precisely when they expect weak performance per unit of spend. A marketing mix model gives you a useful response curve, but the Meridian documentation itself warns that extrapolation risk grows the further you move past historical spend. The experiment is what anchors the range.
- What does a budget experiment not answer?
- It does not tell you exactly where the optimum sits, and it says nothing outside the range you randomized. An experiment with three levels returns two steps and therefore two marginal returns; the point where the curve crosses break-even lives between the steps, and the experiment only brackets it. It also does not answer how to split budget across channels, which is a reallocation problem, or how to distribute spend through the day, which is a pacing problem.