Budget Reallocation: The Portfolio Experiment
Budget reallocation cannot be tested channel by channel. Why the unit of analysis is the portfolio, what average CPA hides and what it costs to measure.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
When budget is shared, a channel is not a treatment. The treatment is the entire allocation, and the unit of analysis is the portfolio. That is not a methodological preference, it follows from how the platforms actually spend money: Meta states that ad set budget sharing moves up to 20 percent of one ad set daily budget to other ad sets, and that campaign level spend stays the same whether the feature is on or off. Read channel by channel after a reallocation and you get two false verdicts at once: the channel that lost money looks like it improved, and the one that gained money looks like it got worse, because each channel average CPA is contaminated by demand that was already there. In the worked example below, the move wins 360 conversions at identical total spend, and both per-channel readings tell you to reverse it. This guide covers why the portfolio is the only valid unit, what average CPA hides, what the correct stopping rule is, and why this is the least powerful test you will ever run. This guide is part of our complete guide to A/B testing.
A channel is not a treatment when budget is shared
The strongest argument against per-channel reading does not come from statistics. It comes from the platform itself.
Meta describes ad set budget sharing as a feature that lets you share up to 20 percent of your daily budget with other ad sets, to improve performance for campaigns not using a campaign budget. Then it adds the sentence that settles the argument: at a campaign level, spend remains the same whether ad set budget sharing is on or off.
Read that as a statement about treatments. If up to a fifth of one ad set budget can bankroll another, then the spend you observe in any single ad set is not a quantity you fixed. It is a quantity the system chose. You never assigned budget to a channel; you assigned a total to a set of channels and delegated the split.
Advantage+ campaign budget is the explicit version of the same thing. The documentation says the feature automatically manages your campaign budget across ad sets to get the overall best results, continuously distributing in real time to the ad sets with the best opportunities. And it draws the operational conclusion itself: budget may not be spent equally across ad sets, most of it may land on a single ad set if that is where the overall best results come from, and for that reason it is important to analyze results at the campaign level rather than at the ad set level.
Three practical consequences follow.
| what the platform does | what it means for your test |
|---|---|
| shares up to 20% of the daily budget across ad sets | per-channel spend is an outcome, not a treatment |
| redistributes in real time toward the best opportunity | the observed split changes day to day without you touching anything |
| keeps total campaign spend constant | the only number you genuinely fixed is the total |
Notice that the third line is good news. The total is stable, and the total is exactly what a reallocation hypothesis is written about.
In budget reallocation, the treatment is the entire allocation
Reallocating budget swaps one vector for another vector. Nothing is turned on and nothing is turned off.
The design in this guide is a geo experiment with two arms of matched regions, 1,200,000 people per arm, four weeks, an average order value of 40 dollars. Total spend is identical in both arms: 200,000 dollars. Only the split changes.
| arm | channel A | channel B | total |
|---|---|---|---|
| control (current allocation) | $140,000 | $60,000 | $200,000 |
| treatment (moves $40,000 from A to B) | $100,000 | $100,000 | $200,000 |
Two definitions have to be written into the plan before the test ships, and they are not the same definition.
The unit of randomization is the region. Randomization assigns regions to arms, and the region is the level at which the allocation is applied. The unit of analysis is the portfolio: conversions summed across all channels, divided by the arm population. Collapsing those two into one is the mistake that produces every other mistake in this guide.
Why the portfolio and not the channel? Because the outcome of interest is a sum, and the sum is the only quantity randomization protects. The moment you look at channel B inside the treated arm, you are opening a subgroup defined after randomization, with a denominator the treatment itself moved. Metrics whose denominator shifts with the treatment need their own handling, and the mechanism is spelled out in ratio metrics.
A close relative of this design answers a different question. Hermle and Martini, at LinkedIn, propose an asymmetric budget split: keep one ad total budget, divide it unequally between two comparable submarkets, and randomize which copy gets the larger share. What the two designs share is treating budget as a randomized treatment instead of an observed variable. What separates them is the object. There, the question is the return to more budget on the same ad. Here, it is the return to splitting a fixed total differently across different channels.
Worked example: the same money, split a different way
The numbers below came out of the same statistics engine that powers the calculators on this page.
The portfolio reading, the valid one, compares total conversions per arm over the arm population.
| A (control) | B (treatment) | abs. diff. | rel. lift | z | p | 95% CI (pp) |
|---|---|---|---|---|---|---|
| 9,000/1,200,000 = 0.7500% | 9,360/1,200,000 = 0.7800% | +0.0300 pp | +4.0000% | 2.6671 | 0.007652 | +0.0080 to +0.0520 |
That is 360 extra conversions at identical total spend. At 40 dollars per order, incremental revenue is 14,400.00 dollars. Divided by the 40,000 dollars that were moved, the move returned 0.009000 conversions per dollar moved, or 0.3600 dollars of revenue per dollar moved.
Paste 9,000 out of 1,200,000 against 9,360 out of 1,200,000 into the calculator below and the screen shows p = 0.0077 and a lift of +4.0%, which are the table 0.007652 and 4.0000 percent, rounded. The same math also runs in the significance calculator.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Now the per-channel decomposition. It decides nothing, but it belongs in the report, because it is what explains the portfolio result. Add it up and check: 6,300 plus 2,700 gives 9,000 in control, and 5,040 plus 4,320 gives 9,360 in treatment.
| channel | conversions in control | conversions in treatment | change | spend |
|---|---|---|---|---|
| A | 6,300 | 5,040 | -20.00% | $140,000 to $100,000 |
| B | 2,700 | 4,320 | +60.00% | $60,000 to $100,000 |
Channel A fell by a fifth. Channel B rose by three fifths. The portfolio rose 4 percent. All three statements are true at the same time, and only the third one answers the question that was asked.
What per-channel average CPA hides
Here is the trap, and it is the reason this guide exists.
| channel | average CPA in control | average CPA in treatment | what the dashboard suggests |
|---|---|---|---|
| A | $22.22 | $19.84 | A improved, give the money back to it |
| B | $22.22 | $23.15 | B got worse, pull the money out |
Both readings are wrong, and both point at undoing the exact move that earned 14,400.00 dollars.
The mechanism is simple once you stop looking at the average and look at where it comes from. A channel average CPA blends two populations: demand that would have shown up anyway, which is cheap, and demand the last slice of budget bought, which is expensive. Pull money out of A and what leaves is the marginal conversions, the expensive ones. The average of what remains improves. Put money into B and what arrives is marginal conversions, pricier than B old average. The average gets worse. In any reallocation, the source average CPA improves and the destination average CPA worsens, whether the move was smart or stupid. The metric carries no information about the decision: it answers the same way every time.
The truth lives at the margin.
| side of the move | conversions | marginal CPA | marginal return |
|---|---|---|---|
| pulling $40,000 out of A | -1,260 | $31.75 | 1.2600 |
| putting $40,000 into B | +1,620 | $24.69 | 1.6200 |
| net | +360 | 1.6200 minus 1.2600 = $0.3600 per dollar moved |
Pulling 40,000 dollars out of A cost 1,260 conversions, a marginal CPA of 31.75 dollars and a marginal return of 1.2600. Putting those same 40,000 dollars into B earned 1,620 conversions, a marginal CPA of 24.69 dollars and a marginal return of 1.6200. The net is 360 extra conversions, and the net gain per dollar moved is 1.6200 minus 1.2600, which equals 0.3600 dollars.
That 0.3600 is precisely the number the portfolio reading already handed you, arrived at from the other direction: 14,400.00 dollars of incremental revenue divided by the 40,000 dollars moved. Both calculations land on the same figure, and that agreement is how you know the reading is right. When the marginal math and the portfolio math disagree, one of them has an accounting error, and it is worth finding the error before making any decision.
Worth recording what portfolio average ROAS does, too: it climbs from 1.8000 to 1.8720. True, and far too small to serve as a signal. What decides is the gap between marginal returns, not the portfolio average, and that distinction is the same one that separates measuring incrementality from reading reported performance.
The stopping rule: equalize marginal returns
The decision rule is short. Keep moving money while the destination marginal return exceeds the source marginal return, and stop when the two are equal.
In this example, 1.6200 against 1.2600 says the move is worth making. It does not say 40,000 dollars is the right amount. As money flows into B, B marginal return falls; as it leaves A, A marginal return rises. At some point they cross, and past that point moving more destroys value. Each channel response curve, which is incremental outcome as a function of spend, holds that information, and the question of how much to spend on a single channel belongs to budget and saturation experiments.
Google Meridian documentation formalizes this in three optimization scenarios, and all three are worth knowing because they answer different questions.
| scenario | question | where the constraint binds | default per-channel spend constraint |
|---|---|---|---|
| fixed budget | what is the best split of a given total | per channel | 70% to 130% of allocated spend |
| flexible budget with target minimum ROI | how much can be spent until overall ROI hits the target | at the total marketing ROI level, not per channel | 0% to 200% of allocated spend |
| flexible budget with target mROI | how much can be spent until each channel mROI hits the target | at channel level, not at the total level | 0% to 200% of allocated spend |
The first scenario is the direct relative of a reallocation test: total fixed, split variable. The third is the stopping rule written down, because fixing a target mROI per channel is exactly forcing marginal returns to converge on a common number.
Two warnings, and they matter.
First: the model proposes, the experiment verifies. A calibrated mix model returns a candidate allocation; the experiment tests that allocation against the current one and returns the sign of the difference with an interval around it. The experiment does not sweep the space of allocations and does not find the optimum. It answers a binary question about a pair of points. How the two tools fit together is covered in marketing mix modeling with experiments.
Second: extrapolating past historical spend is risky by construction. Meridian documentation warns that extrapolation risk affects incremental outcome estimation for points on the response curve above historical spend, and that the risk grows the further out you go. A proposal that doubles the budget of a channel which has never operated at that level is an extrapolation, not a forecast.
Why this is the least powerful test you will ever run
A reallocation effect is a difference between two marginal returns that, by definition, were already close. Had they been far apart, somebody would have moved the money long ago. Small effect means large sample, and the scaling is quadratic.
| relative MDE | N per arm | duration at 600,000 people per week |
|---|---|---|
| 10.0000% | 218,037 | 6 days |
| 5.0000% | 851,547 | 20 days |
| 4.0000% (the real effect in this example) | 1,324,101 | 31 days |
| 2.0000% | 5,244,878 | 123 days |
The duration column uses the flow from the example itself: 1,200,000 people per arm over 4 weeks is 2,400,000 people in total, or 600,000 per week across both arms. Notice that the real effect in this example would ask for 31 days and the experiment ran for 28, which is another way of seeing the 76.02 percent power: it came up three days of collection short of the 80 percent that was planned. Going from a 4 percent effect to a 2 percent effect quadruples the sample, because sample size scales with the inverse square of the effect. The concept behind that column lives in minimum detectable effect, and the calendar math also runs in the test duration calculator.
Two-proportion normal approximation, traffic split evenly across variations. The date uses your timezone and updates live.
Now the other side of the same coin. With the 1,200,000 people per arm this experiment actually had, power came to 76.02 percent for the 4 percent relative effect, short of the conventional 80 percent. And for a 2 percent relative effect, which is a more modest and perfectly plausible reallocation, power falls to 26.77 percent. Three out of four experiments this size would miss a real 2 percent gain.
None of that is blog pessimism. It is the central result of the literature on measuring advertising returns. Lewis and Rao analyzed 25 field experiments with large US retailers and brokerages, 19 retailers and 6 financial services firms, totaling 2.8 million dollars in digital advertising spend, and they report that the median confidence interval on return on investment is over 100 percentage points wide. Every experiment in the study had more than 500,000 unique users, most passed 1,000,000, and even so the median campaign would have to be 9 times larger to reliably tell a very profitable campaign, at 50 percent ROI, from one that merely breaks even. For a tolerance of 10 percentage points of ROI difference, the campaign would have to be 62 times larger, which the authors describe as nearly impossible for a campaign of any realistic size.
Their explanation is the variability of individual behavior: the standard deviation of individual-level sales is typically 10 times the mean over a typical campaign evaluation window. In the paper stylized example of a highly profitable campaign, the R squared of the treatment variable would be on the order of 0.0000054.
A reallocation test is more demanding still, because the question is finer. Measuring whether a channel works compares spending against not spending. Measuring a reallocation compares two ways of spending the same total, and the gap between them is the gap between two neighboring marginal returns. You traded a hard question for a harder one, and the sample calculation has to reflect that before the test starts, not after. The starting point is how many visitors an A/B test needs.
The boundary with interference between variants
Two things get conflated here, and telling them apart is what keeps you from designing the wrong experiment.
In interference between variants, the problem is arm A spoiling the measurement of arm B by competing for a shared resource. Interference is a defect: it exists against your will, it contaminates the estimate, and you spend effort eliminating it by isolating submarkets or changing the unit of randomization.
In a reallocation test, competing for the resource is the treatment. You are not trying to stop one channel from taking money from the other; you are declaring exactly how much one takes from the other and randomizing that rule across regions. What used to be contamination became the object of study. That is why portfolio level reading is not a concession: it is what keeps the estimate clean, because the sum of channels inside one arm suffers interference from nothing.
The boundaries with the other two pieces in this arc are worth drawing too.
| question | where it lives |
|---|---|
| how much to spend on a channel, and where it saturates | budget and saturation experiments |
| how to split a fixed total across channels | this piece |
| how budget is delivered within the day | budget pacing experiments |
| which channel gets credit for the conversion | attribution model testing |
The last row deserves an extra warning. Changing the attribution model changes the apparent CPA of every channel without a single cent moving anywhere. If attribution changes mid window, reallocation and attribution are confounded and the experiment no longer separates them.
Guardrail metrics
Beyond the usual guardrails, such as sample ratio mismatch, four signals are specific to this design.
| signal | what to watch | what to do if it fires |
|---|---|---|
| total spend per arm | any gap between arm totals above 1 percent | stop reading: if the total diverged, the treatment is no longer just the split |
| realized versus planned split | what each channel actually spent, day by day, against the plan | if the platform reshuffled on its own, document the real split and reanalyze against it |
| the two calculations agreeing | gain per dollar moved from the margin against gain from the portfolio | if the two numbers disagree, you have an accounting error before you have a result |
| attribution or tagging changes | any change of model, window or tag inside the test window | discard the affected period in both arms by the same rule, decided in advance |
The rule that covers all four: before reading the result, confirm that the only difference between the arms was how the same total was split.
Common mistakes
- Reading channel by channel. This is the mistake that spawns the others. The treatment is the whole allocation and the outcome is the sum.
- Deciding on average CPA. In a reallocation the source average always improves and the destination average always worsens. The metric carries no information about the decision.
- Conflating unit of randomization with unit of analysis. You randomize regions and analyze the portfolio. Opening a channel inside an arm opens a post-randomization subgroup.
- Letting total spend drift between arms. If the total changed, you tested two things at once and you will not separate them afterward.
- Sizing for a single channel effect. The portfolio effect is smaller than any isolated channel effect, because it is a difference between two marginal returns. Size for the channel and the test arrives with no power.
- Asking the experiment for the optimal allocation. It tests a pair of points. Finding the optimum is modeling work, and the model needs the experiment to stay calibrated.
- Treating the planned split as the realized split. With shared budgets or automatic distribution, the split that happened may not be the one you typed. Check before analyzing.
- Extrapolating well above historical spend. Meridian documentation itself warns that extrapolation risk grows the further you move from observed spend.
Make this automatic with Donnu
The fix here is not a prettier report. It is a decision about the unit of analysis, made before the test ships.
A media reallocation test is expensive by nature, and most of the cost comes from the effect being small. So it pays to separate what genuinely needs a geo experiment across millions of people from what can be answered on your own site with a fraction of that. Donnu sits in the second layer: the experiment lives on your page, the unit of analysis is declared before the start, and the report shows the absolute difference with its interval beside it, which is the form that does not get fooled by an average that improved for the wrong reason. Donnu is one option among several, and no on-site testing tool replaces a geo experiment when the question is about splitting media. What does not change from tool to tool is the principle: if the treatment is the entire allocation, the deciding number is the portfolio number, and any channel average that disagrees with it is measuring something else.
Frequently asked questions
The questions at the top of this page cover whether you can test one channel at a time, why per-channel CPA misleads after a reallocation, what the correct unit of analysis is, how much traffic the test demands, whether the experiment finds the optimal allocation, and what to do when the platform reshuffles budget on its own.
References
- Meta Business Help Center. About ad set budget sharing. Source for the statement that ad set budget sharing lets you share up to 20 percent of your daily budget with other ad sets, and that at a campaign level spend remains the same whether the feature is on or off. Checked on September 25, 2026. facebook.com.
- Meta Business Help Center. About Advantage+ campaign budget. Source for the description of the feature automatically managing campaign budget across ad sets, continuously distributing in real time, and for the guidance to analyze results at the campaign level rather than at the ad set level. Checked on September 25, 2026. facebook.com.
- Google for Developers, Meridian. Budget optimization scenarios. Source for the three budget optimization scenarios, the default per-channel spend constraints of each, the fact that the target minimum ROI constraint binds at the total marketing ROI level while the target mROI constraint binds at channel level. Checked on September 25, 2026. developers.google.com.
- Google for Developers, Meridian. Incremental Outcome, ROI, mROI and Response Curves. Source for the warning that extrapolation risk also affects incremental outcome estimation for points on the response curve greater than historical spend, and that the risk increases the further out you go. Checked on October 14, 2026. developers.google.com.
- Lewis, R. and Rao, J. The Unfavorable Economics of Measuring the Returns to Advertising, Quarterly Journal of Economics 130(4), 2015, pp. 1941-1973. Source for the 25 field experiments with 19 retailers and 6 financial services firms totaling 2.8 million dollars in spend, the median ROI confidence interval over 100 percentage points wide, the 9 times and 62 times campaign scale factors, the standard deviation of individual sales at 10 times the mean, and the R squared on the order of 0.0000054 for a highly profitable campaign in the paper stylized example. Checked on September 25, 2026. gwern.net.
- Hermle, J. and Martini, G. Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split, arXiv:2207.00206, 2022. Source for the asymmetric budget split design, which holds total budget fixed and randomizes which copy of an ad receives the larger share across comparable submarkets. Checked on September 25, 2026. arxiv.org.
Read also: Interference between variants · Geo experiments · Marketing mix modeling · Ad incrementality testing · Ratio metrics · Leia em português
Frequently asked questions
- Can you test one channel at a time in a budget reallocation?
- No, because the treatment does not fit inside a single channel. Moving money from A to B changes both at once, and the sum is the thing you actually want to know. Isolate channel B, conclude that it improved, and you have measured one half of an arithmetic problem whose two halves carry opposite signs. The treatment is the entire allocation, and the only outcome that answers the question is the summed outcome of both channels within the same arm.
- Why does per-channel CPA mislead you after a reallocation?
- Because average CPA blends the demand you bought with the demand that was already there. In the worked example below, channel A loses 40,000 dollars of budget and its average CPA improves from 22.22 to 19.84 dollars, since A keeps its cheap conversions and sheds its expensive ones. Channel B receives those same 40,000 dollars and its average CPA gets worse, from 22.22 to 23.15 dollars, since the new conversions cost more than its old average. Both readings tell you to undo a move that won 360 conversions.
- What is the unit of analysis in a reallocation test?
- The unit of randomization is the region, or the user set, and the unit of analysis is the portfolio. They are not the same thing and both belong in the plan. You randomize regions across two arms, but the number you compare is total conversions in the arm divided by the arm population, summed over every channel. Reading channel by channel inside an arm opens a subgroup that randomization never promised to balance.
- How much traffic does a reallocation test need?
- Far more than a page test, because the effect is a difference between two similar marginal returns. In the example below, detecting a 4 percent relative lift on a 0.75 percent baseline at 80 percent power requires 1,324,101 people per arm. With the 1,200,000 people per arm this experiment actually had, power came to 76.02 percent. For a 2 percent relative effect power drops to 26.77 percent and the required sample climbs to 5,244,878 people per arm.
- Does the experiment find the optimal budget allocation?
- No. An experiment tests one allocation against another and returns the sign of the difference with an interval around it. Finding the point where marginal returns equalize is modeling work, not testing work. Meridian documents three optimization scenarios that propose allocations; the experiment exists to check whether a proposed allocation really beats the current one in the field. The model proposes, the experiment verifies, and that division of labor is what keeps both tools honest.
- What should you do when the platform reshuffles budget on its own?
- Accept that ad set level has stopped being a readable unit and move the analysis up to campaign level. Meta states that ad set budget sharing moves up to 20 percent of one ad set daily budget to other ad sets, and that campaign level spend stays the same either way. On Advantage+ campaign budget, the same documentation says it is important to analyze results at the campaign level rather than at the ad set level. The platform writes the rule, and the rule tells you where to read.