A/B vs Multivariate vs Split URL Testing: Which to Use

A/B, A/B/n, split URL and multivariate tests answer different questions. If you have one isolated change to validate, use A/B. If you have several competing ideas, A/B/n. If the variation already exists on a page with its own URL, or the redesign is too big to swap in with JavaScript, split URL. If you need to know how two or more changes interact with each other, and you have traffic to spare, multivariate (MVT). The most expensive confusion is running an MVT that your traffic cannot sustain: the test never closes, and you lose months without an answer.

Type	What changes	Number of variations	Traffic required	What it reveals	Best for
A/B	One isolated change, same URL	2 (control plus 1)	Low to medium	Whether the change moves the metric	The default case, any site
A/B/n	Several competing ideas, same URL	3 or more	Medium to high (split across all)	Which variation wins	Several good hypotheses and traffic to sustain them
Split URL	A whole page, its own URL	2 or more pages	Low to medium	Whether the new page beats the old one	Large redesigns, a variation that already exists elsewhere
Multivariate (MVT)	Combinations of several elements	Product of the variants (cells)	High, grows by multiplication	The effect of each element and the interaction between them	High traffic, elements expected to interact

The rest of this article goes deeper into each row, starting with the point most people get wrong: why multivariate testing demands so much traffic, and how to calculate that for your own case.

A/B, the default case

An A/B test compares the control (A) against a single variation (B), on the same URL, with traffic split by random assignment. It is the simplest design there is, which is exactly why it is the right starting point for practically any test.

The strength of A/B is causal clarity. Since only one thing changes between A and B, any difference in conversion can only come from that change (net of chance, which the test statistics measure). There is no ambiguity about what caused what.

The limitation is the same coin from the other side: you only learn about one change at a time. If you have five good ideas for the page hero, a plain A/B forces you to pick one, test it, wait for the result, and only then test the next. For cases like that there is A/B/n and, more rarely, multivariate.

A/B/n and split URL: where each one fits

A/B/n is A/B stretched to more than one variation: control A against B, C, D and so on, all running at the same time, all on the same URL. It makes sense when you have several competing hypotheses about the same change (three different headlines, for example) and enough traffic to split among them without starving each arm. The tradeoff is direct: every extra variation shrinks the slice of traffic left for the others, so the test takes longer to close than a two sided A/B.

Split URL solves a different problem: sometimes the variation is not an element tweak on the same page, it is an entirely new page, with different structure, template or even domain. Forcing that into a DOM swap via JavaScript is fragile and slow. In a split URL test each version lives at its own address, and a redirect (not the browser script) decides which page the visitor sees. It is the right design for full redesigns, for page builder migrations, or when “variation B” already exists: an older landing page you want to compare against the current one, for instance.

Statistically, A/B/n and split URL do not change the mathematics behind the A/B test: they are still comparisons of proportions, one against another. What changes is where the content lives and how many arms divide the traffic.

The three rules that keep a split URL test out of trouble

Split URL is the one design that touches infrastructure search engines care about, so it carries three requirements that classic A/B does not. Google guidance on website testing states them directly:

Rule	What to do	Why it exists
Use a temporary redirect	302, not 301, for as long as the test runs	A 301 tells search engines the move is permanent and transfers the URL identity
Keep a canonical link	rel=canonical on every variation, pointing at the original URL	It consolidates the signal on one address instead of splitting it across test pages
Never treat crawlers differently	Serve the same content to bots and to people	Segmenting by user agent to show something different is cloaking, and it is a policy violation

There is a fourth rule that is operational rather than technical: run the test only for as long as you need. A split URL test left running for months after the answer arrived is a redirect and a duplicate page kept alive with no purpose, and that costs crawl budget and internal clarity for no gain.

What multivariate testing is

MVT tests several variables at the same time, each with two or more variants, across every possible combination between them. Each unique combination is a cell.

A simple example: you want to test two page elements, the headline (2 variants: A and B) and the button color (2 variants: green and amber). That produces 2 by 2 = 4 combinations, that is, 4 cells:

The output of an MVT is not just “which cell converted best”. It is the statistical decomposition of how much each variable contributed on its own (the main effect of the headline, the main effect of the button color) and how much they contributed together, beyond what each would do in isolation (the interaction effect). That last part, the interaction, is the one only a multivariate test can really measure.

The interaction effect: the real reason MVT exists

If page variables never interacted, multivariate testing would just be an expensive way to run several A/B tests at once, with no real gain over testing each element in sequence. The reason MVT exists is the possibility of interaction: when the effect of one change flips sign or changes size depending on what is present in the other.

A numeric example, with hypothetical conversion rates to illustrate the mechanism (these are not real measurements):

Combination	Conversion rate
Headline A plus green button	4.0%
Headline A plus amber button	5.5%
Headline B plus green button	6.0%
Headline B plus amber button	4.5%

Look at what happens: with the green button, headline B beats headline A (6.0% against 4.0%). But with the amber button, headline A beats headline B (5.5% against 4.5%). The “best headline” depends on the button next to it. That inversion is the interaction.

Now imagine you had tested headline and button in sequence, as two separate A/B tests. You hold the button that is already live (say, green), test headline A against B, and B wins. You ship B. Weeks later you test button color with headline B held fixed, and amber loses badly, because headline B plus amber is the worst row in the table. You would never learn that headline A plus amber was in fact the second best combination available. Sequential A/B cannot see this: it always tests one variable with the rest of the page frozen, and the interaction only appears when both vary at the same time.

That does not mean interactions are common. Between page elements that do not compete for the same attention or the same argument (the color of a trust badge in the footer and the headline copy at the top, for instance), interaction tends to be small or nonexistent, and sequential A/B works fine. Strong interactions tend to show up between elements that reinforce or contradict the same message, such as headline and CTA, or price and guarantee.

The traffic problem of multivariate testing

Here is the point most marketing teams underestimate: every cell of a multivariate test is, in practice, a whole A/B test, and it needs the same minimum sample a plain A/B would need.

The combination rule is multiplicative: take the number of variants of each element and multiply them all together.

Two elements with 2 variants each: 2 x 2 = 4 cells. Three elements with 2 variants each: 2 x 2 x 2 = 8 cells. A fourth element joins and the number doubles again. Total sample is not the sample of one cell divided by the number of cells: it is the sample of one cell multiplied by the number of cells, because each cell on its own still needs the same volume any A/B would need to detect the same effect at the same confidence.

Take a worked example with real numbers. Start from a common scenario: a page converting at 5%, and you want to detect a 10% relative improvement (from 5% to 5.5%), at 95% confidence and 80% power, the market default. The sample size per variation (that is, per cell) for that scenario is 31,234 visitors.

For a plain A/B (2 variations), that means a total of about 62,468 visitors before the test closes. Now apply the same per cell sample to a multivariate design, at three traffic levels:

Design	Cells	Total sample	At 10,000/week	At 25,000/week	At 50,000/week
Plain A/B (2 variations)	2	62,468	44 days	18 days	9 days
Multivariate 2x2 (2 elements, 2 variants each)	4	124,936	88 days	35 days	18 days
Multivariate 2x2x2 (3 elements, 2 variants each)	8	249,872	175 days	70 days	35 days
Multivariate 2x2x2x2 (4 elements, 2 variants each)	16	499,744	350 days	140 days	70 days

(Values computed with the same statistical engine as the calculator on this blog, for a 5% baseline, 10% relative MDE, 95% confidence, 80% power, two sided.)

Total sample grows in proportion to the number of cells, and that is where the arithmetic misleads people: the plain A/B already uses 2 cells (2 x 31,234) and the 2x2 MVT uses 4 (4 x 31,234), which is double the A/B and four times the sample of a single cell. With 8 cells the total passes 249,000 visitors, four times a normal A/B and eight times one cell. Read the table by column instead of by row and the practical rule falls out: a multivariate design becomes viable roughly when your weekly traffic is high enough that a plain A/B would close in under two weeks. Below that, the smallest MVT that exists already pushes past a quarter.

Full factorial against partial factorial

There is a partial way out of the traffic problem: instead of testing every combination (the full factorial design described so far), test an orthogonal subset of them. This design, popularized by the engineer Genichi Taguchi in manufacturing, picks a smaller number of combinations such that the main effect of each element can still be estimated without walking the whole grid.

Optimizely documentation for Web Experimentation puts the tradeoff plainly: the drawback of full factorial is that “it requires higher levels of traffic to generate a significant result”, while with partial factorial “you can test far fewer combinations than full factorial while still enjoying many of the same benefits”. The mechanism there is traffic allocation at the combination level, which lets you set specific combinations to zero traffic instead of splitting evenly across every cell.

Honesty matters here: the partial design saves traffic but pays in visibility. It isolates main effects well (the effect of the headline alone, the effect of the button alone), and it loses the ability to detect higher order interactions, exactly the kind of effect that motivated multivariate testing in the first place. If your reason for testing several variables at once is to hunt for interaction, a partial design that sacrifices interaction to save traffic is, in a sense, throwing away the reason for the test.

In practice, for most marketing and product teams, the partial design is rare outside large CRO operations with a dedicated statistician. It is simpler, and almost always cheaper in implementation effort, to run full factorial only when the traffic sustains it, or to swap the multivariate test for a well prioritized sequence of A/B tests.

Decision tree: which one to choose

Boil the decision down to four questions: how much monthly traffic you have, how many changes you want to test, whether they are expected to interact, and how much calendar time you have.

When multivariate is not worth it

The honest verdict: for most mid traffic sites, multivariate testing is not worth it. Not because the method is bad, but because the cost in time almost always outweighs the gain in information.

Consider the worked example above: a simple 2x2 multivariate test, the smallest MVT design that exists, already doubles the duration relative to a normal A/B on the same traffic. A 2x2x2, still modest (only 3 elements, 2 variants each), requires four times the total sample of a plain A/B, which at 10,000 visitors per week means 175 days. Sites receiving a few hundred or a few thousand visitors per week simply do not produce that volume in a reasonable window, and the test becomes a months long project that never closes with confidence.

In practice, most of the “interactions” marketing teams want to capture (headline with button, image with social proof) can be approximated well enough by testing the combinations most likely to win in sequence, through A/B/n, on a fraction of the traffic. You lose the statistical guarantee of having mapped every possible interaction, and you gain an answer in weeks rather than months, which almost always pays.

Multivariate pays off when three things hold at once: high traffic (enough to fill every cell within an acceptable window), a concrete reason to expect strong interaction between the elements, and a calendar that sustains a test running longer than a normal A/B. Outside that combination, prefer plain A/B or A/B/n.

Sizing the sample per cell in practice

The calculator below computes the sample size of one variation, that is, of one cell. Set your current conversion rate, the minimum effect you want to detect and the weekly traffic available, for your real case:

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

To turn that number into a multivariate budget, the step is always the same: take the per cell sample the calculator returned and multiply it by the number of cells in your design (the product of the variants of each element, as shown in the worked example above). If the result runs past a few months at your current traffic, that is already a signal to reconsider the design, either by cutting the number of elements tested or by falling back to a sequence of A/B/n tests.

Make this automatic with Donnu

Choosing between A/B, A/B/n, split URL and multivariate is the first decision that separates a test that closes from one that drags on without an answer. And after choosing, you still have to size the right sample so you neither declare a winner too early nor leave the test running past its usefulness. Donnu handles that second part: you define the hypothesis, Donnu sizes the test against your real traffic, collects with a light snippet that never blocks the page, and returns the verdict with the 95% confidence interval up front, without declaring a winner before the variation accumulates at least 200 visitors and 7 days live.

Start a free 14 day trial and set up your first experiment with the right test type from the beginning. For the statistical foundation behind any of these designs, see A/B testing statistical significance and how to run an A/B test.

References

• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
• Optimizely Support. Multivariate tests for Optimizely Web Experimentation. support.optimizely.com.
• Google Search Central. A/B testing best practices for Search. developers.google.com/search/docs/crawling-indexing/website-testing.
• Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.

Read also:

• How to run an A/B test: the complete step by step guide
• A/B testing statistical significance: the complete guide
• Common A/B testing mistakes and validity threats
• The peeking problem in A/B testing
• Leia em português