Interleaving: Ranking Evaluation on Far Less Traffic

Interleaving is an evaluation method where two rankings are merged into a single list, shown to the same user on the same query, and a click becomes a vote for the ranker that picked that result. That turns a comparison between groups of people into a paired comparison inside the query, and the sensitivity gain is enormous. In this guide worked example, the exact same 71,429 searches return a p-value of 0.097060 in a classic CTR A/B test, meaning nothing, and a p-value of 4.3309 times 10 to the minus 8 under interleaving. This guide covers the two merge algorithms, the sign test that closes the math, the sensitivity gains published by Yahoo, Bing and Airbnb, the real cases where the method points the wrong way, and how to decide whether it fits your product. It is part of our complete A/B testing guide and the extreme case of the topic covered in randomization unit.

The problem: comparing two lists is harder than it looks

You have two rankers for your site search. Ranker B is a new ordering, and the question is whether it is better.

The reflex is to set up an A/B test: half the users see ordering A, half see B, and you compare click-through rate. The problem is that natural variance between queries swamps the signal. A search for a product code has sky-high CTR, a vague category search has low CTR, and the difference between two decent rankers is small next to that dispersion. You are trying to hear a whisper inside a stadium.

Interleaving solves this by changing the question the user answers. Instead of asking “how much do you click on this list?”, it implicitly asks “between these two sets of results right in front of you, which do you prefer?”. The answer arrives in the same instant, from the same user, about the same search intent, and that is what kills the variance.

Chapelle, Joachims, Radlinski and Yue conjecture two reasons for the method sensitivity: it is a paired test, paired on both queries and users, and it directly elicits a preference between two alternatives rather than inferring quality from an absolute number.

The two merge algorithms

Balanced Interleaving

It interleaves the two lists, removing duplicates, and at the end of the query defines k as the smallest cutoff containing every document that was shown. You count how many clicks landed in the top k of A and how many in the top k of B. More clicks for A means a preference for A.

The method has a known bias the authors themselves demonstrate. With A equal to (a, b, c, d) and B equal to (b, c, d, a), the merge produces either (a, b, c, d) or (b, a, c, d) depending on the tie-breaking coin toss, and a user clicking uniformly at random would produce a preference for B more often than for A. That happens because every document except a is ranked higher by B.

Team-Draft Interleaving

It was built to fix exactly that, and the authors analogy is picking teams for a friendly sports match. The two rankers are captains. In each round, a coin flip decides who picks first, and each captain takes their most preferred still-available player, adding it to their team and appending it to the merged list.

The click is credited to the team that picked the clicked result. If team A gets more clicks than B on the query, the query counts as a win for A; fewer clicks, a win for B; the same count, a tie. Queries with no clicks at all are discarded.

The property this guarantees is the one Balanced lacked: a randomly clicking user produces, in expectation, as many preferences for A as for B.

The statistic and the test

The authors define a single summary of the comparison:

To decide whether the preference is significant, the simplest case is the binomial sign test over queries with a preference, against the hypothesis that each ranker wins half the time. Queries without clicks are discarded first.

Notice what this construction deliberately throws away: the number of clicks, the position of the click, time on page. All of it collapses into one binary vote. It looks wasteful and it is precisely the opposite: it discards the source of variance and keeps only the comparative signal. The same reasoning appears in permutation tests and ratio metrics, in different clothing.

The worked example: the same traffic, two opposite reads

Scenario. Site search on a marketplace, 40,000 searches per week, average CTR of 28.00 percent. Ranker B reorders results using a new availability signal. The hypothesis is that it is better, but the expected effect is small, around 2 percent relative on CTR.

First, what measuring that through A/B costs. Put a 28.00 percent baseline, a 2 percent relative MDE, 95 percent confidence and 80 percent power into the sample size calculator:

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

The answer is 101,526 searches per arm, that is, 203,052 searches total, which comes to 36 days at 40,000 searches per week. Five weeks of window to decide on one reordering, and it is only one candidate among several in the queue.

Now interleaving. The experiment runs for 13 days with all search traffic going through the merged list, which is 71,429 searches. Of those, 20,000 had at least one click and enter the math. The outcome:

query outcome	count
wins for ranker B	6,300
wins for ranker A	5,700
ties (clicked, but teams tied)	8,000
total queries with a click	20,000
queries with a preference (excluding ties)	12,000

Applying the statistic: delta equals (6,300 plus 4,000) divided by 20,000, minus 0.5, that is, plus 1.5000 percent in favor of B.

The sign test over the 12,000 queries with a preference: the observed proportion is 0.525000, z is 5.4772, and the p-value is 4.3309 times 10 to the minus 8. This is not a borderline result, it is a closed decision.

The fair comparison. Now take exactly the same 71,429 searches and spend them on a classic A/B test, split into 35,714 in control and 35,715 in the variant. With a CTR of 28.0002 percent in control (10,000 clicks) and 28.5594 percent in the variant (10,200 clicks), paste it into the calculator:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

The screen shows 28.00 percent against 28.56 percent, plus 2.0 percent relative, a p-value of 0.0971 and a verdict that significance has not been reached. In other words: no result. Without the rounding, the absolute difference is plus 0.5592 percentage points, z is 1.6593, the p-value is 0.097060, and the confidence interval runs from minus 0.1013 to plus 1.2197 percentage points, crossing zero.

method	data consumed	result	verdict
CTR A/B test	71,429 searches (35,714 and 35,715 per arm)	z 1.6593, p-value 0.097060	inconclusive
interleaving (Team-Draft)	the same 71,429 searches	z 5.4772, p-value 0.000000043	B wins, comfortably

Those two results come from the same traffic, over the same period, about the same difference between rankers. The entire difference lies in the comparison design, not in the amount of data.

How much traffic interleaving needs, by preference size

Interleaving has no fixed sample size: it depends on how large a delta the ranker pair produces. At 95 percent confidence, 80 percent power, two-sided, and assuming 60 percent of clicked queries express a preference (the rest tie):

delta	queries with a preference	queries with a click	searches (28% CTR)	days at 40,000/week	times less data than A/B
1.0%	7,062	11,770	42,036	8	4.83
1.5%	3,138	5,230	18,679	4	10.87
2.0%	1,764	2,940	10,500	2	19.34
3.0%	783	1,305	4,661	1	43.56
5.0%	281	469	1,675	1	121.23

The last column compares against the 203,052 searches the CTR A/B test requires for a 2 percent relative effect. Two readings come out of it:

• The multiplier is not a constant. It ranges from about 5 times to over 120 times within this one scenario, and it depends entirely on how much paired signal the ranker pair produces.
• The relationship between delta and a CTR difference is not fixed. That is why published numbers vary so much across companies, and why no article can promise you “ten times” before you run it.

There is also a design detail that usually goes unnoticed, and which the authors record: for the same number of queries on the axis, an absolute comparison uses twice as much data as an interleaved comparison, because it requires two non-comparative buckets running at once.

The published numbers, and what they actually say

source	context	reported gain
Chapelle, Joachims, Radlinski and Yue (2012)	Yahoo, six ranker pairs, against the best absolute metric (Clicks@1)	data ratios of 379.1 · 193.1 · 17.8 · 270.7 · 18.9 · 0.69
same authors, time-based analysis	Yahoo, pair BY against DY	interleaving takes about 1 hour to reach inconsistency under 0.01; non-comparative buckets take slightly more than one day; ratio of about 22
same authors, against human judgments	Bing, a set of about 12,000 judged queries	one expert-judged query is worth about ten clicked queries under interleaving
Zhang and co-authors, Airbnb search (KDD 2025)	competitive pair Team Drafting, over a hundred experiments	roughly 50 times speedup relative to A/B testing
same authors, citing Bi and co-authors (Amazon Search, 2022)	balanced interleaving	60 times, on a corpus of 10 comparisons

Three honest observations about that table.

First, the 0.69 matters as much as the 379.1. On one of the six Yahoo pairs, interleaving needed more data than the absolute metric. The authors say exactly that: in all the others interleaving was much more sensitive, significantly reducing the number of queries needed by over an order of magnitude. But there was one exception, and it is in the paper.

Second, those numbers come from empirical resampling over real ranker pairs, not from a formula. They describe what happened in those systems, with those queries.

Third, the judged-query row is the most underrated. It quantifies the extent to which clicks replace human raters, and shows that an evaluation using ten thousand expert-judged queries can be matched by a hundred thousand clicked queries, which appear on their own within a few hours of traffic. The authors further record that, breaking the analysis down from queries to documents, one judged query and document pair appears to provide as much evaluation sensitivity as one or two queries with clicks.

Where interleaving gets it wrong, with documented cases

Sensitivity is not accuracy. A method can be extremely precise and point the wrong way, and interleaving has such cases, all published.

An ambiguous query with unbalanced intents. The 2012 authors construct the case: 49 percent of users have intent A, satisfied by document a; 49 percent have intent B, satisfied by b; 2 percent have intent C, satisfied by c. Ranking R1, which starts with (a, b), satisfies 98 percent of users with its top two results. Ranking R2, which starts with (b, c), satisfies 51 percent. And still, assuming each user clicks only the first relevant result they meet, R2 wins under Team-Draft in 51 percent of comparisons. The better ranking loses.

Clicks measure expectation, judgments measure the landing page. The click happens on the title and snippet, before the user sees the content. The authors give real examples of that divergence: on the query “travel insurance”, small insurers focused on the product are more relevant by the rater standard, but users click well-known brands. On “adobe”, the company home page is usually judged most relevant, yet few users issuing that query want the home page rather than the PDF reader download. On “bank of america”, users prefer the online banking page while editors tend to prefer the company home page.

Set-level optimization. The Airbnb search team documents the case most useful for product teams. A treatment ranker was built to optimize a secondary objective while staying neutral on conversion, reordering listings that had been sorted by booking probability. In the A/B test, conversion did stay neutral. In interleaving, the result was significantly negative: placed side by side in the same list, the control listings looked more attractive, and users migrated to them. The merge changed what was being evaluated.

The same authors list two further limits in their product: when the search list also feeds another surface, such as the map view, blending risks disrupting the experience; and building a semantically meaningful metric for a continuous outcome such as revenue is not straightforward.

Where it fits in the process, and where it does not

The use both companies describe is the same, and it is not replacement: interleaving is a filter, A/B is the business verdict. At Airbnb, across 29 interleaving and A/B pairs, the two agreed directionally 82 percent of the time, with a correlation coefficient of 0.6 between point estimates. Good agreement, far from perfect, and the disagreements are informative rather than noise.

question	who answers it
which of these 8 ranking candidates deserves an A/B window?	interleaving
does the chosen candidate increase revenue?	A/B test
does the candidate hurt any guardrail metric?	A/B test
does the gain persist after 6 weeks?	A/B test with a holdout
is the difference between two rankers perceptible to users?	interleaving

This changes the economics of the entire ranking program, and it is the same argument as experimentation velocity: if every candidate needs five weeks of A/B, you test ten per year; if eight candidates pass through a filter lasting a few days and only the winner occupies the A/B window, the number of ideas evaluated per year changes tier without any traffic increase.

The practical prerequisite is steep. Interleaving requires the ranking system to run two rankers per query, merge the lists with a per-round coin flip, stamp the origin of every result, and log the click with that stamp. That is engineering inside search, not configuration in an A/B testing tool. No page-level CRO tool does this, and that is fine: they are different layers of the product.

How to apply this in practice

1. Confirm your case is an ordering of comparable items. Site search, recommendations, a feed. If it is page layout, the method does not apply.
2. Use Team-Draft, not Balanced. The Balanced bias with near-identical lists is known and demonstrated, and near-identical lists are the common case when tuning a ranker.
3. Flip the coin for who picks first in each round, not once per query. That per-round randomization is what guarantees neutrality against random clicking.
4. Discard queries without clicks before counting. They carry no preference information and only dilute the math.
5. Record ties separately. The tie proportion is a diagnostic: too many ties usually means the two rankers are nearly identical at the top.
6. Do not convert delta into revenue. It measures revealed preference, not business value. A/B is what bridges that, and the boundary is the same one discussed in surrogate metrics.
7. Use it as a queue filter. Run several candidates through interleaving and send only the winner to A/B, which is the pattern described by the Yahoo, Bing and Airbnb teams.
8. Be suspicious when interleaving and A/B disagree, in both directions. According to Airbnb, disagreement happened on roughly 18 percent of pairs, and their case showed the disagreement had a real explanation.

Common mistakes

• Counting clicks instead of queries. A query is one vote, whether it had 1 or 5 clicks. Counting clicks reintroduces the variance the method exists to eliminate.
• Applying the sign test with ties included. Ties enter the delta, but not the denominator of the sign test.
• Running interleaving when the two rankers return very different sets. The merge becomes a list neither ranker would produce, and the user experience genuinely degrades.
• Reading delta as a CTR difference. They are different scales, and there is no fixed conversion between them.
• Concluding A/B testing is obsolete. It answers the business question, which interleaving does not.
• Forgetting users are shared across experiments. Merging lists from several experiments on the same query creates interference; Airbnb solves this with separate traffic lanes per experiment.
• Promising another company multiplier. Yahoo’s 379.1, Airbnb’s 50 and Amazon’s 60 belong to those systems, and the sizing table above shows the multiplier varies by an order of magnitude inside a single scenario.

Make this automatic with Donnu

Let us be direct: interleaving is not something an A/B testing tool delivers, ours included. It lives inside the ranking system, because it requires running two rankers per query and stamping the origin of every result before the page is assembled. Any vendor promising “interleaving via a site script” is describing something else.

What does belong to the experimentation tool is the part that usually gets lost: the record of which candidate was filtered out by which read, and with which declared primary metric. Donnu stores experiment configuration at creation time, with the declared primary metric and guardrails, and keeps frozen per-experiment history. In a two-stage flow, that is what lets you answer, months later, why that specific ranker won the A/B window over another.

The practical recommendation in this guide, for teams with site search that are nowhere near this yet: start by measuring the proportion of queries that get a click and the CTR dispersion across queries. Those two numbers tell you, before any engineering, whether your volume can support a paired evaluation and how much you are losing today by comparing averages. The sample size calculator gives you the other side of the math, the A/B test you run today.

References

• Chapelle, O., Joachims, T., Radlinski, F. and Yue, Y. Large-Scale Validation and Analysis of Interleaved Search Evaluation. ACM Transactions on Information Systems, volume 30, number 1, article 6, February 2012. Source of the description of the Balanced and Team-Draft methods, including the analogy of captains taking turns picking players, the per-round coin flip for who picks first, and crediting the click to the team that selected the result; of the delta statistic defined as wins for A plus half the ties over the total, minus 0.5, with the example of 40, 30 and 30 percent yielding 5 percent; of the use of the binomial sign test; of the Balanced bias with lists (a, b, c, d) and (b, c, d, a), where a randomly clicking user produces more preferences for B; of the ambiguous query example with 49, 49 and 2 percent intents where R2 wins 51 percent of comparisons despite R1 satisfying 98 percent of users; of the data ratios of 379.1, 193.1, 17.8, 270.7, 18.9 and 0.69 against Clicks@1 on the six Yahoo pairs; of the time-based analysis where interleaving takes about one hour and the non-comparative buckets slightly more than a day, a ratio of about 22; of the finding that one judged query is worth about ten clicked queries, from a set of about 12,000 judged queries at Bing, and that one judged query and document pair is worth one or two queries with clicks; of the observation that an absolute comparison uses twice as much data because it requires two non-comparative buckets; of the conjecture that sensitivity comes from the test being paired on queries and users and from directly eliciting preference; and of the click and judgment divergence examples on the queries “travel insurance”, “adobe” and “bank of america”. cs.cornell.edu.
• Zhang, Q. and co-authors. Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking. KDD 2025, Toronto. Source of the use of competitive pair Team Drafting in Airbnb search, with over a hundred experiments conducted; of the validation across 29 interleaving and A/B test pairs, with 82 percent directional alignment and a correlation coefficient of 0.6; of the roughly 50 times speedup relative to A/B testing; of the reference to Bi and co-authors at Amazon Search, who reported 60 times on a corpus of size 10; of the case study where a ranker optimizing a secondary objective stayed neutral on conversion in the A/B test and came out significantly negative in interleaving, because side by side the control listings looked more attractive; and of the three stated limits of the method, which are set-level optimization, lists that feed other surfaces such as the map view, and the difficulty of building a meaningful metric for continuous outcomes such as revenue. arxiv.org.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014, New York. Used here as a counterpoint on why absolute metrics require far more data and why spectacular published results deserve skepticism, in line with the 0.69 exception recorded in the interleaving study. exp-platform.com.

Read next: Randomization unit · Permutation tests · Ratio metrics · Surrogate metrics · Experimentation velocity · Sample size calculator · Leia em português