Multi-Armed Bandits vs A/B Testing: The Complete Guide
Multi-armed bandits vs A/B testing: when adaptive traffic allocation wins, Thompson Sampling explained, and a real numeric cost comparison.
📚 This article is part of the guide A/B Testing Statistical Significance Guide.
A multi-armed bandit is an algorithm that reallocates traffic toward the best-performing variation while a test is still running, instead of holding a fixed split until a predetermined sample size is reached, the way a classic A/B test does. The trade-off is real in both directions: a bandit reduces the conversions you lose while you are still learning, but it does not produce a p-value or a classic statistical-significance verdict at the end. An A/B test does the opposite: it holds a fixed split so the result can be defended formally, at the cost of continuing to show the losing variation until the test is over.
Anyone who has run enough tests eventually hits the same frustration: while your A/B test is still running, half your traffic keeps seeing the variation that, bit by bit, is becoming clearly the worse one. That is the price you pay for testing rigorously, an “opportunity cost” for buying a trustworthy answer. There is an entire family of algorithms built specifically to shrink that price: multi-armed bandits.
This guide explains what a bandit is, how it decides where to send each visitor, when it beats a classic A/B test (and when it does not), and it walks through a real numeric comparison, computed with the same statistics engine that runs the calculators on this blog, of what each approach costs in the same scenario. At the end, a neutral table compares what VWO, Optimizely, Kameleoon, and other vendors actually publish about bandits, with a source for every claim.
What is a multi-armed bandit
The name comes from an old casino image: a “one-armed bandit” is the classic slot machine, with a single lever. A multi-armed bandit is the decision problem of someone standing in front of several of these machines at once, each with an unknown, possibly different payout rate. On every turn, you pick a machine, pull the lever, and observe the result. The goal is simple to state and hard to solve well: maximize total winnings across all turns, knowing that the more you play one machine, the more you learn about it, but also the more you give up playing the others.
Swap “machine” for “page variation” and “turn” for “visitor,” and you have the exact problem a bandit solves in an online test. Each variation (A, B, C…) is an arm. Each visitor who arrives is a turn. The “payout rate” of each arm is that variation’s real conversion rate, which you don’t know in advance and only estimate with more precision as traffic arrives.
The shift in posture relative to an A/B test is subtle, but it changes everything downstream: an A/B test treats data collection and decision-making as two separate stages, first you gather evidence with a fixed split, then you decide. A bandit treats collection and decision as the same thing, happening in real time, turn by turn.
Bandit vs A/B testing: the core difference
A classic A/B test is a fixed-horizon experiment: you decide, before starting, the traffic split (usually 50/50) and the sample size you need, run until you get there, and only then declare a winner. The split doesn’t move an inch between the first visitor and the last, even if it becomes obvious halfway through that one variation is losing badly.
A bandit reallocates traffic continuously. It starts near an equal split (because, at the outset, it has no idea which arm is better) and gradually shifts more visitors toward the arm that appears to be winning, while reducing, without zeroing out, traffic to the arms that appear to be losing. Every new conversion updates the estimate for that arm, and the allocation adjusts again.
That creates a well-defined trade-off, and it is the heart of this entire guide:
- What a bandit gains: less “opportunity cost” during the test itself. Because traffic migrates toward the winner before the test ends, you lose fewer possible conversions while you’re still learning.
- What a bandit gives up: it does not produce a p-value, or classic statistical significance, at the end. Optimizely documents this directly: multi-armed bandit optimizations do not generate statistical significance and do not use a formal control or baseline; what you receive is a performance lift over an equal split, not a statistical verdict with an attached confidence level.
Put differently: A/B optimizes for learning rigorously; bandit optimizes for winning while learning. Neither is “better” in an absolute sense, they solve different problems, and picking the wrong one for your situation is expensive on both sides: running an A/B test when the cost of the losing variation is high wastes conversions for no reason; running a bandit when you need to formally defend a decision leaves you without the number that backs the defense.
Exploration vs exploitation, in practice
The technical name for the trade-off every bandit resolves is the exploration/exploitation trade-off. Exploring means spending traffic on arms you are still uncertain about, hoping to discover that one of them is actually the best. Exploiting means concentrating traffic on the arm that, given the information available so far, looks best.
An algorithm that only explores never converges, it keeps playing blind forever and wastes conversions testing poor arms indefinitely. An algorithm that only exploits (always picks the arm that looks best based on the earliest data) risks locking in on the wrong choice too soon, because early data from any arm is noisy: a genuinely better arm can look worse over the first few dozen visitors purely by chance, and a purely greedy algorithm would never give it a second look after that. The three classic algorithms resolve this balance in different ways.
Epsilon-greedy: the simplest to explain
Epsilon-greedy reserves a small, fixed slice of traffic, called epsilon (ε, say 10%), for pure random exploration across all arms, and sends the rest (1-ε) to whichever arm currently looks best. It is easy to implement and easy to explain, but it has a structural flaw: it explores the same way forever, spending the same 10% on clearly bad arms after thousands of visitors that it spent on day one, when the uncertainty was genuine. Variants of the method (like decaying epsilon, which shrinks ε over time) try to patch this by hand, but they remain a manual adjustment rather than an automatic response to each arm’s actual uncertainty.
UCB (Upper Confidence Bound): optimism in the face of uncertainty
UCB swaps random exploration for a more elegant rule: for every arm, it computes the estimated conversion rate plus a bonus for optimism, proportional to the uncertainty around that arm (less data, bigger bonus), and always picks the arm with the highest combined value. A lightly tested arm gets an automatic “benefit of the doubt” that makes it attractive even with a middling estimated rate, which pushes exploration exactly toward where information is missing, with no fixed parameter like ε. As more data arrives, each arm’s bonus shrinks and the choice converges naturally toward the actual best one.
Thompson Sampling: the bayesian production standard
Thompson Sampling is the algorithm most commonly seen in production when the underlying system is bayesian, exactly the ground Donnu already operates on. The idea, explained without formulas: for every arm, you keep a probability distribution, how likely each possible conversion rate is for that arm, given what has been observed so far, typically a Beta distribution when the metric is a conversion rate. For every new visitor, the algorithm draws a random value from each arm’s distribution and sends the visitor to whichever arm drew the highest value. It then updates that arm’s distribution with the observed outcome (converted or not), narrowing the curve and making it more precise.
The trick is that this random sampling already embeds the right balance between exploring and exploiting, with no manual parameter like epsilon-greedy’s ε. An arm with little data has a wide distribution (lots of uncertainty), so the draw sometimes pulls a high value from it even if its average looks middling, and it ends up receiving exploration traffic organically. An arm with lots of data and a clearly poor rate has a narrow, low distribution, so it almost never draws a value high enough to be chosen, which reduces the traffic it gets without zeroing it out entirely.
This is also why Thompson Sampling fits so naturally into a product that is already bayesian by default: the belief distribution over each arm is exactly the same kind of object a bayesian A/B engine already computes to answer “what is the probability B beats A.” You can see that same posterior math at work in the live calculator below, the one that already powers Donnu’s bayesian read of a two-variant test:
Beta-Binomial model with a uniform Beta(1,1) prior and a 95% credible interval. Deterministic calculation, updates live.
Bandit and bayesian A/B testing share the same underlying math; the difference is what you do with it: decide traffic in real time, or report a probability at the end.
When to use a bandit instead of A/B (and when not to)
The right question isn’t “which is better,” it’s “what does this specific decision need.” A bandit tends to be worth more when:
- Showing the losing variation is expensive and immediate. Testing prices, for example: every visitor who sees the “wrong” price is a potentially lost sale right now, not just a data point. A bandit reduces that bleed faster than a fixed 50/50 A/B test.
- Traffic is very high and the decision is continuous optimization, not a one-off milestone. A homepage headline seeing millions of visits a day, or a recommendation algorithm that needs to keep adapting without stopping, fits a bandit running continuously better than a sequence of A/B tests closed one at a time.
- You have many candidates (A/B/n with dozens of variations) and can’t give every one of them equal traffic. A bandit already allocates less traffic to weak arms early on, which avoids burning sample on variations that are clearly worse.
Classic A/B remains the right call when:
- You need an answer that can be defended formally. A product or business decision that will be cited in a report, presented to an investor, or used to justify a bigger investment usually needs a number with a confidence level attached, not “it improved over an equal split.”
- The change needs to last and be analyzed by segment afterward. If you want to know how the variation performed on mobile vs desktop, by acquisition channel, or by cohort, you need data collected under a stable, comparable allocation, which is exactly what a bandit, by continuously shifting allocation, makes harder.
- The decision is binary and final: switching checkout providers, redesigning the signup flow, or any change where reverting is expensive. You want a clear “does this work, yes or no” verdict, not a continuous optimization.
The trade-off nobody hides
It’s worth repeating plainly, because it’s the part that “automatic optimization” marketing tends to smooth over: a bandit does not produce a declared winner at 95% confidence the way an A/B test does. Optimizely’s own documentation is direct about it, multi-armed bandit optimizations do not generate statistical significance; the algorithm optimizes total conversions by reallocating traffic toward whichever variation is currently performing best, without formally analyzing why one variation beats another, and without using a control or baseline in the classic A/B sense.
This isn’t an implementation gap some vendor will patch in a future release, it’s the nature of the problem a bandit solves. A bandit is an optimization tool (maximize cumulative outcome during the test period); A/B is a causal inference tool (estimate, with a known margin of error, whether a specific change causes a specific improvement). These are different questions, and a good bandit answers the first one well without answering the second.
In practice, this matters the moment you need to defend the decision. “Variation B converted 30% higher and received most of the traffic over the course of the test” is a legitimate sentence for a bandit, but it doesn’t carry the same weight as “B converted 30% higher, with a p-value of 0.003 and a 95% confidence interval between +0.5 and +2.5 percentage points,” the sentence a properly sized A/B test lets you say (the exact numbers our statistical significance guide walks through with the same worked example, control at 210 conversions out of 4,200 visitors, variation at 273 out of 4,200). If your decision culture requires the second kind of sentence, whether for a board, an investor, or just so your own product team can sleep at night, a bandit alone doesn’t deliver that, even though it may well have delivered more conversion along the way.
A worked numeric example
To move past theory, let’s use the same statistics engine that runs the calculators on this blog and compare, with concrete numbers, what a fixed A/B test costs in lost opportunity against an adaptive bandit, in the same scenario.
The scenario: a signup page with 14,000 visitors per week, a current conversion rate of 4%, and a variation B that, in reality (though you don’t know this in advance), converts at 4.6%, a 15% relative lift. You want to detect a minimum effect of 15% relative, at 95% confidence and 80% power, the market standard.
Run these numbers in the calculator below (it’s already pre-filled with this scenario) and check the required sample size and duration:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
The blog’s sample size engine (the same sampleSizePerVariant function in src/lib/stats.ts, with no changes to the frozen math) returns 17,943 visitors per variant, 35,886 in total, running for 18 days at that weekly traffic level.
The cost of a fixed 50/50 A/B test
For all 18 days, half the traffic (17,943 visitors) keeps seeing variation A, even though, well before the end, the data is already pointing to B. Working out expected conversions under this design:
- Expected conversions from A: 17,943 x 4% ≈ 718.
- Expected conversions from B: 17,943 x 4.6% ≈ 825.
- Total expected with a fixed A/B: ≈ 1,543 conversions across the 35,886 visitors.
Now the theoretical ceiling: if every visitor, from the very first one, had gone to B (the actually-better variation, in a hypothetical world where you already knew that with total certainty), the result would be:
- 35,886 x 4.6% ≈ 1,651 conversions.
The difference, ≈ 108 conversions, is the “opportunity cost” that the fixed 50/50 design pays for its own discipline: it keeps half the traffic on the weaker variation from start to finish, because the allocation was decided before any data arrived and doesn’t change until the test ends. This is exactly the cost a bandit tries to shrink.
An adaptive allocation in the same scenario
To illustrate the logic (not to simulate any specific commercial engine exactly, since the real curve depends on the algorithm, the chosen prior, and the observed variance), let’s split the same 18 days and the same 35,886 visitors into three equal phases, with the allocation to B climbing as evidence accumulates:
| Phase | Days | Visitors (approx.) | Allocation to B | Allocation to A |
|---|---|---|---|---|
| 1: still uncertain | 1 to 6 | ~11,962 | 50% | 50% |
| 2: evidence starts to show | 7 to 12 | ~11,962 | 65% | 35% |
| 3: strong evidence, never 100% | 13 to 18 | ~11,962 | 80% | 20% |
Summing both variations across the three phases: A receives about 12,560 visitors and B about 23,326. Expected conversions work out to:
- A: 12,560 x 4% ≈ 502 conversions.
- B: 23,326 x 4.6% ≈ 1,073 conversions.
- Total illustrative bandit result: ≈ 1,575 conversions.
Comparing all three numbers side by side, on the same total traffic:
The central point of this example isn’t the exact number (it changes with the algorithm, the prior, and the real conversion rate), it’s the shape of the result: a well-configured bandit lands systematically between the fixed A/B and the theoretical ceiling, never reaching the ceiling because no serious bandit stops exploring entirely (that would mean betting everything on a read that could still be wrong), but also never falling as far behind as a 50/50 split held from start to finish. And, as shown above, it delivers that gain without producing, at the end, a p-value or classic statistical significance.
Comparison table: classic A/B vs bandit vs contextual bandit
| Criterion | Classic A/B | Bandit (simple) | Contextual bandit |
|---|---|---|---|
| Traffic allocation | Fixed (e.g., 50/50) from start to finish | Changes over time, as evidence accumulates | Changes per visitor, based on their context |
| Produces formal statistical significance | Yes, p-value and confidence interval | No, in the classic sense (Optimizely is explicit about this) | No, in the classic sense |
| Best for | Decisions that need to be defended and documented | Continuous optimization, high cost of showing the losing variation | The “winner” changes by profile (device, channel, behavior) |
| Segment analysis after the test | Straightforward, the allocation was stable | Difficult, the allocation already shifted on its own during the test | It’s the whole point of the method |
| Requires more data/infrastructure | No, it’s the simplest design | A bit more (continuous updating) | Considerably more (a model per context attribute, real-time data) |
| Risk of a wrong decision | Low, if sized and run correctly | Low opportunity cost, but no formal proof of causation | Same as a simple bandit, plus the complexity of maintaining multiple models |
No row in this table is “always the best.” It exists so you can choose based on what your specific decision requires, not on what sounds most modern.
What major tools publish about bandits (a neutral look)
Much of the A/B testing market already offers some flavor of bandit, usually as a feature alongside classic A/B, not as a replacement for it. It’s worth looking at what each vendor publishes about its own approach, with the usual caveat that product specs change and the best place to confirm details is always the latest official documentation.
- VWO (SmartStats). VWO documents its SmartStats engine as bayesian, with a non-informative prior (every possible conversion rate starts equally likely) that gets updated as data arrives, allowing continuous monitoring of a test without the statistical penalty of a fixed frequentist design. VWO’s own help center states that the technique cuts testing time roughly in half compared to the traditional design. The usual hedge applies: a time-reduction figure published by a vendor about its own product should be read as the company’s best case, not as a guaranteed average for every account.
- Optimizely (multi-armed bandit and contextual bandit). Optimizely documents the two formats separately. The simple bandit searches for the single best-performing variation for everybody; the contextual bandit, which the company describes as part of its AI layer (Opal), personalizes the choice using visitor attributes such as device, location, or behavioral history. As already cited, Optimizely’s documentation is clear that bandit optimizations do not generate statistical significance or use a formal control, unlike the same company’s classic experimentation product.
- Kameleoon (CUPED). Kameleoon doesn’t sell bandit as its core differentiator, it sells CUPED (Controlled Experiment Using Pre-Experiment Data), a variance-reduction technique based on the control-variates method: it uses pre-experiment data as a covariate to reduce the noise in the measured metric, without needing more traffic. Kameleoon itself reports up to a 60% reduction in required sample size when the technique is applied well. CUPED and bandit solve different problems (CUPED speeds up a classic A/B test that still produces significance; bandit trades formal significance for continuous optimization), and it’s worth not conflating the two when comparing vendors.
- Spotify Engineering (separating personalization from experimentation). In January 2026, Spotify’s engineering blog published a description of how the company deliberately separates the technical stack that does personalization, which includes contextual bandits, from the stack that runs formal experimentation. The post illustrates a contextual bandit choosing between checkout button variations based on visitor attributes (like cart contents, age, and location), instead of running a traditional A/B test to find “the” single winning button. It also reinforces the central point of this guide: a recommendation algorithm that only optimizes for immediate engagement, always suggesting already-familiar songs, improves short-term metrics but can reduce discovery of new content and hurt long-term satisfaction. The part that matters most here: even with bandits running in production, Spotify still evaluates those systems through formal A/B tests comparing versions of them, exactly the division of labor this guide argues for, continuous optimization on one side, a formal verdict on the other.
| Vendor | Published approach | What’s left unconfirmed without checking directly |
|---|---|---|
| VWO | Bayesian (SmartStats), non-informative prior, sequential testing | The “cuts time in half” figure is vendor-published |
| Optimizely | Simple and contextual MAB (via Opal), explicit documentation that it doesn’t generate significance | Exact internal bandit algorithm details |
| Kameleoon | CUPED (variance reduction, not bandit itself) | Whether the company offers native bandit beyond CUPED, check current docs |
| Spotify Engineering | Contextual bandits in production (personalization), evaluated by separate formal A/B | Implementation details are internal, the public post is high-level |
Donnu positions itself on this map as a native bayesian tool: the same engine that already computes “what is the probability B beats A” for a classic test is the mathematical foundation a bandit like Thompson Sampling needs. Today the product focuses on A/B testing with an honest bayesian read; bandit is the natural extension of that engine, not a generic feature bolted on top.
Automate this with Donnu
You just saw both real sides of this choice: a classic A/B test delivers a verdict that holds up to scrutiny, with a p-value, a confidence interval, and a stable allocation that’s easy to analyze by segment afterward, at the cost of keeping traffic on the losing variation until the end. A bandit reduces that opportunity cost by reallocating traffic in real time, at the cost of not delivering, at the end, formal statistical significance. No tool exempts you from choosing: you still need to decide whether your situation calls for causal-inference rigor or continuous optimization, and the answer changes from test to test.
Donnu already runs today on a native bayesian engine, exactly the mathematical foundation that underlies both worlds: the same calculation of “what is the probability B beats A” that declares an honest winner in an A/B test is also what feeds a bandit like Thompson Sampling when the decision calls for continuous optimization instead of a single verdict. You don’t have to guess which tool is right, Donnu already starts from the statistics both scenarios require.
Read more: the complete A/B testing statistical significance guide, Bayesian A/B testing explained in full, and the peeking problem, why stopping early ruins a test. Prefer Portuguese? Read this guide’s sibling piece: Multi-Armed Bandits x Teste A/B.
Start a free 14-day trial and run your next experiment on a statistical foundation that is bayesian by default.
References
- Optimizely. Run a multi-armed bandit (MAB) optimization in Feature Experimentation and Multi-armed bandit (glossary). docs.developers.optimizely.com and optimizely.com/optimization-glossary/multi-armed-bandit.
- Optimizely. Run Contextual Multi-Armed Bandit optimizations. docs.developers.optimizely.com.
- VWO. Enhanced SmartStats, Bayesian powered sequential testing engine. vwo.com/why-us/technology/statistics; What is SmartStats (time-reduction claim). help.vwo.com; The Frequentists, the Bayesians and the Epistemologists (uninformative prior). vwo.com/stats-blog.
- Kameleoon. What we learned from running 200+ experiments on CUPED. kameleoon.com/blog/cuped.
- Spotify Engineering. Why We Use Separate Tech Stacks for Personalization and Experimentation (January 2026). engineering.atspotify.com.
- Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I. & Wen, Z. A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning, 2018. arxiv.org/abs/1707.02038.
- Auer, P., Cesa-Bianchi, N. & Fischer, P. Finite-time Analysis of the Multiarmed Bandit Problem (the paper that formalizes UCB). Machine Learning, 47, 235-256, 2002.
Frequently asked questions
- What is a multi-armed bandit, in plain terms?
- It is an algorithm that decides, visitor by visitor, which variation to send the next person to, based on what it has learned so far. Instead of fixing a traffic split up front and only analyzing results at the end, like a classic A/B test, a bandit gradually shifts traffic away from variations that look weak and toward the ones that look strong, while the test is still running.
- Does a bandit fully replace A/B testing?
- No. A bandit is a strong fit for continuous optimization, when the cost of showing the losing variation is high and you do not need a formal, defensible verdict. A/B testing remains the right tool when you need an answer that holds up to statistical scrutiny, can be documented, and can be sliced by segment afterward. Many teams run both: A/B to decide what ships, bandit to keep optimizing what has already been validated.
- Can a bandit tell me a variation "won with 95% confidence"?
- Not in the same sense an A/B test can. Bandits optimize a metric over time by reallocating traffic; they do not run the null-hypothesis and p-value machinery that backs a classic significance claim. Optimizely is explicit about this in its own documentation: multi-armed bandit optimizations do not generate statistical significance. What you get at the end is a performance lift over an equal split, not a verdict with an attached confidence level.
- When does a contextual bandit make more sense than a simple bandit?
- When the "winner" changes depending on who is looking at it: device, location, acquisition channel, or behavioral history. A simple bandit searches for one champion variation for everybody; a contextual bandit uses those attributes to pick the best variation for each visitor profile, which costs more data and more operational complexity.
- Is Thompson Sampling better than UCB or epsilon-greedy?
- It depends on the setting, but Thompson Sampling is the algorithm most commonly used in production bayesian systems because its exploration is proportional to genuine uncertainty, with no manual tuning parameter like epsilon-greedy needs. UCB is a close, deterministic alternative with similar theoretical guarantees. Epsilon-greedy is the simplest to explain but explores at a constant rate even after uncertainty has largely resolved, which wastes traffic on clearly losing arms late in the test.
- Does Donnu offer multi-armed bandits?
- Donnu runs today on a native Bayesian engine, which is the same statistical foundation that powers bandits like Thompson Sampling. The current product focuses on A/B testing with an honest Bayesian read; bandit is the natural extension of that same engine for teams that need continuous optimization, and it is the direction the product roadmap points toward.