Thompson Sampling Explained: Bandit Algorithm
Thompson sampling explained: how it draws from each arm's Beta posterior and why it converges faster than epsilon-greedy, with a worked example.
📚 This article is part of the guide A/B Testing Statistical Significance Guide.
Thompson Sampling is the Bayesian algorithm behind most production multi-armed bandits: it is how a bandit decides, visitor by visitor, which variation gets the next click. Instead of fixing a traffic split up front and only analyzing results at the end, the way a classic A/B test does, it draws a random value from the current belief about each arm and sends the visitor to whichever arm drew the highest value, then updates that belief with the outcome. This guide walks through the mechanics behind the draw (the Beta distribution and its conjugate update), works through the math step by step with a three-variation example using numbers computed by the same Bayesian engine that runs the calculators on this blog, and states plainly when the technique earns its keep over a formal A/B test, and when it does not.
The problem Thompson Sampling solves: exploration vs. exploitation
Every multi-armed bandit has to balance two opposing forces. Exploring means spending traffic on arms you are still uncertain about, to learn more. Exploiting means concentrating traffic on the arm that looks best given what you know so far. An algorithm that only explores never converges and wastes conversions indefinitely; an algorithm that only exploits risks locking in on the wrong answer too early, because the first data points from any arm are noisy.
Three classic families answer this problem, each with a different philosophy:
- Epsilon-greedy sets aside a small, fixed slice of traffic (epsilon) for random exploration across all arms, and sends the rest to whichever arm currently looks best. It is simple, but it explores at the same rate forever, spending the same epsilon percent on arms that are already clearly losing after thousands of visitors.
- UCB (Upper Confidence Bound) adds an optimism bonus to each arm’s estimated rate, sized to how uncertain that arm still is, and always picks the highest combined value. The bonus shrinks as more data arrives, so exploration tapers off on its own, with no fixed parameter like epsilon.
- Thompson Sampling replaces the deterministic bonus calculation with a genuinely random draw from each arm’s belief distribution, the subject of this guide.
The key difference between the three shows up in how exploration decays over the course of a test. In epsilon-greedy it stays constant unless someone manually lowers epsilon; in UCB and Thompson Sampling it shrinks on its own, purely as a consequence of more data reducing each arm’s uncertainty:
How Thompson Sampling decides, step by step
The mechanism is simpler than it looks at first, and it needs no manual parameter like epsilon. For every arm in the test, the algorithm keeps a probability distribution representing the current belief about that arm’s true conversion rate, given everything observed so far. When the metric is a binary conversion (converted or did not convert), that belief distribution is a Beta(alpha, beta), the natural conjugate pair of the binomial distribution.
The cycle repeats for every new visitor:
In plain terms, an arm with little data has a wide Beta (high uncertainty), so its draw sometimes lands on a high value even when the average looks middling, and it ends up receiving exploration traffic organically, with nobody reserving a fixed slice for it on purpose. An arm with lots of data and a clearly weak rate has a narrow, low Beta, so it almost never draws a value high enough to win, which reduces the traffic it gets without cutting it off completely.
The conjugate update: from Beta(1,1) to Beta(1 plus conversions, 1 plus non-conversions)
The part that makes Thompson Sampling cheap to compute, and therefore so common in production, is that for a binary conversion metric the belief update has a closed-form formula, with no heavy simulation required. Starting from a uniform Beta(1,1) prior, which says “before any data, every conversion rate between 0% and 100% is equally likely,” the update rule after observing a number of visitors and conversions is:
This is the same formula, with the same Beta(1,1) prior, that already runs underneath the Bayesian calculator on this blog. The reason it is this simple is conjugacy: when the prior is Beta and the observation is binomial (converted or not), the posterior stays Beta, so the update is just adding conversions to alpha and non-conversions to beta. There is no heavy numerical integration or simulation behind this particular step, just a sum.
A worked example: three variations, one round of traffic
To move past theory, let’s run the numbers. Picture a landing page running a test with three variations (A, B, and C) under Thompson Sampling. At the start, with no data yet, all three arms share the same uniform Beta(1,1) prior, a 33% share of traffic each. After a first round of 800 visitors per arm, the observed results were:
| Arm | Visitors | Conversions | Observed rate |
|---|---|---|---|
| A | 800 | 40 | 5.00% |
| B | 800 | 56 | 7.00% |
| C | 800 | 32 | 4.00% |
Applying the update formula, Beta(1+conversions, 1+visitors-conversions), to each arm:
- Arm A: Beta(1+40, 1+800-40) = Beta(41, 761).
- Arm B: Beta(1+56, 1+800-56) = Beta(57, 745).
- Arm C: Beta(1+32, 1+800-32) = Beta(33, 769).
The mean of a Beta(alpha, beta) distribution is alpha divided by (alpha plus beta), and the standard deviation is the square root of alpha times beta, divided by the square of (alpha plus beta) times (alpha plus beta plus 1). Working both out for each arm:
| Arm | Posterior | Posterior mean | Standard deviation |
|---|---|---|---|
| A | Beta(41, 761) | 5.11% | 0.78% |
| B | Beta(57, 745) | 7.11% | 0.91% |
| C | Beta(33, 769) | 4.11% | 0.70% |
Notice the pattern: the further apart the means and the smaller the overlap between curves, the more predictable it becomes which arm wins the draw in the next round, and that mechanism, with no manual parameter anywhere, is exactly what pushes growing traffic toward B from here on. A and C still keep receiving some traffic, though, because their distributions, while lower, still have a tail that occasionally draws a competitive value. That residual exploration is what keeps the algorithm honest in case B’s round happened to be a lucky one.
Check the math on the Bayesian calculator
The posterior of each arm in Thompson Sampling is the same mathematical object a Bayesian A/B testing calculator uses to answer “what is the probability that B beats A.” To compare arm B against arm A from the example above, enter the numbers from the table (A: 800 visitors, 40 conversions; B: 800 visitors, 56 conversions) into the calculator below:
Beta-Binomial model with a uniform Beta(1,1) prior and a 95% credible interval. Deterministic calculation, updates live.
Running those numbers through the same Bayesian engine that powers the calculator (conjugate Beta-Binomial, Beta(1,1) prior, 95% credible interval), the result is: A’s posterior rate at 5.11% (credible interval 3.70% to 6.74%), B’s posterior rate at 7.11% (credible interval 5.43% to 8.98%), a 95.34% probability that B beats A, a relative lift of the means of +39.0%, and an expected loss of just 0.02 percentage points from choosing B, against 2.02 percentage points from choosing A. Running B against C the same way (C: 800 visitors, 32 conversions), the probability that B beats C climbs to 99.57%, consistent with the wider gap between those two curves in the figure above.
That is the connection between the two techniques: a Bayesian A/B test runs this calculation once, at the end of the test, to report a verdict. Thompson Sampling runs the equivalent draw for every visitor, throughout the whole test, to decide traffic in real time. The underlying math is identical, what changes is the frequency and the purpose of the calculation.
Why Thompson Sampling tends to converge faster than epsilon-greedy
“Converges faster” here means the algorithm loses less possible conversion during its own learning process, the accumulated regret. The structural reason for that advantage lies in how each method decides how much to explore at any given moment:
| Criterion | Epsilon-greedy | UCB | Thompson Sampling |
|---|---|---|---|
| How exploration is decided | Fixed slice (epsilon) of traffic, always random across arms | Optimism bonus added to the estimated rate, larger when data is scarce | Random draw from each arm’s belief distribution |
| Does exploration shrink on its own as data grows? | No, epsilon stays constant unless someone lowers it manually | Yes, the bonus shrinks as uncertainty falls | Yes, the distribution narrows as uncertainty falls |
| Needs a manual parameter | Yes, the value of epsilon (and, with decaying epsilon, the decay rate too) | No, the bonus is computed straight from the data | No, the prior is the only knob, and a uniform prior already works well in most cases |
| Where it wastes the most traffic | On arms that are already clearly weak, after lots of data | Less than epsilon-greedy, but the bonus is a deterministic approximation of uncertainty | Tends to waste the least, because each arm’s real variance feeds directly into the draw |
Epsilon-greedy treats uncertainty as a “how long should I explore” problem, solved with a fixed number chosen in advance. Thompson Sampling treats uncertainty as part of each arm’s own distribution, so the amount of exploration adjusts itself automatically: newly arrived arms explore a lot (wide distribution), well-established arms explore little (narrow distribution), with nobody deciding that by hand. That self-regulation, documented in empirical evaluations such as Chapelle and Li (2011), is what typically gives Thompson Sampling lower accumulated regret than epsilon-greedy in the same scenario.
When to use Thompson Sampling instead of a classic A/B test (and when not to)
The right question is not “which technique is more modern,” it is “what does this specific decision require.” Thompson Sampling tends to earn its keep when:
- The cost of showing the losing variation is high and immediate, as with pricing or an offer, where every visitor on the losing side is a potential sale lost right now, not just one more data point for a report.
- The decision is continuous optimization, with no single “end of test” milestone: a homepage headline or a recommendation engine that needs to keep adapting fits a bandit running continuously better than a one-shot test.
- There are many arms (an A/B/n test with several variations) and there is not enough traffic to give every one of them an equal share until the end; Thompson Sampling already reduces traffic to weak arms early on.
A classic A/B test remains the right tool when:
- You need a verdict that holds up to formal scrutiny, with a p-value or a confidence interval, for a report, an investor, or a business decision that will be cited later. Thompson Sampling does not produce that kind of classic statistical significance.
- The change needs to be analyzed by segment afterward (mobile vs. desktop, by acquisition channel), which requires a stable, comparable traffic split, exactly what a bandit makes harder by shifting the allocation continuously.
- The decision is an audit or a documented, fair comparison between exactly two versions, where any deviation from the agreed split needs a recorded justification. An allocation that changes on its own over the course of the test is harder to defend in that setting than a fixed split.
Many mature teams run both in sequence, as covered in our multi-armed bandits vs. A/B testing guide: an A/B test to formally validate that a change works, Thompson Sampling to continuously optimize among variations already validated, once the question has shifted from “does this work?” to “which one works best, always?” When the winner also depends on who is looking, a contextual bandit takes the same Thompson Sampling mechanics a step further, learning a different champion for different visitor segments instead of one overall winner.
Automate This in Donnu
You have just seen the full mechanics behind Thompson Sampling: each arm’s Beta posterior, the conjugate update that adds conversions and non-conversions onto the prior, the draw that decides the next visitor, and why that self-regulating exploration tends to beat a fixed epsilon. Donnu A/B already runs on that same native Bayesian engine today, the same Beta-Binomial math that declares an honest winner in a classic A/B test is the foundation a bandit like Thompson Sampling uses to optimize continuously. You do not need to guess which tool fits, start from the statistics both scenarios already require.
Start a free 14-day trial and run your own Bayesian comparison on real traffic.
Read the full guide to multi-armed bandits vs. A/B testing, the companion piece on contextual bandits explained for when the winner changes by visitor profile, and the Bayesian A/B testing guide for the broader frequentist-vs-Bayesian picture. Leia em português: Thompson Sampling explicado.
References
- Thompson, W. R. On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika, 25(3/4), 285-294, 1933. The original paper that gives the technique its name. doi.org/10.2307/2332286.
- Chapelle, O. & Li, L. An Empirical Evaluation of Thompson Sampling. Advances in Neural Information Processing Systems (NeurIPS), 2011. proceedings.neurips.cc.
- Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I. & Wen, Z. A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning, 2018. arxiv.org/abs/1707.02038.
- Optimizely. Multi-armed bandit (glossary entry). optimizely.com/optimization-glossary/multi-armed-bandit.
- Netflix Technology Blog. Artwork Personalization at Netflix (describes using bandit methods to choose personalized cover art). netflixtechblog.com.
Frequently asked questions
- What is Thompson Sampling, in plain terms?
- Thompson Sampling is a multi-armed bandit algorithm that decides which variation to show each visitor by drawing a random value from the current belief distribution (posterior) of every arm, then routing the visitor to whichever arm drew the highest value. William R. Thompson introduced the idea in a 1933 paper, and it is now the most common Bayesian method used in production bandit systems.
- Why does Thompson Sampling typically converge faster than epsilon-greedy?
- Because the amount of exploration adjusts itself to the real uncertainty of each arm: arms with little data have wide posteriors and get sampled often, which is automatic exploration, while arms with lots of data and a clearly weak rate have narrow posteriors and almost never win the draw. Epsilon-greedy, by contrast, reserves the same fixed slice of traffic for random exploration at all times, even after uncertainty has largely disappeared.
- Do I need to understand Bayesian statistics to use Thompson Sampling in practice?
- Not to run the product day to day: the mechanics, a Beta distribution as belief, an update from observed conversions, a draw, and picking the highest value, are the same math that already powers a Bayesian A/B testing calculator. Understanding the logic helps you interpret why traffic shifts the way it does, but the calculation itself ships ready to use inside the tool.
- When should I NOT use Thompson Sampling instead of a classic A/B test?
- When the decision needs a formal, documentable verdict: a p-value, a confidence interval, or a traffic split stable enough to slice by segment afterward. Thompson Sampling keeps changing the allocation continuously, so it does not produce classic statistical significance and it does not preserve a comparable traffic split from the start of the test to the end.
- Do Thompson Sampling and a Bayesian A/B test use the same math?
- Yes. Both techniques start from the same conjugate Beta-Binomial model: the posterior of each arm, or variation, is Beta(1 plus conversions, 1 plus non-conversions), built from a uniform Beta(1,1) prior. The difference is what you do with that posterior. A Bayesian A/B test reports the final probability that B beats A, once, at the end. Thompson Sampling draws from that same posterior for every visitor to decide traffic in real time.