Multi-Armed Bandits for Pricing Tests (2026)

A pricing test is the one place where the “cost of showing the losing variation” stops being an abstraction: every visitor who sees the wrong price is a potential lost sale or lost margin, right now, not a data point you will interpret later. That single fact is why multi-armed bandits, and Thompson Sampling in particular, get pulled into pricing conversations far more often than into a headline or button-color test. This guide covers why revenue per visitor, not conversion rate, has to be the reward signal in a pricing test, walks a worked example computed with the same statistics engine that runs this blog’s calculators, and is honest about the one thing a bandit still cannot give you: a formal, defensible statistical verdict.

Why pricing tests bleed differently than a copy test

Every A/B test that runs a fixed split pays an opportunity cost while it runs: half the traffic keeps seeing whichever variation turns out to be the weaker one, for as long as the test takes to reach its planned sample size. For a headline or a CTA color, that cost is usually small and recoverable, a few percentage points of conversion on a metric that resets with the next test.

A pricing test is a different animal. The “loss” from showing the wrong price is not a soft signal, it is dollars: a visitor shown a price too high to convert simply leaves, and a visitor shown a price too low converts at a rate that understates what the business could have collected from that same person. Both directions of error compound linearly with every visitor who passes through the test while the fixed split holds, and unlike a headline test, you usually cannot “make it back later,” a visitor who churns after a bad pricing experience does not typically come back to try a better one.

This is the same trade-off this blog’s multi-armed bandits vs A/B testing guide frames in general terms: a bandit reduces the opportunity cost of a fixed split by shifting traffic toward the better-performing arm while the test is still running, at the cost of not producing a classic statistical verdict at the end. Pricing is simply the case where that opportunity cost is at its most literal and its most expensive, which is exactly why it is the sharpest example of when the trade-off is worth taking.

The reward signal that actually matters: revenue per visitor

Here is the trap that a naive pricing test falls into: if you optimize purely for conversion rate, the cheapest price you test will, almost by construction, win. Lowering a price nearly always raises the share of visitors who buy. That does not mean it raises revenue.

The metric that actually answers “which price is better for the business” is revenue per visitor (RPV): conversion rate multiplied by the price charged to each converting visitor. A price that converts fewer visitors can still be the stronger choice if it converts them at high enough value to more than offset the lower conversion rate. This is precisely the numeric-metric case that vendor documentation on bandits treats differently from a simple conversion metric: Optimizely’s own support documentation describes running an Epsilon-Greedy bandit for numeric metrics like revenue, using robust statistical estimators of the mean, variance, and skew of the metric’s distribution to build the confidence bounds the algorithm needs, rather than the simpler math a binary conversion metric allows.

The numbers behind that figure are worked out step by step later in this guide, but the direction of the point holds generally: choosing the reward metric is not a formality in a pricing test, it decides which price you ship.

How a bandit protects margin while the test is still running

A bandit, in the general case, reallocates traffic toward whichever arm is currently performing best on the chosen reward metric, instead of holding a fixed split until a predetermined sample size arrives, the way a classic A/B test does. Thompson Sampling, already covered in depth in this blog’s Thompson Sampling guide, is the algorithm most commonly used in production Bayesian systems: it keeps a belief distribution over each price’s true reward, draws a random value from each distribution for every new visitor, and routes that visitor to whichever price drew the highest value. Prices with little data have wide, uncertain distributions and still get sampled often; prices with a lot of data and a clearly weak reward have narrow, low distributions and are drawn less and less, without ever being fully cut off.

Applied to a pricing test, that behavior maps directly onto the pain point: as evidence accumulates that one price point produces more revenue per visitor, the bandit sends it more traffic, which means fewer visitors are shown the weaker price for the remainder of the test. A classic 50/50 A/B test, by contrast, keeps the split flat from the first visitor to the last, on purpose, because moving the split mid-test is exactly what would compromise the formal verdict it is built to produce.

Bandit and Bayesian A/B testing share the same underlying math, the difference is purely what you do with it: decide traffic in real time as a bandit, or report a probability and a verdict once, at the end, as a classic Bayesian or frequentist A/B test does.

A worked numeric example: the same price test, three ways

To move past the concept, here is a concrete scenario, computed with the same significance function from this blog’s stats.ts engine that runs the calculator further down this page.

The scenario: a SaaS trial-to-paid upgrade page gets 10,000 visitors in one week. Price A is the current price, $29 per month, converting at 8.0% (400 of 5,000 visitors shown that price). Price B is a higher price being tested, $39 per month, converting at a lower 6.4% (320 of 5,000 visitors). This exact effect size, an 8% baseline dropping to 6.4%, a 20% relative decline, needs about 4,921 visitors per variant for 80% power at 95% confidence, using this blog’s frozen sampleSizePerVariant formula; the 5,000 visitors per arm in this scenario land just above that bar, so this is a properly powered test, with no margin to spare, not a lucky small sample.

Price	Visitors	Conversions	Conversion rate	Revenue per visitor
A ($29)	5,000	400	8.0%	$2.32
B ($39)	5,000	320	6.4%	$2.50

What the classic A/B verdict says

Run those exact numbers, 5,000 visitors and 400 conversions for A, 5,000 visitors and 320 conversions for B, through the calculator below (it uses the same two-proportion z-test as the worked example) to see the verdict for yourself:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Working the same numbers by hand: the pooled rate is 7.2%, the standard error works out to about 0.00517, and the z-score is about -3.10, giving a two-sided p-value of about 0.002, well under the usual 0.05 threshold. Read on conversion rate alone, this is a statistically significant result, and A is the declared winner, since it converts at a higher rate with a 95% confidence interval on the difference of roughly -2.6 to -0.6 percentage points, a range that excludes zero.

That verdict is significant, and it is also the wrong business answer, if the real question was which price makes more money. Revenue per visitor for A is 8.0% times $29, or $2.32. Revenue per visitor for B is 6.4% times $39, or $2.50, a relative lift of about +7.6% in B’s favor. A pricing test that reports only the conversion-rate verdict above would confidently ship the lower-margin price. This is exactly why the reward signal has to be revenue per visitor from the start, not conversion rate with a mental note to “check revenue too” afterward; by the time a fixed A/B test declares a conversion-rate winner, the wrong price may already look validated.

The same total traffic, run as a bandit instead

Now take the identical 10,000 visitors and the identical one-week window, but let a bandit reallocate traffic toward B (the higher-revenue price) as evidence accumulates, illustrated here in three phases rather than continuously, to keep the arithmetic auditable:

Phase	Days	Visitors (approx.)	Share to B (higher revenue)	Share to A
1: still uncertain	1 to 2	~2,857	50%	50%
2: evidence builds	3 to 5	~4,286	70%	30%
3: strong evidence	6 to 7	~2,857	85%	15%

Summed across the three phases, A receives about 3,143 visitors and B about 6,857. Expected revenue works out to:

• A: 3,143 x $2.32 ≈ $7,292.
• B: 6,857 x $2.496 ≈ $17,115.
• Total illustrative bandit revenue: ≈ $24,407.

Compare that against the fixed 50/50 design and the theoretical ceiling of sending every visitor to B from day one:

The bandit’s edge here, about $327, or roughly 1.4% more revenue than the fixed design over this single week, looks modest at this scale. It is not the headline number worth remembering. The number worth remembering is that this gain compounds for as long as the price stays live and traffic keeps flowing, which for a pricing decision, unlike a one-off headline test, is often measured in months or years, not one week. A structural, small percentage advantage on ongoing revenue is a materially different thing from the same percentage advantage on a test that runs once and ends.

Note also what this worked example deliberately does not do: it does not run stats.ts’s incrementalRevenue helper, because that function multiplies a rate difference by a single shared average order value, which fits a same-price scenario like a checkout-flow test, not a scenario where the price itself differs by arm. Reusing it here would misapply the formula to a case it was not built for; the honest approach, shown above, is to compute revenue per visitor directly for each price and compare the totals.

Contextual bandits for segment-based pricing

A simple bandit searches for one champion price for everybody. A contextual bandit goes a step further: it picks the price shown to each visitor based on attributes such as plan tier, company size, geography, or usage history, instead of assuming one price fits every visitor equally well. Optimizely’s documentation on contextual bandits describes exactly this pattern, personalizing a decision using attributes like device, location, or behavioral signals rather than serving the same variation to everyone.

For pricing, this maps to a real and common situation: the price a self-serve individual signup will tolerate is rarely the price an enterprise buyer with a dedicated budget line will tolerate, and a single-price test forces you to average across both, which can hide a genuine per-segment answer. A contextual bandit can, in principle, learn a different effective price per segment continuously, rather than requiring a separate fixed A/B test for every segment you care about.

It is worth flagging a specific, checkable constraint here rather than a general one: Optimizely’s own documentation on contextual bandits states that revenue should not be used directly as the primary optimization metric for that mode, because it can hurt the model’s effectiveness, a caveat that does not apply to a standard, non-contextual bandit. That is exactly the kind of detail that changes between vendor releases, so treat it as a reason to check current documentation before configuring a contextual bandit around a revenue metric, not as a permanent rule.

The added personalization also multiplies the fairness question covered next: a contextual bandit that quietly converges on “segment X pays less than segment Y” is doing exactly what it was told to do, optimize the reward metric, and that is precisely why the decision of which attributes are acceptable to price on cannot be left to the algorithm.

The fairness and legal side of testing prices

This section is a flag, not legal advice; consult counsel before shipping any pricing personalization, and this changes by jurisdiction and moves quickly. That said, three things are worth knowing before you optimize a price by segment:

• Price discrimination and personalization are, as a category, generally legal, but heavily scrutinized. A 2019 review in the Journal of Business Ethics frames algorithmic pricing personalization as a practice whose legality is rarely the binding constraint, its ethicality and the perception of fairness by the people paying different prices for the same thing usually matters more in practice.
• Regulators are actively watching this specific practice. According to a WilmerHale client alert on personalized pricing, the US Federal Trade Commission has studied so-called “surveillance pricing,” and New York’s Algorithmic Pricing Disclosure Act, in effect since November 2025, requires businesses that set a price using a visitor’s personal data to disclose that fact on the page. Multiple other US states have introduced similar bills. Treat any specific figure or law here as a snapshot, and confirm the current rule in your jurisdiction before relying on it.
• The technical and the legal questions are separate, and both apply. A bandit or contextual bandit deciding to show a higher price to one segment is a business and legal decision dressed up as a statistics question. The algorithm will optimize whatever metric and whatever attributes you give it; deciding which attributes are acceptable to price on is a human, documented decision, not something to infer from what the model happened to converge on.

None of this is a reason to avoid pricing experiments. It is a reason to keep a human, documented decision about which attributes are in bounds, separate from the algorithm that optimizes within those bounds.

When a classic A/B test is still the better call

A bandit earns its keep for continuous tuning of a price within an already-approved range, for example, adjusting a discount percentage or a trial length up or down as evidence accumulates, where the cost of a wrong guess is ongoing and the decision does not need to be defended to anyone beyond the team running it.

A classic, fixed-split A/B test is still the right tool when:

• The decision needs a defensible verdict. A board-level pricing decision, such as whether to raise the headline price of a core plan, usually needs a number with a confidence level attached, a p-value and a confidence interval, not “revenue improved over an equal split.” That is exactly the sentence this blog’s statistical significance guide walks through with a comparable worked example.
• The new price needs to be analyzed by segment or channel afterward. A stable, known split makes it possible to ask, after the fact, whether the new price performed differently by acquisition channel or by cohort. A bandit’s allocation, having already shifted on its own during the test, makes that kind of clean post-hoc slicing harder.
• The change is close to irreversible or expensive to unwind. Repricing an entire product line, or removing a legacy grandfathered price, is the kind of decision you want to make once, with a formal answer, rather than tune continuously.

Many teams reasonably use both in sequence: a classic A/B test to decide whether a new price point is validated at all, and a bandit afterward to keep tuning within the range that test already approved.

Automate this on Donnu

The core tension in this guide is specific to pricing: every visitor shown the losing price during a fixed-split test is a direct hit to margin or a lost sale, not a soft signal you can shrug off until the test ends. Donnu already runs on a native Bayesian engine, the same statistical foundation a bandit like Thompson Sampling needs, and it tells you honestly when a price test does not yet have enough data to trust its verdict, rather than letting you read a premature “winner” into noise and burn margin shipping it. Continuous, bandit-style price optimization is the natural next step for that same engine, and it is the direction the roadmap points toward for teams that need to keep tuning a price after the initial test has already validated it.

Read more on this blog: the full multi-armed bandits vs A/B testing guide, Thompson Sampling explained in depth, and the statistical significance guide for the classic-test side of this decision.

Start a free 14-day trial and run your next price test on a statistical foundation built to tell you the truth about it, not just the number you were hoping for.

References

• Optimizely. Maximize lift with multi-armed bandit optimizations (bandits do not generate statistical significance; Epsilon-Greedy for numeric metrics). support.optimizely.com.
• Optimizely. Stats accelerator overview (robust estimators of mean, variance, and skew used to build confidence bounds for numeric metrics like revenue). support.optimizely.com.
• Optimizely. Contextual bandits (segment-based personalization, the revenue-as-primary-metric caveat). support.optimizely.com.
• Wilmerhale. Personalized Pricing: What Business Lawyers Need to Know (FTC surveillance pricing study, New York’s Algorithmic Pricing Disclosure Act). wilmerhale.com.
• Mapping the Ethicality of Algorithmic Pricing: A Review of Dynamic and Personalized Pricing. Journal of Business Ethics, Springer Nature. link.springer.com.
• den Boer, A. Multi-armed bandits in dynamic pricing. Lancaster University STOR-i. lancaster.ac.uk.
• Bouneffouf, D. & Rish, I. A Survey on Practical Applications of Multi-Armed and Contextual Bandits. arXiv, 2019. arxiv.org/abs/1904.10040.