Product Recommendation A/B Testing: Does It Convert?
Product recommendation a/b test: the metric that misleads, cannibalization, an honest control, latency cost and how to size the experiment properly.
📚 This article is part of the guide Ecommerce Checkout Optimization: The A/B Testing Playbook.
A product recommendation A/B test compares a version of the store with automatic suggestions against one without them (or with a different suggestion logic), to find out whether personalization creates new orders or simply moves the same sale around. It is one of the tests where the metric the tool puts on its main screen is exactly the one that misleads the most. This article is part of the complete checkout optimization playbook and covers what to test, which metric decides, how to build an honest control, how much traffic the experiment needs and the traps specific to personalization.
What a recommendation block actually does
A recommendation block solves a concrete problem: in a store with a large catalog, most products are never seen, and a visitor who cannot find the right item leaves without buying. Recommendations shorten that search. The problem is that they do it by occupying space on a page that already had a job, and the attention detour does not always pay for itself.
Three positions account for most implementations, and each tests a different hypothesis:
- On the product page. The hypothesis is discovery: “this item is not quite what you want, here are similar ones”. It helps people who arrived from a generic search and gets in the way of people who already decided.
- In the cart. The hypothesis is basket size: “people who bought this also bought that”. It is the position with the highest chance of raising order value and the highest risk of adding friction at the most sensitive step of the funnel.
- After purchase (confirmation page or email). The hypothesis is repurchase: no risk to the current order, effect measured in the next cycle. It is the safest position and the slowest to evaluate.
The metric that misleads: widget clicks
Every recommendation tool shows, on its opening dashboard, the click-through rate on the block and the revenue attributed to it. Both numbers almost always look excellent, and neither answers the question the test is asking.
Click-through measures visual appeal, not increment. A well designed carousel gets clicks in any store. Attributed revenue is worse, because it carries a silent assumption: that a purchase made after a widget click would not have happened without it. In practice, much of it would have.
That has a name: cannibalization. Three common shapes:
- The visitor was already going to buy product A, clicks a recommendation, buys product B. The store records a sale attributed to recommendations, and total orders did not move.
- The visitor would have bought anyway, and the recommended item is cheaper than the original. Recommendations show up as a win and the basket shrinks.
- The visitor enters a browsing spiral between suggested products, decides nothing and leaves. No widget dashboard records that case.
The correct reading ignores clickers entirely and compares all visitors in each variation: orders per visitor, revenue per visitor. It is the only way to measure increment instead of attribution.
The honest control: best sellers against the algorithm
The second structural mistake in this kind of test is the choice of control. Comparing “with recommendations” against “with nothing” measures the value of filling that space with anything, not the value of personalization. A static best-sellers list for the category is a strong, cheap control: high social proof, guaranteed stock and zero added latency.
| Recommendation logic | What it uses | When it usually wins | Cost and risk |
|---|---|---|---|
| Category best sellers | Aggregate sales history | Small catalogs, new traffic, fast-moving products | Practically none; it is the control to beat |
| Manual rules (curation) | Relationships defined by someone who knows the catalog | Catalogs with obvious item relationships (kits, accessories) | Maintenance cost; ages silently |
| Bought together (collaborative) | Item co-occurrence in past orders | Large catalogs with heavy order history | Cold start on new products; reinforces what already sells |
| Session behavior based | The visitor own recent browsing | Long exploratory sessions | Depends on personal data; needs a legal basis under privacy law |
| Predictive model by profile | History, profile and context combined | Large logged-in, returning bases | Infrastructure, latency and less explainability |
One practical consequence of that table: if your store still has no well placed best-sellers block, that is the test to run first. It is cheaper, faster to implement, and it establishes the baseline any algorithm will later have to prove itself against.
Sizing the test
Set your current order rate, the minimum gain that would justify keeping recommendations (including the monthly cost of the tool) and your real weekly traffic:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
A worked example with real numbers
A store converts 2.2% of visitors into orders and gets 50,000 visits per week. The team wants to turn on a recommendation block on the product page.
Detecting an 8% relative improvement (from 2.2% to 2.376%) at 95% confidence and 80% power requires 113,296 visitors per variation, which takes about 32 days. Aiming at 12% relative (from 2.2% to 2.464%), the requirement drops to 51,299 per variation and the test closes in about 15 days. Look at the size of those numbers: this is why recommendation tests with a few thousand visitors per side almost always end inconclusive, and the team ends up deciding on widget click-through, which is precisely what should not happen.
Suppose the test ran to 52,000 visitors per variation and closed with 1,144 orders in control (2.20%) against 1,248 in the recommendation variation (2.40%). Running those four numbers through the significance engine used across this blog: z = 2.15, p-value about 0.0315, with a 95% confidence interval for the difference between +0.02 and +0.38 percentage points, and an observed relative improvement of +9.1%. The result is significant, and the interval is an important reminder of how imprecise a significant result can be: the true gain could be as small as 0.02 points, practically nothing.
Does the gain pay for the tool?
This is the question that closes the test, and the confidence interval is what makes it answerable. Take the same store: 50,000 weekly visits, an average order value of $60, gross margin of 35%, and a recommendation tool costing $1,200 per month. Project each end of the interval and use 4.33 weeks per month:
| Reading of the gain | Extra orders per week | Monthly revenue | Monthly gross contribution | Net of the $1,200 tool |
|---|---|---|---|---|
| Lower bound, +0.02 pt | 10 | $2,598 | $909 | −$291 |
| Point estimate, +0.20 pt | 100 | $25,980 | $9,093 | +$7,893 |
| Upper bound, +0.38 pt | 190 | $49,362 | $17,277 | +$16,077 |
The same statistically significant result covers a range from “does not pay the subscription” to “pays it fourteen times over”. That is not a flaw in the test, it is what a p-value of 0.03 on a small effect honestly means. For an investment decision, use the lower bound; to prioritize the next test, use the point estimate. The statistical significance guide details how to read each of those numbers, and the revenue per visitor calculator assembles the money side.
One-off test or permanent holdout
Recommendations are one of the few ecommerce levers where it pays to keep a control group after the test ends. The reason is simple: the algorithm changes on its own. It relearns from new orders, reacts to catalog and stock changes, and what worked in March can be neutral in September without anybody touching a setting.
A one-off test answers “turning recommendations on beat not turning them on, during those weeks”. A permanent holdout, a small fixed slice of traffic (typically between 2% and 5%) that never sees recommendations, answers a different and more durable question: “how much is the algorithm delivering this quarter”. It is the same guardrail logic email teams use to measure an automation program over time.
| One-off test (A/B) | Permanent holdout | |
|---|---|---|
| Question it answers | Is it worth turning on? | How much is it worth right now? |
| Duration | Weeks, until the sample closes | Continuous, read quarterly |
| Traffic slice in control | 50% | Between 2% and 5% |
| Cost | Half the traffic without the lever for weeks | A small slice permanently without the lever |
| When to use | Before buying or building | After the lever becomes part of the store |
The cost of a holdout is real and deserves to be stated honestly: if recommendations do convert better, you are leaving money on the table in that small slice, every day. The trade is knowing, with a number instead of faith, whether the tool you pay for every month is still delivering. Because the slice is small, the reading takes longer to accumulate sample, so treat the holdout as a quarterly report, not a Monday morning dashboard.
Mandatory guardrails
| Guardrail | Why watch it | Warning sign |
|---|---|---|
| Average order value | Recommendations can swap an expensive item for a cheap one | Orders rise and revenue per visitor stays flat or falls |
| Load time | The block almost always comes from an external service | The recommendation variation loads noticeably slower, especially on mobile |
| Return rate | A suggested purchase is likelier not to be what the person wanted | Returns rise in the winning variation in the following weeks |
| Stock-outs | The algorithm tends to concentrate traffic on the same products | Recommended items sell out and the block starts suggesting unavailable ones |
| Catalog diversity sold | Collaborative filtering reinforces what already sells | The long tail disappears from sales and the store depends on a handful of items |
The latency guardrail deserves emphasis because it is the most frequently forgotten. A block that adds requests and JavaScript after the main content charges a toll on every visit, including the ones that never looked at the carousel. If the block delays product rendering, part of the discovery gain comes back as abandonment from slowness, and the test measures the sum of both effects without separating them.
Personalization and personal data
Recommendations based on best sellers or on order co-occurrence work with aggregate data. Recommendations based on an individual visitor behavior, or on the history of an identified customer, process personal data, which requires a legal basis and transparency under privacy regimes such as the GDPR in Europe and the LGPD in Brazil.
Two practical implications for test design: the personalized variation has to respect the same consent policy as the control (if personalization depends on cookies that a share of visitors refuses, that share behaves like control and dilutes the measured effect), and the notice shown to users about data use has to be identical on both sides, otherwise the consent difference becomes a confounder no calculator can fix afterwards.
Traps specific to recommendation tests
- Novelty effect. A new carousel attracts attention purely because it is new in the first weeks, especially among returning customers. Run the test for at least two complete weekly cycles and check whether the effect holds in the second one.
- Turning recommendations on store-wide at once. It produces an aggregate number that never says where they work. Test by position.
- Cold start on new products. Collaborative algorithms cannot recommend what has never been bought, and newly launched products stay invisible exactly when they need exposure most.
- Swapping the algorithm mid-test. It is the personalization version of moving the threshold mid-test: visitors at the start saw a different product from visitors at the end, inside the same variation.
- Reading only the aggregate. The effect is usually positive for people arriving from generic searches and neutral or negative for people arriving on a specific product search. Declare those segments before running, as recommended in common A/B testing mistakes.
Make this automatic with Donnu
Testing product recommendations needs three things at once: a sample large enough for a small effect on a low base rate, a reading by orders and revenue per visitor instead of widget clicks, and an honest confidence interval to compare against the real cost of the tool.
Donnu A/B delivers that on your site: a light snippet that does not slow the product page down, automatic sample sizing and Bayesian statistics that do not invent certainty. Start a 14-day free trial and find out whether your recommendations create new sales or just move the sale that already existed.
Read also: Ecommerce Checkout Optimization: The A/B Testing Playbook · Product Page A/B Testing · Upsell and Cross-Sell A/B Testing · Leia em português: Teste A/B de Recomendação de Produto
References
- Baymard Institute. Checkout Usability Research. Base of usability issues documented in large-scale ecommerce research, including navigation and product discovery. baymard.com/research/checkout-usability.
- Baymard Institute. Cart Abandonment Rate Statistics. Average of 50 studies on cart abandonment and the distribution of abandonment reasons. baymard.com/lists/cart-abandonment-rate.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020, on novelty effects, guardrail metrics and segment reading. Companion material at experimentguide.com.
- European Commission. Data protection in the EU (GDPR). Legal basis and transparency for processing personal data, which includes personalization based on individual behavior. commission.europa.eu/law/law-topic/data-protection_en.
Frequently asked questions
- Which metric decides a product recommendation test?
- Orders per visitor and revenue per visitor, never clicks on the recommendation block. Widget click-through is the metric the recommendation tool shows first and the one that always looks good, because any visually appealing block gets clicks. What it cannot separate is whether that click produced a new purchase or simply diverted a purchase that was already going to happen. Only a bottom-of-funnel metric, measured across all visitors in the variation and not only the ones who clicked, answers that.
- What is cannibalization in a recommendation test?
- It is when the recommendation captures a purchase that would have happened anyway, usually by replacing the product the person had already decided on with a suggested one. The widget dashboard records this as a win, because there was a click and there was revenue attributed to the block. In store aggregate, however, order count does not move and the basket may even shrink, if the recommended item is cheaper than the original. That is why the correct reading compares all visitors in each variation, not the buyers attributed to the widget.
- Does an algorithm beat a plain best-sellers list?
- Not always, and that is the honest comparison many tests avoid. A curated best-sellers list for the category is a strong control: it already surfaces products with high social proof and guaranteed availability, with no infrastructure cost and no added latency. An algorithm has to beat that control, not an empty page. If your test compares personalization against no recommendation at all, it measures the value of having a block, not the value of the algorithm.
- How much traffic does a recommendation test need?
- A lot, because the expected effect is usually small and the base order rate is low. With the math used across this blog, a store converting at 2.2% that wants to detect an 8% relative improvement needs about 113,296 visitors per variation. Aiming at 12% relative drops the requirement to about 51,299 per variation, which at 50,000 weekly visits closes in around 15 days. Recommendation tests with a few thousand visitors per side almost always end inconclusive.
- Does a recommendation block slow the page down, and does that affect the test?
- Yes, and it is a cost that rarely enters the math. Most recommendation blocks load from an external service, after the main content, adding requests and JavaScript to the page. If the variation with recommendations is noticeably slower, part of the discovery gain is handed back as abandonment from slowness, especially on mobile connections. Measure load time per variation as a guardrail, and load the block in a way that never blocks rendering of the main product.