Personalization vs A/B Testing: When to Use Which
Personalization vs A/B testing: the different questions each one answers, the real traffic cost of segmenting and a decision rule with a calculator.
📚 This article is part of the guide AI Personalization and A/B Testing: How They Work Together.
Personalization and A/B testing are not alternatives: they are tools that answer different questions, and the choice between them should almost never be “which of the two”. An A/B test measures whether a change causes an effect. Personalization decides what to serve each person. Confusing them costs money in two opposite directions: personalizing too early burns traffic validating rules nobody asked for, and testing without ever looking at heterogeneity hides effects that are really there. This guide, part of the guide to AI personalization and A/B testing, covers the decision rule, the real traffic cost of each path, with a calculator and a worked example, and the design that combines both without fooling yourself.
The two questions, side by side
The fastest way to get this wrong is to treat personalization as “a smarter A/B test”. It is not a better version of a test, it is a different thing.
| Dimension | A/B test | Personalization |
|---|---|---|
| Question it answers | Does this change cause an effect, and how big? | Which experience do I serve this visitor? |
| How it assigns people | Random draw, equivalent groups | By visitor attribute, deliberately different groups |
| What it produces at the end | A single decision, applied to everyone | A set of rules, one per segment |
| What it needs to work | Enough sample and a primary metric | An attribute observable during the visit and evidence that it matters |
| How you prove it worked | The test itself | A holdout kept live, measuring the whole program |
| Main failure mode | Stopping early and reading noise | Optimizing internal metrics without producing additional business |
Notice the most important row: an A/B test proves itself, personalization does not. A well sized test carries its own evidence. A personalization system needs an external instrument, the holdout group, to know whether it is producing results. That asymmetry is the practical reason to start by testing.
The decision rule, in one question
Before any arithmetic, one filter resolves most cases: is the right answer the same for everyone?
If the change you are considering is the kind that probably improves life for any visitor, test it and ship it to everyone. A checkout with fewer steps, a form with fewer required fields, more legible pricing, a faster page: almost nobody prefers the opposite. Personalizing those changes creates complexity to manage a difference that does not exist.
If the right answer depends on something you already know matters and can observe at the moment of the visit, personalization starts to make sense. Both conditions hold together: knowing it matters without being able to observe it never becomes a rule, and being able to observe it without evidence that it matters is the definition of segmenting for the sake of segmenting.
The traffic cost: why personalization is more expensive to prove
Here is the part that almost never enters the conversation. Personalizing is not just harder to implement, it is more expensive to prove, and the cost shows up as time.
Consider a site with 56,000 visitors per month (about 14,000 per week) and a conversion rate of 2.4%. The team wants to detect a 15% relative improvement, at 95% confidence and 80% power.
Running a single test across the whole site, the required sample is 30,443 visitors per variation, which takes 31 days.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Set the calculator above to a 2.4 baseline rate, a 15 minimum detectable effect (relative) and 14,000 visitors per week to reproduce that number. Now suppose that instead of a single change, the team decides to build four rules, one per segment (new against returning, mobile against desktop, for example), each segment holding roughly a quarter of the traffic, that is 3,500 visitors per week. The required sample per variation does not change, because it depends on the baseline rate and the effect, not on the size of the segment. What changes is how long it takes to accumulate it: 122 days inside a single segment.
| Design | Traffic feeding it | Sample per variation | Duration |
|---|---|---|---|
| Single test across the whole site, 15% MDE | 14,000/week | 30,443 | 31 days |
| One rule per segment, 15% MDE | 3,500/week per segment | 30,443 | 122 days per segment |
| One rule per segment, 30% MDE | 3,500/week per segment | 8,126 | 33 days per segment |
The third row shows the only honest way out when the segment is small: lower your ambition about the size of the effect. Validating one rule per segment in an acceptable window requires betting on large differences between segments. If your hypothesis is that mobile visitors react 5% better to a different layout, the data will not arrive in useful time. If the hypothesis is that they react 30% better, it might, and it is worth asking why a difference that size went unnoticed until now.
There is also a hidden multiplicity cost. Reading four segments at 95% confidence each is not the same as reading one. The probability of finding at least one false positive among the four reaches roughly 18.6%, almost one in five. That does not invalidate segment reading, but it explains why a “discovery” found inside a cut has to be confirmed before it becomes a permanent rule.
Why proving heterogeneity costs four times as much
If personalization is justified by different segments reacting differently, the rigorous way to establish that is an interaction test: an experiment designed from the start to measure whether the effect of the change differs between two groups. It is worth being precise about its price, because the arithmetic surprises people.
In a balanced two by two design (control and variation, crossed with segment 1 and segment 2), the main effect is estimated by averaging across both segments, while the interaction is the difference of two differences. Writing v for the variance of one cell:
| Quantity being estimated | How it is built from the four cells | Variance | Standard error |
|---|---|---|---|
| Main effect | Average of the two variation cells minus average of the two control cells | v | 1 unit |
| Interaction | (variation minus control in segment 1) minus (variation minus control in segment 2) | 4v | 2 units |
The interaction estimate carries twice the standard error of the main effect from the same data. Since required sample scales with the square of the standard error, detecting an interaction as large as the effect itself needs 4 times the sample per cell. And because interactions are usually smaller than main effects, the practical number is worse: an interaction half the size needs 16 times.
Applied to the same site: the main effect needs 30,443 per variation and 31 days. An interaction of equal magnitude needs 121,772 per cell, and with four cells that is 487,088 visitors, about 244 days at 14,000 per week. If the interaction is half the size of the main effect, the design does not fit in a calendar year.
This is why “we will run an interaction test to be sure” is, for most teams, not an option that exists. It also reframes the practical advice: the cheap instruments below are not shortcuts around rigor, they are what you use because the rigorous instrument is priced out of reach.
What personalization delivers that testing does not
None of this means personalizing is a mistake. It means it solves a specific problem, and it is worth being precise about which.
An A/B test produces a single decision. If variation B wins, everyone starts seeing B, including the people for whom A was better. When there is real heterogeneity of effect, that is when the same change improves results for one group and hurts another, the single decision leaves value on the table, and in extreme cases the average effect lands near zero while hiding two large effects with opposite signs.
The delicate point: that figure describes a possibility, not a pattern. Most changes that improve conversion improve it for almost everyone, and strong heterogeneity is less common than intuition suggests. That is why the practical order is to test first, read the result by segment as a clue, and only personalize when the clue repeats.
How to know whether the heterogeneity is real
Three instruments, in increasing order of cost and reliability.
1. Domain knowledge, declared in advance. If your product is sold one way to consumers and another way to companies, and the funnel itself is already different, you do not need a test to know the two groups react to different things. Business evidence counts, as long as it precedes the data rather than being an explanation invented after seeing the result.
2. Pre registered segment reading. Pick two or three cuts before running the test, write them down next to the hypothesis, and look only at those at the end. The limit on the number is what makes the reading worth anything: with a few cuts defined in advance, the false positive risk stays controlled; with ten cuts defined afterwards, it stops being interpretable.
3. A dedicated interaction test. The rigorous version, priced in the section above. In practice it only pays for itself when the per segment rule will live in production for a long time and moves meaningful revenue.
The order matters. Almost every program that jumps straight to per segment rules skipped the first two steps, and that is why it cannot answer whether the rules are helping or merely fragmenting the operation.
The ecommerce case and the SaaS case
The same reasoning lands very differently depending on the business model, and it is worth being explicit, because popular recommendations usually come from one of the two contexts without saying which.
In ecommerce, the most useful attribute is generally available on the very first visit: traffic source, device, product category viewed, new against returning customer. Volume helps too, because the baseline rate of intermediate steps such as add to cart is high enough to close a sample in a short window. The most real and most defensible heterogeneity is between new visitor and returning customer: they arrive with different information and need different things on the page. The typical risk is personalizing product recommendations and never comparing against a group that receives no recommendation at all.
In SaaS, the relevant attribute is almost never visible at the moment of the visit: company size, use case, team maturity. It appears after signup, which pushes personalization inside the product rather than onto the landing page. Volume also tends to be lower and the top of funnel conversion rate quite low, which makes per segment validation far more expensive. The practical consequence is that most SaaS companies gain more from running well designed single tests and reserving personalization for onboarding, where the attribute is already known and the effect is larger.
| Aspect | Ecommerce | SaaS |
|---|---|---|
| Attribute available during the visit | Source, device, category viewed, recurrence | Little: almost everything relevant appears after signup |
| Baseline rate useful for closing a sample | High on intermediate steps | Low at the top, better inside the product |
| Where personalization usually pays | Storefront, recommendations, shipping and offers | Onboarding and activation, after signup |
| Most common mistake | Measuring recommendations by the recommender’s own dashboard | Segmenting the top of funnel without volume to validate |
The design that combines both without fooling yourself
In mature practice the two coexist in a sequence, not in a choice. Four steps:
- Test the change across the whole site. Decide on it. It is the cheapest and most reliable instrument available for knowing whether the change works.
- Read the result across a few segments, defined before running. Two or three, chosen by hypothesis rather than by convenience. This is exploratory reading and it exists to generate the next question.
- If one segment repeatedly shows different behavior, test the specific rule inside it. Now with its own sample, properly sized, accepting that it will take longer. A clue confirmed twice is worth a dedicated test.
- If the rules pile up into a system, hold out a group. Once a set of rules is running together, the only way to know whether the set produces results is to compare against a group that receives none of them.
The symmetric mistake, equally common, is the opposite: never looking at a single segment, deciding everything by the average and concluding that “personalization is hype”. That leaves value on the table too, just silently. The difference between reading segments as hypotheses and reading segments as decisions is what separates the two practices.
Common mistakes on both sides
| Mistake | Which side | Why it hurts | Correction |
|---|---|---|---|
| Personalizing before having evidence of heterogeneity | Personalization | Multiplies validation cost with no known return | Run the single test first and use the segment as a hypothesis |
| Creating a segment after looking at the result | Personalization | The cut found in the data is usually noise | Define segments before running, in small numbers |
| Measuring personalization by the system’s own dashboard | Personalization | The dashboard shows internal hit rate, not additional business | Hold out a group and compare against it |
| Deciding everything by the average and never reading a cut | A/B testing | Hides real heterogeneity when it exists | Read a few pre defined segments as exploration |
| Treating a win in a small segment as a rule | Both | Insufficient sample and multiplicity combined | Confirm with a dedicated test before it becomes a rule |
| Believing personalization removes the need for a comparison group | Personalization | Without a control there is no causal estimate | A global holdout kept live |
Make this automatic with Donnu
The decision between testing and personalizing gets much easier when the feasibility arithmetic shows up before the discussion does, and the calculator above settles that part in a minute. What comes next is where most teams slip: reading the result without turning an imprecise estimate into a permanent rule. Donnu covers that stretch: you define the hypothesis and the metric, and Donnu returns a verdict with the 95% confidence interval up front, without declaring a winner before the variation accumulates at least 200 visitors and 7 days live, so that “it worked” and “it worked enough to become a rule” stay visibly different sentences.
Start a free 14 day trial and settle the next decision by arithmetic rather than by preference. For the full picture on the topic, see the guide to AI personalization and A/B testing.
References
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
- Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.
- Google Search Central. A/B testing best practices for Search. developers.google.com/search/docs/crawling-indexing/website-testing.
Read also:
Frequently asked questions
- What is the difference between personalization and A/B testing?
- They answer different questions. An A/B test answers "does this change cause an effect, and how big is it", by comparing two groups made equivalent through random assignment that see different things. Personalization answers "which experience do I serve to each visitor", and by construction it serves different things to different people. One measures, the other delivers. Using one in place of the other is usually expensive.
- When is personalizing better than running an A/B test?
- When you already have evidence that different segments react differently to the same change, not merely a suspicion that they would. Without that evidence, personalizing is a bet on an interaction nobody has measured, and the price is high: every per segment rule needs its own sample to be validated, which multiplies learning time.
- Does personalization need more traffic than testing?
- Yes, and the gap is usually underestimated. Validating a single change across the whole site consumes the whole traffic. Validating four rules, one per segment, splits the same traffic into four queues and each queue needs the same sample per variation. In the worked example in this guide, the same test takes 31 days across the whole site and 122 days inside a segment that holds a quarter of the traffic.
- Can I use the segment reports from my A/B test to decide where to personalize?
- As a source of hypotheses, yes. As a decision, no. Every segment looked at after the fact is one more test, and the chance of finding at least one false positive climbs fast: looking at 4 segments at 95% confidence each, the probability of at least one false positive among them reaches roughly 18.6%. A small number of segments defined before the test runs is the only way to read cuts without turning noise into strategy.
- Is an interaction test worth running before personalizing?
- Only when the rule will stay in production for a long time and touches meaningful revenue, because the arithmetic is brutal. In a balanced design, the estimate of an interaction carries twice the standard error of a main effect, so detecting an interaction as large as the effect itself takes 4 times the sample per cell. In this guide the main effect needs 30,443 per variation and 31 days, while the equivalent interaction needs 121,772 per cell and about 244 days at the same traffic.
- Does personalization ever replace A/B testing?
- No. Even a mature personalization program needs a holdout group that never receives any personalization, in order to know whether the whole system produces incremental results. Without that holdout, the dashboard shows how often the system hit its own rules, not how much additional business showed up because of it.
- What is the simplest rule for choosing between the two?
- Ask whether the decision is the same for everyone. If the right answer is probably single (a shorter checkout, a form with fewer fields, clearer pricing), run an A/B test and ship it to everyone. If the right answer depends on an attribute you already know matters and can observe at the moment of the visit, personalize and measure the program against a holdout.