Novelty Effect in A/B Testing: How to Detect It
What the novelty effect is, how to tell a real decaying effect from a statistical artifact, and the three checks that separate a fad from a durable win.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
The novelty effect is a short-lived lift that comes from a change being new rather than from it being better, and the reason it matters is that most tests are read in exactly the window where it is strongest. A redesign that returning visitors click on because they noticed something moved will produce a very convincing first week. This guide covers how to tell a decaying effect from a durable one, why most suspected novelty is actually a statistical artifact, the new-versus-returning split that settles it, and how to design a test that answers the question instead of inviting the argument. It is part of our complete guide to A/B testing and pairs with how many visitors an A/B test needs.
What the novelty effect and primacy actually are
Two opposite distortions show up when a change is first exposed to people who already know the old version.
Novelty inflates the variation early. Users notice something is different, investigate it, click it, and the metric moves for reasons unrelated to the change being an improvement. Kohavi and colleagues describe this as a bias that dies quickly if the feature is not truly useful.
Primacy depresses the variation early. Experienced users are temporarily less efficient with a navigation or layout they have not learned yet, which gives an inherent advantage to control until they adapt.
Both are properties of returning visitors. A first-time visitor has no previous version to be surprised by and nothing to relearn, which is the single most useful fact in this whole topic and the basis of the diagnostic below.
The surprising part: most suspected novelty is not real
This is where the honest version of the topic diverges from the popular one. Kohavi, Deng, Longbotham and Xu report that in practice novelty and primacy effects are uncommon, and that the cases they did observe concentrate in recommendation systems, where a new algorithm introduces one-time diversity or draws on a finite pool of items, so the early advantage naturally fades. Their earlier analysis of puzzling experiment outcomes is blunter: most cases of suspected primacy and novelty effects are not real, but a statistical artifact.
The artifact comes from how teams look at results. A cumulative-effect chart in the first few days is dominated by noise, because the sample is small and the confidence interval is enormous. A curve that wanders upward for four days looks like a trend, and an experimenter who wants the feature to succeed extrapolates it. Most experiments actually have a stable effect with high early variance, and the apparent trend disappears as the sample grows.
The same work reports something that should end a common argument: they could not find a single experiment where a statistically significant result in one direction became statistically significant in the other direction because of these effects. A test that is clearly negative after two weeks is not going to turn positive once users get used to it. Failing fast is the better move.
So novelty is a hypothesis worth testing, not a default explanation. The three checks below are how you test it.
Worked example: the sticky bar that faded
An ecommerce team ships a sticky add-to-cart bar. Traffic is 25,000 visitors per week, split evenly, so each arm gets 12,500 per week. The primary metric is purchase rate, baseline 4.00 percent.
Week by week, control holds at 4.000 percent while the variation does this:
| Week | Variation rate | Relative lift | p-value | 95 percent interval on the difference |
|---|---|---|---|---|
| 1 | 4.720 percent | +18.00 percent | 0.0053 | +0.214 to +1.226 pp |
| 2 | 4.360 percent | +9.00 percent | 0.1550 | -0.136 to +0.856 pp |
| 3 | 4.160 percent | +4.00 percent | 0.5226 | -0.330 to +0.650 pp |
| 4 | 4.120 percent | +3.00 percent | 0.6307 | -0.369 to +0.609 pp |
Each row is 12,500 visitors per arm, control at 500 purchases per week.
Now the cumulative view, which is what the dashboard shows and what most reviews discuss:
| Through week | Visitors per arm | Cumulative lift | p-value | Verdict at 5 percent |
|---|---|---|---|---|
| 1 | 12,500 | +18.00 percent | 0.0053 | significant |
| 2 | 25,000 | +13.50 percent | 0.0028 | significant |
| 3 | 37,500 | +10.33 percent | 0.0048 | significant |
| 4 | 50,000 | +8.50 percent | 0.0072 | significant |
The cumulative test is significant at every single checkpoint, and it is significant at the end. A team reading only this table ships the bar and books a +8.5 percent improvement in the plan. That number will not appear next quarter.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Here is why. Pool weeks three and four alone, which is the closest thing to the steady state: 25,000 visitors per arm, control at 1,000 purchases (4.000 percent), variation at 1,035 (4.140 percent). Result: +3.50 percent relative, z = 0.792, p-value = 0.4283, interval -0.206 to +0.486 percentage points. Inconclusive, and the interval comfortably includes zero.
The cumulative number is not wrong, it is answering a different question. It reports the average effect over the whole test, including the week when the bar was new. What you need for a decision is the effect that will still be there next month, and only the weekly view estimates that.
The new versus returning split that settles it
The weekly decay is suggestive. The segment split is close to decisive, because novelty has a mechanism and the mechanism has a signature.
In week one of the same experiment, splitting by visitor type at 60 percent new and 40 percent returning:
- New visitors: 7,500 per arm, control 285 purchases (3.800 percent), variation 300 (4.000 percent). Lift +5.26 percent, p-value 0.5270, interval -0.420 to +0.820 percentage points. Inconclusive.
- Returning visitors: 5,000 per arm, control 215 purchases (4.300 percent), variation 290 (5.800 percent). Lift +34.88 percent, p-value 0.0006, interval +0.642 to +2.358 percentage points. Strongly significant.
That is the fingerprint. The entire week-one effect lives in the segment that had something to notice. A visitor who has never seen your site cannot be surprised that the add-to-cart bar moved, so if the change were genuinely better, new visitors should benefit too. They did not.
One caution to keep this honest: this split has to be planned, not discovered. Slicing a result by segment after the fact generates many silent comparisons and at least one of them will look significant, which is the multiple comparisons problem in a different costume. New versus returning earns an exception only because you can name it before launch as a standing check for every test that touches a familiar interface.
Designing so the question never comes up
The cheapest way to win the novelty argument is to make it unnecessary before the test starts.
Run whole weeks, at least two. Kohavi and colleagues recommend two weeks specifically to expose effects that fade. Whole weeks also keep weekday and weekend mix balanced, which removes a second confound that gets misdiagnosed as novelty.
Size for the effect you expect to survive, not the one you hope to see. This is the part that hurts. If your steady-state expectation is a 5 percent relative improvement on a 4 percent baseline, that needs 154,304 visitors per arm and 87 days at 25,000 per week. An 8.75 percent expectation needs 51,266 per arm and 29 days. The 18 percent effect you saw in week one needs only 12,626 per arm and 8 days, which is exactly why a short test finds it and nothing else.
Two-proportion normal approximation, traffic split evenly across variations. The date uses your timezone and updates live.
Declare the primary analysis window in advance. Writing “the decision uses weeks two onward” in the experiment record before launch turns an argument into a rule. Deciding to drop week one after seeing that it helps is peeking wearing the costume of rigor.
Name the segments up front. New versus returning, on any change to a familiar surface. Two segments declared in advance cost you almost nothing; twelve discovered afterwards cost you the result.
| Design decision | Weak version | Version that survives review |
|---|---|---|
| Duration | Stop when significant | Whole weeks, minimum two, fixed in advance |
| Sizing | Powered for the week-one effect | Powered for the expected steady-state effect |
| Primary window | Whole test, cumulative | Declared steady-state window, decided before launch |
| Segments | Sliced afterwards until something appears | New versus returning, named in the plan |
| Decision rule | Ship if the cumulative p-value is below 0.05 | Ship if the steady-state estimate still justifies the change |
When the fading effect is still worth shipping
A decaying effect is not automatically a worthless one, and the analysis should not pretend otherwise.
Kohavi and colleagues describe an MSN experiment where a link opened in a new tab: the change had a real novelty component, with 20 percent of feedback messages on the first day concerning the feature, falling to 4 percent in week two and 2 percent by weeks three and four, and the improvements to key metrics were sustained over time. Novelty and durable value are not mutually exclusive.
The practical rule is to decide on the steady-state estimate and to record the initial spike separately as what it is: a one-time effect that will not repeat. In our example, if weeks three and four had come back at a clear and significant +3.5 percent instead of an inconclusive one, shipping would be correct and the plan should carry 3.5 percent, not 8.5 percent. The error is not shipping a change with a decaying effect. The error is forecasting the peak.
There is one situation where the pattern is expected rather than suspicious: recommendation and personalization changes, where a new algorithm introduces one-time diversity or exhausts a finite pool of items to recommend. Kohavi and colleagues cite the example of a changed people-you-may-know algorithm, where the new ordering evaluates better at first and the effect dies down once the top recommendations have been consumed. If you are testing in that territory, plan a longer window from the start.
Automate this with Donnu
The novelty argument is expensive because the evidence usually has to be reconstructed by hand after the review has already formed an opinion. Donnu A/B keeps the weekly effect visible next to the cumulative one instead of only showing the running total, holds the new versus returning split as a standing segment rather than an ad hoc slice, and records the primary analysis window with the experiment so the decision rule is fixed before the first number arrives.
Start a free 14-day trial and see the weekly shape of your next test, not just its average.
References
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the statement that novelty and primacy effects are uncommon in practice, of the recommendation-system caveat, and of the MSN new-tab experiment with its feedback percentages and sustained metric improvement. exp-platform.com.
- Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T. and Xu, Y. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. KDD 2012. Source of the finding that most suspected primacy and novelty trends are statistical artifacts, and of the observation that no experiment was found in which a significant result reversed direction because of these effects. exp-platform.com.
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. On experiment duration, statistical power and continuous recruitment of users. exp-platform.com.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
Read also: What is A/B testing · How many visitors do you need · Minimum detectable effect · The peeking problem · Free test duration calculator · Leia em português
Frequently asked questions
- What is the novelty effect in A/B testing?
- The novelty effect is a short-lived response to a change simply because it is new. Returning visitors notice something different, explore it, click it, and the metric rises for reasons that have nothing to do with the change being better. It fades as the change stops being new. The mirror image is the primacy effect, where experienced users are temporarily less efficient with a new layout and the change looks worse at first than it will be. Both distort the reading of a short test, and both affect returning visitors far more than new ones.
- How do I know if my result is novelty and not a real win?
- Three checks, in this order. First, split the effect by week instead of reading only the cumulative number, because a cumulative curve hides decay by construction. Second, compare new visitors against returning ones: a novelty effect is concentrated in returning visitors and near zero in new ones, since a first-time visitor has no old version to be surprised by. Third, look at whether the steady-state weeks alone still support the decision. If weeks three and four on their own are inconclusive, you do not have a confirmed win, you have a first week.
- Is the novelty effect common?
- Less common than the folklore suggests. Kohavi, Deng, Longbotham and Xu write that in practice novelty and primacy effects are uncommon, and that the cases they observed concentrate in recommendation systems, where either the one-time diversity of new recommendations or a finite pool of items produces a short-lived boost. Their earlier work goes further: most suspected novelty and primacy trends turn out to be statistical artifacts of reading a cumulative curve early, not real decay. Treat novelty as a hypothesis you test, not as a default explanation for a result you dislike.
- How long should I run a test to rule out novelty?
- Two weeks is the working minimum recommended for exactly this reason, and it should cover whole weeks so that weekday and weekend traffic are balanced. Two weeks lets you compare the first week against the second, which is the cheapest decay check available. If your change targets returning visitors specifically, or your purchase cycle is longer than a week, plan for a longer window and size the test for the effect you expect to remain after the novelty fades, not for the effect you saw on day one.
- What is the difference between novelty and primacy effects?
- They point in opposite directions. Novelty inflates the variation early: users try the new thing because it is new, and the boost decays. Primacy depresses the variation early: experienced users are slower with an unfamiliar layout until they learn it, so the change looks worse than it is. Kohavi and colleagues report that they could not find a single experiment where a statistically significant result in one direction became statistically significant in the other because of these effects, which is a strong argument against extending a clearly negative test in the hope that users will adapt.
- Can I just exclude the first week from the analysis?
- You can, and it is a reasonable robustness check, but only if you decide to do it before you look at the result. Choosing to drop the first week after seeing that it helps your preferred conclusion is a form of peeking dressed as rigor. The cleaner version is to declare in the experiment record that the primary analysis covers the steady-state period, define that period up front, and size the test so that the steady-state window alone carries enough sample to answer the question.
- Does the novelty effect apply to new visitors?
- Almost not at all, and that is what makes the new-versus-returning split such a sharp diagnostic. A first-time visitor has no memory of the previous version, so there is nothing to be surprised by and nothing to relearn. If a large effect in week one lives almost entirely in returning visitors while new visitors show a small and inconclusive difference, novelty is the leading explanation. If both segments move together, the effect is more likely to be real.