Common A/B Testing Mistakes & Validity Threats

A/B testing mistakes rarely announce themselves. A test can run cleanly, produce a tidy-looking p-value, and still be wrong, because the threat to its validity was in how the data was collected or read, not in the math applied afterward. This guide walks through the six mistakes and validity threats that show up most often in real experiment programs, each with the warning sign that gives it away and the concrete fix.

This is the survey-level companion to two deeper guides: our complete guide to A/B testing statistical significance covers the math those mistakes corrupt, and our dedicated peeking problem guide goes deep on mistake #1 below with a full worked example. For the end-to-end process these mistakes tend to derail, see how to run an A/B test step by step.

Statisticians call these “validity threats” rather than simple mistakes because that’s precisely what they are: they don’t make your math wrong, they make the data the math is applied to unreliable. A two-proportion z-test computed correctly on SRM-corrupted data still produces a real, well-formed p-value, it’s just a p-value that answers a question you didn’t actually ask, because the groups it’s comparing aren’t the clean random split you think they are. That’s what makes these threats dangerous: nothing in the output flags that anything went wrong, you have to know to check.

1. Peeking: stopping the moment it looks significant

Checking your test’s dashboard repeatedly and ending it the first time the result crosses your significance threshold is the most common validity threat of all, precisely because it requires no bug, just an understandable habit. Each additional look is a new chance for random noise alone to cross the line, and Evan Miller’s widely cited 2010 analysis showed that continuous monitoring can push a nominal 5% false positive rate to roughly 26% in practice (Miller, 2010).

Warning sign: “It hit significance on day 3, let’s ship it,” said before the planned sample size or duration was reached.

Fix: Decide your sample size and duration in advance, using a baseline rate, minimum detectable effect, significance level, and power, then treat the interim dashboard as informational only. Our dedicated guide to the peeking problem walks through the full math and the stopping rules that do hold up.

2. Sample Ratio Mismatch (SRM)

SRM is when your observed traffic split, say 60/40, doesn’t match your configured split, say 50/50, by more than random chance would explain. Unlike most items on this list, SRM isn’t a statistical subtlety, it’s a signal that something in your randomization, tracking, or redirect logic is actively broken. Fabijan et al. (2019), drawing on data across four companies and hundreds of millions of users, found SRM shows up more often in practice than most teams assume, and traced it to causes ranging from bot traffic and caching bugs to asymmetric redirect logic between variants.

Warning sign: Your intended 50/50 split lands at anything meaningfully off that, with no explanation, and a simple significance test on the split itself (visitors in A vs. visitors in B, treated as its own two-proportion test) comes back significant.

Fix: Check your traffic ratio every time you check results. If SRM is present, the entire test’s data is compromised, however good the primary metric looks, and needs a root-cause fix (bots, caching, redirect asymmetry, bad randomization unit) before you rerun it.

3. Novelty (and primacy) effects

A novelty effect is a temporary lift or drop in a metric that happens simply because a variant is new, not because it’s genuinely better. Returning visitors especially may click around a redesigned page out of curiosity, inflating early engagement metrics that fade once the new version becomes familiar. The mirror image, a primacy effect, is when a change performs worse at first because visitors are unfamiliar with it, then recovers as they adapt.

Warning sign: A strong effect in the first few days that steadily shrinks as the test continues, especially when it’s more pronounced among returning visitors than new ones.

Fix: Segment your results by new versus returning visitors and by day, rather than reading only the cumulative total. If the effect is concentrated early and among returning visitors, extend the test past the novelty window (commonly one to two weeks) and make the decision on the stabilized, later-period data.

4. Simpson’s Paradox

Simpson’s Paradox is when a trend that holds within every subgroup of your data reverses once the subgroups are combined into a single overall number (Stanford Encyclopedia of Philosophy). In A/B testing, it most often shows up when a variant loses in every individual segment, mobile, desktop, each traffic channel, but appears to win in the combined total, because the segments were unevenly weighted between A and B.

A simplified illustration: suppose mobile converts at a lower baseline rate than desktop, and variant B happened to receive a larger share of desktop traffic than variant A did (perhaps due to a channel-specific rollout quirk). B can then look like the aggregate winner purely because it got more of the higher-converting segment, even if A actually converts better within mobile and within desktop individually.

Segment	A conversions / visitors	A rate	B conversions / visitors	B rate
Mobile	40 / 1,000	4.0%	18 / 500	3.6%
Desktop	30 / 500	6.0%	57 / 1,000	5.7%
Combined	70 / 1,500	4.7%	75 / 1,500	5.0%

In this table, A wins within mobile (4.0% vs 3.6%) and within desktop (6.0% vs 5.7%), yet the combined total shows B ahead (5.0% vs 4.7%), purely because B’s traffic mix skewed more heavily toward the higher-converting desktop segment.

Warning sign: A test’s segmented results disagree with its aggregate result, especially when segment traffic shares differ noticeably between A and B.

Fix: Check whether traffic composition (device, channel, geography) was balanced between variants before trusting an aggregate number, and report both the aggregate and the key segments rather than picking whichever tells the story you expected.

5. Testing too many things at once

Changing the headline, the CTA color, the hero image, and the price display all in one variant means that if the test wins, you don’t know which change moved the metric, and if it loses, you don’t know which change (if any) was actually harmful.

Warning sign: A variant bundles several unrelated changes, and the team can’t articulate a single hypothesis explaining why it should perform differently.

Fix: Isolate one variable per test whenever possible. When you genuinely need to test several elements together, use a multivariate test designed to estimate each element’s contribution, rather than an A/B test that only tells you the bundle’s net effect.

6. The multiple comparisons problem

Running many simultaneous tests, or slicing one test’s results into many segments and looking for any significant one, inflates your chance of a false positive the same way peeking does, just across tests or segments instead of across time. If you check 20 independent segments at a 5% significance threshold, you’d expect roughly one of them to look significant purely by chance, even with zero real effect anywhere.

Warning sign: “It wasn’t significant overall, but it was significant for users on Safari in the afternoon,” reported after the fact, with no segment defined in advance.

Fix: Define any segments you plan to analyze before the test starts, and if you are genuinely running many comparisons, apply a correction (such as a stricter significance threshold per comparison) so your overall false positive rate across all of them stays where you intended it.

A quick illustration of the scale of the problem: at a 5% threshold with no correction, checking 5 independent segments carries roughly a 23% chance that at least one comes back significant purely by chance, and that climbs past 40% by 10 segments. A common, simple correction (Bonferroni) divides your significance threshold by the number of comparisons, so 5 segments would each need to clear roughly 0.01 instead of 0.05 to count. It’s a conservative fix, but far better than reporting the one segment that happened to look good and staying silent about the nineteen that didn’t.

Checking your own test against these threats

Two of the checks above, catching a peeking-inflated result and confirming your primary metric’s significance, both start with the same calculation: a two-proportion significance test. Paste your visitor and conversion counts below to get the p-value, lift, and confidence interval your investigation should be built on:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

All six, side by side

Mistake	Warning sign	Fix
Peeking	“It’s significant on day 3, ship it”	Fix sample size and duration in advance; decide only at the endpoint
Sample Ratio Mismatch	Traffic split meaningfully off the configured ratio	Test the split itself for significance; find and fix the root cause before trusting any metric
Novelty / primacy effect	Strong early effect that fades over 1 to 2 weeks	Segment by day and by new vs. returning visitor; decide on stabilized late-period data
Simpson’s Paradox	Segmented results disagree with the aggregate result	Check traffic composition balance between variants; report segments alongside the aggregate
Testing too much at once	Variant bundles several unrelated changes	Isolate one variable per test, or use a proper multivariate design
Multiple comparisons	A segment found significant only after the fact, with no plan to check it	Pre-register segments; apply a stricter threshold when checking many comparisons

Do this automatically on Donnu

Every mistake above is avoidable with process, not more statistics knowledge: a pre-committed sample size, an SRM check, and honest segment reporting catch nearly all of them. Donnu builds the sample-size lock, SRM monitoring, and Bayesian significance reporting into the platform by default, so a validity threat gets flagged before it quietly becomes a wrong decision.

Start a free 14-day trial and run your next experiment on a foundation that catches these automatically. For the deepest look at mistake #1, see our peeking problem guide, and for the full process these mistakes tend to derail, see how to run an A/B test step by step.

References

• Miller, E. How Not To Run an A/B Test. 2010. evanmiller.org.
• Fabijan, A. et al. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. KDD 2019. microsoft.com/research.
• Simpson’s Paradox. Stanford Encyclopedia of Philosophy. plato.stanford.edu.
• Kohavi, R., Tang, D. & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion site at experimentguide.com.