Quantile Metrics in A/B Testing: The Median and the p90
Quantile metrics like p90 page load time break the standard formula. Why the naive reading produced 31% false positives and how to read one correctly.
Complete, honest guides on CRO, A/B testing and experimentation: statistics, sample size, test design and what to do with the results. No fluff. (page 8 of 24)
Quantile metrics like p90 page load time break the standard formula. Why the naive reading produced 31% false positives and how to read one correctly.
Equivalence testing proves a variant did not hurt. How to set the margin, read the 90 percent interval and size an A/B test to establish a tie.
Futility stopping uses conditional power to shut down an A/B test that cannot win. How to compute it, where to put the threshold and what the error costs.
Observed power after an A/B test is the p-value rewritten. Why it tells you nothing about a non-significant result, and what to read instead.
Bot traffic only breaks an A/B test when it lands unevenly across variants. How to tell noise from bias, detect it with A/A tests and filter it.
Regression to the mean makes the page you picked for being worst improve on its own. How to separate statistical recovery from a real A/B test effect.
Consent, blockers and browser policy erase part of your data. Tracking loss only biases an A/B test when it hits one variant harder than the other.
Heterogeneous treatment effects: a significant result on mobile and none on desktop does not prove the effect differs. The interaction test explained.
Reading 20 metrics in one A/B test gives a 64% chance of at least one false positive. How Bonferroni and Benjamini-Hochberg control multiple metrics.