Equivalence Testing: Proving a Variant Did Not Hurt
Equivalence testing proves a variant did not hurt. How to set the margin, read the 90 percent interval and size an A/B test to establish a tie.
Complete, honest guides on CRO, A/B testing and experimentation. The most complete articles on the web, no fluff. (page 2 of 17)
Equivalence testing proves a variant did not hurt. How to set the margin, read the 90 percent interval and size an A/B test to establish a tie.
Futility stopping uses conditional power to shut down an A/B test that cannot win. How to compute it, where to put the threshold and what the error costs.
Observed power after an A/B test is the p-value rewritten. Why it tells you nothing about a non-significant result, and what to read instead.
Bot traffic only breaks an A/B test when it lands unevenly across variants. How to tell noise from bias, detect it with A/A tests and filter it.
Regression to the mean makes the page you picked for being worst improve on its own. How to separate statistical recovery from a real A/B test effect.
Consent, blockers and browser policy erase part of your data. Tracking loss only biases an A/B test when it hits one variant harder than the other.
Heterogeneous treatment effects: a significant result on mobile and none on desktop does not prove the effect differs. The interaction test explained.
Reading 20 metrics in one A/B test gives a 64% chance of at least one false positive. How Bonferroni and Benjamini-Hochberg control multiple metrics.
Sample size for revenue per user comes from the coefficient of variation, not the conversion rate. The continuous formula and the common mistake.