Hierarchical Models in A/B Testing: Partial Pooling
Hierarchical models estimate how much segments really differ instead of assuming it. The math across 8 segments, weakly identified tau, rank reversal.
Complete, honest guides on CRO, A/B testing and experimentation: statistics, sample size, test design and what to do with the results. No fluff. (page 4 of 24)
Hierarchical models estimate how much segments really differ instead of assuming it. The math across 8 segments, weakly identified tau, rank reversal.
Interleaving merges two rankings into one list and reads preference from clicks. The method, the sign test, the published gains, and where it fails.
Page speed moves conversion, but most numbers in circulation are folklore. What is provable, how slowdown experiments work, and the real traffic math.
SEO split testing randomizes pages, not users. What Google allows, the statistical error that invalidates most SEO test reports, and the right math.
A Bayes factor measures how far the data moved belief between two hypotheses. The math, Lindley paradox in numbers, and the limits of the method.
Empirical Bayes shrinkage pulls every segment toward the test average. The math across eight segments, the shrinkage factor, and when the top is real.
Value of information prices what an A/B test buys you. The ceiling, the value by sample size, and the cases where the answer is not to test.
A doubly robust estimator is right if the propensity model OR the outcome model is right. The math, four worked scenarios and the hard limit.
Regression adjustment uses pre-experiment covariates to shrink the standard error without more traffic. The math, Freedman critique and Lin fix.