Synthetic Control: The Control Group That Never Existed
With only one treated market, the synthetic control method builds a control by weighting several markets. How to fit it, validate it, and where it breaks.
Complete, honest guides on CRO, A/B testing and experimentation: statistics, sample size, test design and what to do with the results. No fluff. (page 6 of 24)
With only one treated market, the synthetic control method builds a control by weighting several markets. How to fit it, validate it, and where it breaks.
A user who joined yesterday never had seven days to stay. How cohort maturity distorts D7 and D30 retention in A/B tests, and how to close the cohort.
Conversion lag makes an early read reward whoever converts fastest, not whoever converts most. How to measure your arrival curve and when to read the test.
When the metric is a duration and not a rate, censoring breaks the average. How to read time to event in A/B tests with Kaplan-Meier, RMST and log-rank.
Differential attrition makes the conditional mean lie. How Lee bounds fence in the real effect when the treatment changes who is left in the analysis.
Intention to treat measures the effect on who was assigned, not on who used the feature. Why per-protocol overstated the result 2.4x, and how to fix it.
Surrogate metrics trade the long outcome for early signals. How to validate a proxy, why correlation is not enough, and the test that exposes the failure.
Cluster randomization in A/B testing: assigning accounts and analysing users pushes false positives from 5 to 14.5 percent. The design effect fixes it.
Meta-analysis of A/B tests: pool 12 experiments into one estimate, measure heterogeneity with I squared, correct the 1.34x winner exaggeration.