Multiple Metrics in One A/B Test: Controlling FDR
Reading 20 metrics in one A/B test gives a 64% chance of at least one false positive. How Bonferroni and Benjamini-Hochberg control multiple metrics.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
A modern A/B test does not report one metric, it reports dozens. And every metric read at the 5 percent level is an independent draw with a 5 percent chance of flagging an effect that does not exist: with 20 independent metrics and no real effect anywhere, the chance of at least one false positive is 64.15 percent. This guide covers the exact arithmetic, the difference between controlling FWER and controlling FDR, the Benjamini-Hochberg procedure step by step over a real 20-metric panel, and the practical decision of which correction belongs in which part of the report. It is part of our complete A/B testing guide and is the metric-axis counterpart to what testing multiple variants covers on the variant axis.
Multiple metrics are a family of tests
Multiple comparison correction is usually taught with variants: an A/B/n test with 4 arms makes 3 comparisons against control, and everyone accepts that needs adjusting. The metric case is bigger and gets less attention, because the panel grows by convenience rather than by decision.
Kohavi, Deng, Frasca, Walker, Xu and Pohlmann describe exactly this in Bing’s operation: they report results not on one metric but on hundreds, mostly to aid debugging and analysis. Their organizational answer to the multiple outcomes issue was to standardize the success criteria on a small set of metrics, such as sessions per user. In other words: the primary fix was not statistical, it was shrinking the family.
The ceiling arithmetic is simple. If k metrics are independent and none has a real effect, the probability that none is significant at level alpha is 1 minus alpha, raised to k. The probability of at least one is the complement:
| metrics read at the 5% level | chance of at least one false positive |
|---|---|
| 5 | 22.62% |
| 10 | 40.13% |
| 20 | 64.15% |
| 50 | 92.31% |
| 100 | 99.41% |
| 200 | 99.996% |
The honest caveat is worth recording: metrics from one experiment are not independent. Menu clicks and product page views rise together. Correlation lowers the real false positive probability relative to the curve above. What correlation does not do is push it anywhere near 5 percent, and the direction of the error is always the same: the whole panel carries a higher error rate than the nominal rate of each line.
Familywise error and false discovery proportion
There are two different ways to say “I controlled the error”, and choosing between them is a business decision, not a statistical one.
FWER (familywise error rate) is the probability of committing at least one Type I error in the whole family. Controlling FWER at 5 percent means: if nothing has an effect, my chance of announcing anything is 5 percent. It is the right ruler when a single false claim already ruins the decision.
FDR (false discovery rate) was defined by Benjamini and Hochberg in 1995 as the expected proportion of erroneously rejected hypotheses among all rejected hypotheses. Controlling FDR at 10 percent means: among everything I announce, I expect at most 10 percent to be noise. It is the right ruler when you are exploring a panel and a few false discoveries among several true ones are an acceptable cost.
The authors prove two properties linking the rulers, and they are the entire argument for FDR:
- When every null hypothesis is true, FDR and FWER are equivalent. In that case the number of true rejections is zero, so the proportion of errors is either 0 or 1, and its expectation becomes the probability of at least one error. Controlling FDR implies controlling FWER in the weak sense.
- When some null hypotheses are false, FDR is smaller than or equal to FWER, and the two can be quite different. Any procedure that controls FWER also controls FDR, but a procedure that controls only FDR can be less stringent, and that is where the power gain comes from. The authors further note that the larger the number of non-true null hypotheses, the larger the gap between the two rates, and therefore the larger the potential gain.
The Benjamini-Hochberg procedure, step by step
The procedure fits in three lines. Let m be the p-values sorted from smallest to largest, and q the false discovery rate you accept:
- Sort the p-values: P(1) less than or equal to P(2), and so on through P(m).
- Find the largest index i such that P(i) is less than or equal to i divided by m, times q.
- Reject every hypothesis from position 1 through i.
Benjamini and Hochberg prove, in Theorem 1 of the paper, that for independent test statistics and for any configuration of non-true null hypotheses, this procedure controls FDR at q. They also record, in an explicit remark, that independence among the statistics corresponding to the false null hypotheses is not needed for the proof.
Step 3 is the part almost everyone gets wrong. The procedure is step-up: the decision comes from the largest index satisfying the inequality, and every position below it is rejected along with it, even if one of them would not clear its own threshold in isolation. Comparing line by line and rejecting only the ones that pass is a different, more conservative procedure, and it is not Benjamini-Hochberg. The worked example below lands exactly on that case.
Worked example: 20 metrics, one test
A store runs a test with 60,000 users per arm and reports 20 metrics. Every p-value below comes from the two-sided two-proportion z test on the raw counts, and you can reproduce any row by pasting the four numbers into the calculator.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
| metric | A (of 60,000) | B (of 60,000) | z | p-value | relative lift |
|---|---|---|---|---|---|
| Overall conversion | 3,000 | 3,126 | 1.6526 | 0.098418 | +4.20% |
| Add to cart | 9,000 | 9,210 | 1.6897 | 0.091091 | +2.33% |
| Checkout started | 6,000 | 6,132 | 1.2640 | 0.206225 | +2.20% |
| Account signup | 4,200 | 4,116 | -0.9548 | 0.339674 | -2.00% |
| Search usage | 15,000 | 15,390 | 2.5889 | 0.009629 | +2.60% |
| Menu click | 21,000 | 20,790 | -1.2725 | 0.203211 | -1.00% |
| Product page visit | 30,000 | 30,450 | 2.5981 | 0.009373 | +1.50% |
| Coupon applied | 2,400 | 2,478 | 1.1402 | 0.254199 | +3.25% |
| Wishlist add | 1,800 | 1,746 | -0.9205 | 0.357296 | -3.00% |
| Reviews click | 5,400 | 5,508 | 1.0845 | 0.278126 | +2.00% |
| Newsletter signup | 1,200 | 1,266 | 1.3429 | 0.179292 | +5.50% |
| Chat opened | 900 | 858 | -1.0091 | 0.312915 | -4.67% |
| Shipping info click | 7,200 | 7,092 | -0.9625 | 0.335784 | -1.50% |
| Category filter | 12,000 | 12,180 | 1.2954 | 0.195179 | +1.50% |
| Share | 600 | 633 | 0.9447 | 0.344833 | +5.50% |
| Footer click | 3,600 | 3,528 | -0.8793 | 0.379229 | -2.00% |
| Login | 10,800 | 10,692 | -0.8131 | 0.416165 | -1.00% |
| Second session in 7 days | 8,400 | 8,568 | 1.3919 | 0.163962 | +2.00% |
| Return requested | 780 | 822 | 1.0564 | 0.290777 | +5.38% |
| Support contact | 1,500 | 1,443 | -1.0638 | 0.287406 | -3.80% |
Read line by line at the 5 percent level, the panel delivers two wins: product page visit (p equal to 0.009373) and search usage (p equal to 0.009629). And it delivers a quiet defeat nobody puts on the slide: the primary metric, overall conversion, is not significant (p equal to 0.098418).
Applying both corrections
Sorting the 20 p-values and applying Bonferroni (fixed threshold of 0.05 divided by 20, that is 0.0025) and Benjamini-Hochberg with q equal to 0.10 (moving threshold of i divided by 20, times 0.10):
| position | metric | p-value | Bonferroni threshold | BH threshold | P(i) below the BH threshold? |
|---|---|---|---|---|---|
| 1 | Product page visit | 0.009373 | 0.00250 | 0.00500 | no |
| 2 | Search usage | 0.009629 | 0.00250 | 0.01000 | yes |
| 3 | Add to cart | 0.091091 | 0.00250 | 0.01500 | no |
| 4 | Overall conversion | 0.098418 | 0.00250 | 0.02000 | no |
| 5 | Second session in 7 days | 0.163962 | 0.00250 | 0.02500 | no |
| 6 | Newsletter signup | 0.179292 | 0.00250 | 0.03000 | no |
| 7 | Category filter | 0.195179 | 0.00250 | 0.03500 | no |
| 8 | Menu click | 0.203211 | 0.00250 | 0.04000 | no |
| … | (the remaining 12 rows) | above 0.20 | 0.00250 | up to 0.10000 | no |
The largest index satisfying the inequality is i equal to 2. So the procedure rejects positions 1 and 2, that is both, even though position 1 (0.009373) sits above its own threshold (0.00500). That is step-up in action: line 1 is carried by line 2.
| ruler applied | metrics declared significant |
|---|---|
| raw p-value below 0.05 | 2 |
| Bonferroni (alpha divided by 20) | 0 |
| Benjamini-Hochberg (q equal to 0.10) | 2 |
Three readings of the same panel, three different answers, and none of them is “wrong”: they control different things. Bonferroni says that if you require less than a 5 percent chance of making any false claim in this panel, there is nothing to claim. Benjamini-Hochberg says that if you accept up to 10 percent of your claims being noise, these two pass.
What the example does not change
Neither correction makes overall conversion significant, and neither turns “search usage went up” into a reason to ship. The honest reading of the panel is: the test did not move the primary metric, and two secondary metrics showed signal that survives FDR control but not FWER control. That is a hypothesis for the next round, not a result.
The problem also grows through time and variants
Metrics are one of three multiplicity axes, and the three multiply each other.
Kohavi and coauthors quantify the iterations axis precisely. Assuming the feature does nothing, running k iterations makes the probability of statistical significance (positive movement in a two-sided test) grow from 2.5 percent to 1 minus 0.975 raised to k. They record that if a team tries five treatments, the 2.5 percent rate becomes 12 percent; and that six iterations of five-treatment experiments give more than a 50 percent chance of getting a positive statistically significant result. Reproducing the arithmetic: 1 minus 0.975 to the power of 5 is 11.89 percent, and 1 minus 0.975 to the power of 30 is 53.21 percent.
The two mechanisms they use against this are explicit and worth copying: threshold adjustment (requiring lower p-values on projects with multiple treatments or iterations) and a replication stage (once the funnel narrows, a final run, preferably with higher statistical power, that determines the result). In the same paper, for slice-and-dice analysis by segment, they report educating experimenters about false positives and encouraging them to adjust the probability threshold, focusing on strong signals and smaller p-values, on the order of 0.0001.
Kohavi, Deng, Longbotham and Xu raise the same alarm from the volume side: because they run thousands of experiments a year, a false positive rate of 0.05 implies hundreds of false positive results for a given metric, and that is exacerbated when multiple uncorrelated metrics are used.
Which ruler for which part of the report
The practical answer is not to pick one correction for everything, it is to split the report into families with different rules, and declare that split before running.
| part of the report | question it answers | recommended ruler |
|---|---|---|
| Primary metric (1 metric) | does this test win? | no correction; the family has size 1 by construction |
| Secondary success metrics | what else moved with it? | Benjamini-Hochberg with q between 0.05 and 0.20 |
| Guardrail metrics | did this break anything? | sensitive alerting, no correction that reduces sensitivity |
| Segment slices | who did it work for? | pre-declared in the plan, or labelled as exploration |
| Launch decision across variants | which arm ships? | FWER correction (Bonferroni or similar) |
The guardrail metrics row usually surprises people, and it deserves the detail. A guardrail metric exists to detect harm, not to detect gain. The useful null hypothesis there is “nothing got worse”, and the expensive error is Type II: letting a real regression through. Correcting for multiple comparisons makes it harder to reject the null, which increases exactly that error. Applying Bonferroni to the guardrail panel is optimizing against the panel’s own purpose. Its proper treatment is in guardrail metrics.
The primary metric row deserves a note too. It only has size 1 if you declared which one it is before looking. Picking the primary after seeing the p-values is the case where the family has size 20 and you are pretending it has size 1. That is the entire reason the pre-registered analysis plan exists.
The honest counterargument
There is a respectable line that disagrees with this entire framing, and leaving it out would be dishonest. Gelman, Hill and Yajima argue that the multiple comparisons problem can disappear entirely when viewed from a hierarchical Bayesian perspective. Their argument has two parts:
- They challenge the Type I error paradigm itself, because they rarely believe a null hypothesis can be strictly true.
- They hold that the problem is not multiple testing but insufficient modeling of the relationship between the corresponding parameters. A multilevel model performs partial pooling (estimates and intervals are shifted toward each other), whereas classical procedures keep the centers stationary and widen the intervals. In their account, comparisons made with multilevel estimates become appropriately more conservative without sapping power the way many traditional methods do.
Those same authors do acknowledge that FDR control methods make particular sense in fields where one expects a handful of real effects among a vast quantity of zero effects, and that FDR leads to a less conservative procedure with respect to Type I error but is more powerful at detecting real effects.
The practical translation for A/B testers: if your engine is Bayesian, modeling the hierarchy of metrics is a legitimate and probably better path. What is not legitimate is using that argument as a licence to read 20 raw p-values and pick whichever pleased you. The Bayesian treatment of test decisions is in Bayesian A/B testing and expected loss.
Checklist before reading the panel
- Is the family of tests declared in writing? How many metrics, how many arms, how many segments, all before the first look.
- Is the primary metric named on its own? One, not “the three main ones”.
- Does the secondary panel have a declared ruler? Benjamini-Hochberg with q chosen beforehand, not after.
- Are guardrails outside the correction? They need sensitivity, not stringency.
- Are the segments that will be read listed? An unlisted slice is exploration and ships labelled as such.
- Has anyone counted how many iterations this theme has had? The 1 minus 0.975 to the k arithmetic accumulates across rounds.
- Is a replication stage planned? It is the most reliable mechanism and the easiest to forget.
Common mistakes
- Defining the family after seeing the results. Every correction assumes the denominator was chosen up front. Choosing it afterward voids the guarantee.
- Applying Benjamini-Hochberg line by line. Comparing each p-value against its own threshold and rejecting only those that pass is not the procedure. The correct version finds the largest index that passes and rejects everything below it.
- Applying Bonferroni to the whole panel and concluding nothing ever works. With 100 metrics the threshold becomes 0.0005 and the panel goes permanently mute. That signals a badly defined family, not rigor.
- Correcting guardrails. It raises the chance of letting real harm through.
- Counting only metrics and forgetting segments and iterations. The axes multiply.
- Treating q and alpha as the same thing. A q of 0.10 on FDR is not “looser than 0.05”: it is a different quantity, with a different numerator and denominator.
- Announcing the secondary metric that passed as if it were the test result. Surviving FDR qualifies the metric for a confirmation round, not for the victory slide.
Make this automatic with Donnu
The expensive error here is not picking the wrong correction, it is a panel that never makes clear how many questions were asked. Twenty metrics in a table, all with the same confidence interval and the same color, hide the fact that a 5 percent ruler designed for one question is being used on twenty.
At Donnu, the primary metric is declared at experiment creation and stays visually separate from the secondary panel, which carries the Benjamini-Hochberg adjusted p-value alongside the raw one, with the q used written into the table. Guardrail metrics live in their own panel, with their own alert threshold and no correction that reduces their sensitivity. If you want to redo any row by hand, the significance calculator takes the raw counts and the multiple variants calculator handles the arm axis.
References
- Benjamini, Y. and Hochberg, Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, Series B, volume 57, number 1, pages 289 to 300, 1995. Source of the FDR definition as the expected proportion of rejected null hypotheses that were erroneously rejected, of the four-cell error table, of the two properties (FDR equals FWER when every null is true, implying weak FWER control; and FDR is less than or equal to FWER when some nulls are false, with the potential power gain growing with the number of non-true nulls), of the step-up procedure (the largest i with P(i) less than or equal to i over m times q, rejecting every position through i), of Theorem 1 establishing FDR control at q for independent statistics, and of the remark that independence of the statistics corresponding to the false nulls is not needed for the proof. math.tau.ac.il.
- Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013. Source of the report that Bing reports results on hundreds of metrics, mostly to aid debugging and analysis, and that the answer to the multiple outcomes issue was standardizing the success criteria on a small set of metrics such as sessions per user; of the 1 minus 0.975 to the k formula for k iterations and the numbers derived from it (five treatments lifting 2.5 percent to 12 percent, and six iterations of five-treatment experiments passing 50 percent); of the two protection mechanisms (threshold adjustment and a replication stage with higher statistical power); of the guidance to use smaller p-values, on the order of 0.0001, in slice-and-dice analysis by segment; and of the use of an Empirical Bayesian False Discovery Rate control algorithm in interaction detection, in a system that sometimes runs hundreds of thousands of hypothesis tests. exp-platform.com.
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the observation that, running thousands of experiments a year, a false positive rate of 0.05 implies hundreds of false positive results for a given metric, a situation exacerbated when multiple uncorrelated metrics are used, and of the distinction between false positives expected from the statistics and false positives caused by bad design, data anomalies or instrumentation errors. exp-platform.com.
- Gelman, A., Hill, J. and Yajima, M. Why We (Usually) Don’t Have to Worry About Multiple Comparisons. 2009. Source of the counterargument: the challenge to the Type I error paradigm, the thesis that the problem is insufficient modeling of the relationship between parameters rather than multiple testing, the contrast between partial pooling (multilevel models shift estimates toward each other) and the widened intervals with stationary centers of classical procedures, and the acknowledgement that FDR control is less conservative on Type I error and more powerful at detecting real effects, making particular sense when few real effects are expected among many null ones. arxiv.org/abs/0907.2478.
Read also: Multiple variants without false positives · Guardrail metrics · Pre-registered analysis plan · Primary metric and OEC · The peeking problem · Significance calculator · Leia em português
Frequently asked questions
- What is the chance of a false positive when I read 20 metrics in one A/B test?
- If the 20 metrics were independent and none had a real effect, the chance that at least one lands below a p-value of 0.05 is 1 minus 0.95 to the power of 20, that is 64.15 percent. With 5 metrics it is 22.62 percent, with 10 it is 40.13 percent and with 50 it is 92.31 percent. In practice the metrics in one test are correlated and the real number sits below that ceiling, but the order of magnitude holds: reading many metrics without correction almost guarantees finding something.
- What is the difference between FWER and FDR?
- FWER is the probability of committing at least one Type I error across the whole family of tests. FDR, defined by Benjamini and Hochberg in 1995, is the expected proportion of erroneously rejected hypotheses among all rejected ones. The authors show that when every null hypothesis is true the two coincide; when some are false, FDR is smaller than or equal to FWER, which is why procedures that control only FDR can be less stringent and gain power.
- How does the Benjamini-Hochberg procedure work?
- Sort the m p-values from smallest to largest. Find the largest index i for which the p-value in position i is less than or equal to i divided by m, times the target rate q. Reject every hypothesis from position 1 through that position i. It is a step-up procedure: the decision comes from the largest index that passes, and all positions below it are rejected together, even if one of them would not clear its own threshold on its own. Benjamini and Hochberg prove this controls FDR at q for independent test statistics.
- Bonferroni or Benjamini-Hochberg for A/B test metrics?
- It depends on the cost of the error. Bonferroni compares each p-value against alpha divided by the number of tests and controls FWER: use it when a single false claim already compromises the decision, typically on the primary metric of a launch decision. Benjamini-Hochberg controls the proportion of false discoveries and suits the exploratory reading of the secondary metric panel, where a few false discoveries among many true ones are tolerable. Both approaches require declaring the family of tests before looking at the data.
- Do guardrail metrics belong in the multiple comparison correction?
- They should not enter the same way. A guardrail metric asks an inverted question: you are not trying to detect an effect, you are trying to rule out harm. Correcting for multiple comparisons makes it harder to reject the null, which raises the chance of letting a real regression through. Guardrails call for sensitive alerting and their own threshold, discussed separately, not the same correction applied to success metrics.
- Does anyone argue against correcting for multiple comparisons?
- Yes, and the argument is serious. Gelman, Hill and Yajima argue that within a Bayesian multilevel model the multiple comparisons problem can disappear entirely: the model shifts estimates toward each other (shrinkage) instead of holding the centers fixed and widening the intervals, which they say yields more efficient estimates, especially where group-level variation is low, which is exactly where multiple comparisons are a particular concern. It is an alternative to the Type I error paradigm, not a licence to read 20 raw metrics and pick the one you liked.