Minimum Detectable Effect: How to Choose the MDE
What minimum detectable effect means, how to pick an MDE you can defend, absolute versus relative, and what your traffic can actually resolve.
📚 This article is part of the guide A/B Testing Statistical Significance: Plain-English Guide.
The minimum detectable effect is the smallest improvement your test is built to reliably catch, and it is the input that decides everything else about the experiment: how many visitors it needs, how long it runs, and which real improvements it will simply fail to notice. Most teams choose an MDE without knowing they have chosen one, by picking a duration or accepting a default in a calculator. This guide covers what the MDE actually means, how to pick one you can defend to a skeptical stakeholder, the absolute versus relative trap that silently mis-sizes tests, and how to read what your traffic can and cannot resolve. It is part of our complete guide to A/B testing and pairs with the arithmetic in how many visitors an A/B test needs.
What MDE means, precisely
The MDE is a statement about the experiment, not about the world. Setting a 15 percent relative MDE says: if the true effect were a 15 percent improvement, this design would detect it with the probability given by its power, typically 80 percent. It does not say the effect will be 15 percent, and it does not promise that a 15 percent result will be significant on any given run.
Two consequences follow immediately, and both are routinely missed.
Effects smaller than the MDE are not disproved, they are unresolved. A test sized for 15 percent that meets a true 9 percent effect will usually end with a confidence interval crossing zero. The correct record is “inconclusive at this sample”, not “the change did nothing”.
Power is not certainty. At 80 percent power, one run in five misses a true effect exactly the size you designed for. That is the standard convention, not a flaw, but it means a single inconclusive test on a promising idea is weak evidence against the idea.
Absolute or relative: the same design, two units
An MDE can be stated in percentage points (absolute) or as a share of the baseline (relative). Converting between them is arithmetic, and getting the conversion wrong is one of the fastest ways to size a test an order of magnitude too small.
At a 4 percent baseline, both statements below describe the identical experiment and produce the identical requirement:
| Statement | Meaning | Visitors per variation |
|---|---|---|
| Absolute MDE of 1 percentage point | 4.0 percent to 5.0 percent | 6,745 |
| Relative MDE of 25 percent | 4.0 percent to 5.0 percent | 6,745 |
| Absolute MDE of 0.4 percentage points | 4.0 percent to 4.4 percent | 39,475 |
| Relative MDE of 10 percent | 4.0 percent to 4.4 percent | 39,475 |
The trap is that the two units carry opposite intuitions. “One percentage point” sounds like a modest ask; at a 4 percent baseline it is a quarter of the entire rate, an outcome most page changes never produce. Meanwhile “10 percent better” sounds ambitious while being, at this baseline, a movement of 0.4 percentage points that costs almost six times the traffic.
The same absolute change also costs wildly different amounts depending on where your baseline sits:
| Baseline | Absolute MDE of 1 percentage point | Equivalent relative effect | Visitors per variation |
|---|---|---|---|
| 2 percent | 2.0 to 3.0 percent | 50 percent | 3,826 |
| 4 percent | 4.0 to 5.0 percent | 25 percent | 6,745 |
| 10 percent | 10.0 to 11.0 percent | 10 percent | 14,751 |
| 30 percent | 30.0 to 31.0 percent | 3 percent | 33,275 |
House rule worth adopting: state the MDE in relative terms, and write the absolute equivalent next to it. Relative travels better across pages with different baselines, and writing both in the experiment record makes the ambition impossible to misread later. Set your own numbers here:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
How to choose an MDE you can defend
The MDE is a business decision wearing statistical clothing. Four steps make it defensible.
1. Start from the decision, not the calculator
Ask what improvement would actually change behavior: at what lift do you ship the variation permanently, at what lift do you invest further in this area, below what lift is the change not worth the maintenance cost. Set the MDE just under the smallest of those thresholds. If nobody can name a number, the test does not yet have a purpose, and finding that out before launch is a saving, not a delay.
2. Sanity-check the ambition against the change
Small changes produce small effects. A button color, a microcopy tweak, a reordered field: these rarely move a conversion rate by 20 percent, so a test sized for a 20 percent MDE on such a change is almost guaranteed to end inconclusive. A full page rewrite, a different offer, a removed step in the flow: these plausibly can. Match the ambition of the MDE to the ambition of the change, or expect to learn nothing.
3. Price it in traffic and calendar time
Now run the arithmetic, because this is where most MDEs die. Sample scales with roughly one over the square of the effect.
Two-proportion normal approximation, traffic split evenly across variations. The date uses your timezone and updates live.
| Visitors per variation available | Relative MDE it can resolve at a 4 percent baseline | Absolute equivalent |
|---|---|---|
| 2,500 | about 42.6 percent | +1.70 percentage points |
| 5,000 | about 29.3 percent | +1.17 percentage points |
| 10,000 | about 20.3 percent | +0.81 percentage points |
| 25,000 | about 12.6 percent | +0.51 percentage points |
| 50,000 | about 8.9 percent | +0.35 percentage points |
| 100,000 | about 6.2 percent | +0.25 percentage points |
Read that table the other way round and it becomes a planning tool: your traffic already chose an MDE for you. The design question is only whether you accept that number or change something else about the test.
4. Write it down before launch
The MDE, the baseline it assumes, the sample it implies and the planned end date belong in the experiment record before the first visitor is randomized. That single habit is what separates “the test was inconclusive” from “we kept going until it looked good”, and it is the cheapest defense against the peeking problem there is.
Worked example: designed for 15 percent, met with 9.5 percent
A checkout page converts at 4.0 percent. The team wants to detect a 15 percent relative improvement, so 4.0 percent to 4.6 percent, at 95 percent confidence and 80 percent power. The engine returns 17,943 visitors per variation, 35,886 in total, about 13 days at 20,000 eligible visitors a week.
Two ways this can end, using exactly that sample.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
The effect arrives at the size planned. Control finishes with 718 conversions on 17,943 visitors (4.00 percent), the variation with 826 (4.60 percent). Result: z = 2.81, p-value = 0.0050, confidence interval on the difference of +0.18 to +1.02 percentage points, relative improvement +15.0 percent. Significant, as designed.
The effect is real but smaller. Same sample, control at 718, variation at 786 (4.38 percent). That is a +9.5 percent relative improvement, which on a checkout page is a meaningful commercial gain. Result: z = 1.79, p-value = 0.0732, confidence interval -0.04 to +0.79 percentage points. Not significant, and the interval includes zero.
The second outcome is the entire point of understanding MDE. Nothing failed. The design promised to resolve 15 percent effects and was handed a 9.5 percent one. Reading that as “the new checkout does not work” would be a factual error, and shipping the original because of it would throw away a real improvement. The correct record is: inconclusive at this sample, observed direction positive, and a test able to resolve a 9.5 percent effect at this baseline would need roughly 44,000 visitors per variation, which is a decision for the next planning round, not a mid-flight extension.
The MDE belongs to a metric, not to the test
One MDE covers one metric, and most experiments quietly track several. A checkout test typically has a primary metric (completed purchases), a secondary one (add to cart), and at least one guardrail that must not degrade (refund rate, page load time, support contacts). Each of those has its own baseline, so each has its own detectable effect at the same sample size.
That has two practical consequences. First, a test powered for the primary metric is usually underpowered for the guardrails, because guardrails are frequently rarer events with lower baselines. A guardrail that shows no significant degradation in a test sized for purchases has not been cleared, it has simply not been examined with enough sample to say anything. Treat guardrails as directional monitoring with a threshold for alarm, not as passed checks.
Second, reading several metrics for significance multiplies the chance that one of them crosses the line by accident. With five independent metrics at a 5 percent threshold, the probability that at least one produces a false positive is about 23 percent, not 5 percent. The fix is to name the primary metric in the experiment record before launch and let it carry the decision, treating the rest as context. Our guide to common A/B testing mistakes covers the correction methods when you genuinely need to decide on more than one metric at once.
Common MDE mistakes
| Mistake | What it produces |
|---|---|
| Never stating an MDE at all | You still have one, set implicitly by whenever you stopped; you just cannot tell an inconclusive test from a failed idea |
| Confusing percentage points with relative percent | Tests sized several times too small, then read as if they were conclusive |
| Setting the MDE from available traffic, then reporting it as ambition | The design becomes “whatever we could see”, and small real effects get labeled as no effect |
| Pairing a tiny change with a large MDE | An expensive way to guarantee an inconclusive result |
| Extending the test because the p-value is close | Peeking by another name; the guarantee the design offered no longer applies |
| Treating inconclusive as proof of no effect | Kills good ideas and hides the fact that the test was never able to answer |
If your program keeps landing in the last two rows, the structural fix is either variance reduction, which cuts the sample needed for the same MDE, covered in our guide to CUPED, or a design that legitimately allows interim analysis, covered in sequential testing explained.
Automate this with Donnu
An MDE only protects you if it is fixed before the test starts and visible while it runs. Donnu A/B computes the sample your real traffic supports for the effect you chose, keeps the planned end date in view instead of inviting a daily verdict, and reports the confidence interval next to the observed effect so an inconclusive result reads as inconclusive rather than as a near miss. When your traffic cannot resolve the effect you care about, it says so before you spend two weeks finding out.
Start a free 14-day trial and set the MDE for your next test against your own baseline.
References
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on statistical power, minimum detectable effect and experiment design. Companion material at experimentguide.com.
- Gelman, A. and Carlin, J. Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 2014. The formal account of why underpowered designs exaggerate the effects they do detect, via the exaggeration ratio. stat.columbia.edu.
- Deng, A., Xu, Y., Kohavi, R. and Walker, T. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. WSDM 2013. Variance reduction as the way to lower the detectable effect without more traffic. exp-platform.com.
Read also: How many visitors do you need for an A/B test · A/B testing statistical significance · Sequential testing explained · What is CUPED · Free MDE calculator
Frequently asked questions
- What is the minimum detectable effect (MDE)?
- The minimum detectable effect is the smallest difference between variations that your test is built to reliably detect, given its sample size, confidence level and power. It is a property of the experiment design, not a prediction of the result. An MDE of 15 percent relative does not mean you expect a 15 percent lift, it means that if the true effect were 15 percent, this test would catch it about 80 percent of the time at the power you selected, and that smaller true effects would frequently slip through as inconclusive.
- How do I choose a good MDE?
- Start from the decision, not from the statistics. Ask what improvement would be large enough to change what you do, ship it, roll it back, invest more in this area, and set the MDE just below that threshold. Then check the traffic bill: if that MDE needs more visitors than you can collect in a reasonable window, you have a real conflict to resolve openly, either by testing a bigger change, moving to a higher-baseline metric, or accepting that the page is not testable today. What you must not do is quietly raise the MDE to whatever your traffic allows and keep calling the result conclusive.
- What is the difference between absolute and relative MDE?
- Absolute MDE is stated in percentage points, for example moving from 4 percent to 5 percent is 1 percentage point. Relative MDE is stated as a share of the baseline, so that same change is a 25 percent relative improvement. They describe the identical experiment and produce the identical sample size when converted correctly: at a 4 percent baseline, both 1 percentage point absolute and 25 percent relative require 6,745 visitors per variation. Confusion between the two is one of the most common sources of a test being sized ten times too small, because a 1 percentage point change sounds modest while at a 4 percent baseline it is a quarter of the whole rate.
- What happens if the real effect is smaller than my MDE?
- The test usually ends inconclusive, and it is important not to read that as evidence the change did nothing. Concretely, a test sized for a 15 percent relative MDE at a 4 percent baseline needs 17,943 visitors per variation. If the true improvement turns out to be around 9.5 percent, that same test returns a p-value near 0.073 with a confidence interval crossing zero: a real gain the experiment cannot confirm. Recording it as inconclusive, rather than as a failure, is what keeps a program honest, and the interval tells you how much larger the test would have to be.
- Can I lower the MDE after the test has started?
- Not without giving up the guarantee the design provided. Extending a running test because the result is close is a form of peeking: the decision to keep collecting data was made after looking at the data, which inflates the false-positive rate above the level you designed for. The disciplined alternatives are to decide the extension rule before launch, for example a pre-registered two-stage design, or to use a sequential method that is built to allow interim looks with a valid error rate at every point.
- Does a larger MDE make my test better or worse?
- Neither, it makes it different. A large MDE gives you a fast, cheap test that can only see large effects, which is the right design for a redesign or a substantially different offer. A small MDE gives you a slow, expensive test that can resolve subtle differences, which is the right design for a high-traffic page where a 3 percent improvement is worth real money. The mistake is not picking a large MDE, it is picking one implicitly and then interpreting an inconclusive result as proof the change had no value.
- How does MDE relate to sample size and duration?
- Sample size scales with roughly one over the square of the MDE, so halving the effect you want to detect multiplies the visitors required by about four. On a 4 percent baseline, 5,000 visitors per variation resolves about a 29 percent relative effect, 25,000 resolves about 12.6 percent, and 100,000 resolves about 6.2 percent. That quadratic curve is why sizing feels forgiving at the ambitious end and brutal at the subtle end, and why the honest question in most programs is not how long to run but which effects your traffic will never let you see.