Common Organizational Blockers to A/B Testing
The seven organizational blockers that stall A/B testing programs, why each one survives, and the specific move that removes it. With live calculators.
📚 This article is part of the guide How to Build an Experimentation Culture (2026).
Most A/B testing programs do not stall because the statistics are hard. They stall because of organizational blockers: no traffic to size the test, a release process that treats testing as a gate, a senior stakeholder who reverses results, no owner for the primary metric, an engineering queue, serial approval gates, and a team with no memory of what it already tested. Six of those seven are process problems, which is why replacing the tool almost never unblocks anything. This article walks through each blocker, what keeps it alive, and the specific move that removes it, with live calculators for the two that are actually arithmetic in disguise. It is part of the guide on how to build an experimentation culture.
The seven blockers, and which ones are real
Before working through them one at a time, it helps to see them sorted by what kind of problem they actually are. The distinction matters because the fix is completely different: an arithmetic problem is solved with a calculation and a decision about what you are willing to detect, while a process problem is solved by changing who decides what and when.
| Blocker | How it sounds in the room | What it really is | Where the fix lives |
|---|---|---|---|
| Not enough traffic | “We are too small to A/B test” | Arithmetic, plus an unstated expectation about effect size | Sizing the test and renegotiating the MDE |
| Testing slows us down | “We cannot wait three weeks for every release” | Process: testing applied as a gate instead of a filter | Deciding which changes deserve a test |
| Results get overridden | “The data says B but we are shipping A” | Governance: no rule for when a business veto is legitimate | A decision rule agreed before the test runs |
| No metric owner | “Marketing says it won, product says it did not” | Ownership: two teams reading two different numbers | One named owner per primary metric |
| Engineering queue | “The variation is in the backlog for next quarter” | Process: every variation treated as a product feature | A build path that does not consume sprint capacity |
| Serial approval gates | “Legal has to review each variation” | Process: approval placed after the work instead of around it | Pre-approved guardrails instead of case-by-case review |
| No institutional memory | “Did we test this already? Nobody knows” | Knowledge: results stored by test name, not by question | A repository organized by page and hypothesis |
Only the first is genuinely a statistics problem, and even then not in the way people expect. The rest are decided by how the organization is wired.
Blocker 1: “We do not have enough traffic”
This is the only blocker with a calculable answer, and the answer is usually more nuanced than either side of the argument expects. The sentence “we do not have enough traffic” is almost never true on its own. What is true is “we do not have enough traffic to detect the effect size we implicitly assumed, in the window we implicitly assumed”. Once those two hidden assumptions are stated out loud, the conversation stops being about feelings.
Take a signup flow with a 3.1% baseline conversion rate and 9,200 visitors a week. At 95% confidence and 80% power, with a two-sided test:
| Relative gain you want to detect | Sample per variation | Duration at 9,200 visits a week |
|---|---|---|
| 10% | 51,438 | 79 days |
| 15% | 23,387 | 36 days |
| 25% | 8,796 | 14 days |
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Set the calculator to a 3.1 baseline rate, a 15 minimum detectable effect (relative) and 9,200 visitors per week to reproduce the middle row. The shape of that table is the whole lesson: the required sample scales with roughly the inverse square of the effect you want to detect, so halving your ambition on effect size does far more for feasibility than doubling your patience.
There are three honest exits from this blocker, and one dishonest one.
- Test bigger changes. A redesigned step produces a larger effect than a button color, and a larger effect is cheaper to detect. Low traffic is an argument for bolder variations, not timid ones.
- Move the metric up the funnel. Add-to-cart happens far more often than purchase, so it reaches sample size much faster. The tradeoff is honest and must be stated: you are measuring a proxy, and a proxy can move without revenue following.
- Accept the window. A 36-day test is not a failure, it is a fact about your traffic. What breaks programs is promising a two-week answer and then stopping a 36-day test at day 14.
The dishonest exit is running the test anyway and reading it early. That is covered in detail in the guide on CRO for low-traffic sites, and the next blocker shows the exact shape of the damage.
Blocker 2: “Testing slows us down”
This objection is usually aimed at the wrong target. Testing does not slow a team down; testing everything does. Programs that apply experimentation as a gate on all work end up with a release queue behind a statistical process that was never meant to carry it.
The workable rule is that experimentation is a filter on high-stakes changes, not a gate on every change. Three questions decide whether something deserves a test:
- Is being wrong expensive? A change to the checkout of a store that processes real revenue is a different risk from a change to a footer link.
- Is it hard to reverse? A pricing page change that anchors expectations is harder to walk back than a copy tweak.
- Do reasonable people disagree? If everyone in the room predicts the same outcome, a test buys less information than it costs, unless the stakes are high enough to justify verifying the consensus.
Everything that fails all three should ship without a test. That single rule usually cuts the “testing queue” by more than half, and it removes the most common structural cause of the complaint.
The second cause is running tests one at a time. Independent surfaces can run simultaneous experiments without interfering, and a program that runs three tests in parallel on unrelated flows has three times the learning rate with the same traffic per test. The practical limit is overlap: two tests touching the same funnel step can interact, and that is a design problem worth solving rather than a reason to serialize everything. The guide on how many A/B tests to run per month covers how to set that cadence against real traffic.
The third cause is the one nobody says out loud: the team stops the test early because the deadline arrived, and then discovers the result cannot carry the decision. Here is what that looks like with real numbers. The signup flow above needed 23,387 visitors per variation for a 15% relative effect. Suppose it is stopped at 13,200 per variation because the quarter is closing, with 409 conversions in the control and 478 in the variation:
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Enter 13,200 visitors and 409 conversions for A, and 13,200 visitors and 478 conversions for B. The calculator returns a p-value of about 0.018, a relative lift of about +16.9%, and a 95% confidence interval on the absolute difference running from about +0.09 percentage points to +0.96 percentage points. The test is statistically significant, and it is still a bad basis for a revenue promise: translated to relative terms, the interval stretches from roughly +2.8% to +30.9%. A finance team told “this is worth 17% more signups” is being given the midpoint of a range that includes an almost negligible gain.
That is the honest cost of the shortcut. Not a wrong answer, a uselessly imprecise one, presented with the confidence of a precise one. The fix is not to stop reporting the number, it is to report the interval alongside it, which is covered in the statistical significance guide.
Blocker 3: results get overridden by seniority
Every experimentation program eventually produces a result somebody senior does not like. What happens next determines whether the program survives.
The mistake is treating all overrides as the same thing. Three distinct situations get confused, and only one of them is actually a problem.
| Situation | What is happening | Legitimate? | What to do |
|---|---|---|---|
| Business veto | The decision depends on something the test did not measure: a legal constraint, a brand commitment, a partner agreement | Yes | Ship the override, record the reason in the repository, keep the test result intact |
| Distrust of measurement | The stakeholder suspects the test itself is broken | Yes | Answer with an A/A test or a sample ratio mismatch check, not with an argument |
| Pure disagreement | The result is valid, understood, and reversed because someone senior prefers the other option | No | This is the case that kills programs, and it is a governance failure rather than a personality one |
The acronym for the third case is HiPPO, “highest paid person’s opinion”, popularized by Avinash Kaushik and Ronny Kohavi around 2006. The problem is never that an executive has an opinion. The problem is reversing a published result without a reason outside the metric, because in a single decision it teaches everyone that running the test was theater.
The structural fix is boring and effective: agree the decision rule before the test runs. Who decides, on which metric, at which threshold, and under what conditions a business veto is acceptable. A rule written before anyone knows the outcome is a rule everyone can accept. The same rule written after the result is a negotiation. The article on getting executive buy-in covers how to establish that agreement without turning it into a confrontation.
Blocker 4: nobody owns the primary metric
This blocker is quiet and expensive. It shows up as a meeting where marketing says the test won and product says it did not, and both are right, because they are reading different numbers.
It has one root cause: the primary metric was never named and owned before the test started. When that happens, every stakeholder brings the metric that best matches their goal, and the discussion becomes a negotiation about which number counts, held after everyone already knows which number favors their position.
The fix has three parts, and all three have to happen before the test runs:
- One primary metric per test, named in writing. Not a dashboard, one number.
- One named owner for that metric. A person, not a team.
- Guardrail metrics declared up front. The things that are not allowed to get worse, listed before you can see whether they did.
Guardrails are what let you say no to a winner honestly. A variation that lifts add-to-cart by 8% and drops revenue per visitor by 3% is not a winner, and without a declared guardrail that trade is invisible until somebody notices the revenue chart weeks later.
Blocker 5: the engineering queue
In many organizations the real bottleneck is not statistics or approval, it is that every variation is treated as a product feature, entering the same backlog as everything else. A test idea that takes twenty minutes to design waits six weeks to be built.
The unblocking move is to split test ideas by what they actually touch. Changes to copy, layout, ordering and offer framing live in the presentation layer and can be built and shipped by the experimentation team with a snippet-based tool, with no sprint capacity consumed. Changes that touch pricing rules, the data model or checkout logic genuinely need engineering, and should take that route. In most programs the first category is the large majority of the backlog, which means the queue was never really about engineering capacity.
Two guardrails keep that from becoming a mess: a technical reviewer for anything that touches a page with real revenue on it, and a hard rule that presentation-layer experiments are removed once the test ends, with winners implemented properly in the codebase.
Blocker 6: serial approval gates
Legal review, brand review and compliance review are legitimate. Applying them case by case, after the variation is built, is what turns them into a blocker: every test waits in a queue whose length nobody controls, and the wait is invisible in any velocity metric.
The fix is to move the approval from the artifact to the boundary. Instead of reviewing each variation, agree once on what is pre-approved: which claims can be made, which pages are off limits, which elements always need review. A one-page set of guardrails signed off by legal turns 90% of tests into work that needs no review at all, and concentrates the review effort on the 10% that genuinely carries risk.
| Approval design | What gets reviewed | Typical effect on velocity |
|---|---|---|
| Case by case, after build | Every variation, individually | High and unpredictable wait, invisible in reporting |
| Pre-approved guardrails | The boundary, once; then only exceptions | Most tests need no review; risky ones get real attention |
| No review at all | Nothing | Fast until the first incident, then usually replaced by case-by-case review |
The middle row is the only one that survives contact with a real organization, and it requires an uncomfortable conversation up front rather than a comfortable one repeated forever.
Blocker 7: no institutional memory
The last blocker is the one that compounds. A team that cannot answer “what do we already know about this page” retests refuted hypotheses, argues from anecdote, and loses everything it learned when a person leaves.
The failure mode is usually not the absence of documentation. It is documentation organized around the wrong question. A repository sorted by test name and quarter answers “what did we run in Q3”, which nobody asks. It should answer “what do we already know about this checkout step”, which everyone asks, and that means indexing by page and by hypothesis, not by experiment.
The other half is the habit: consulting the repository has to be the first step of writing a hypothesis, not an optional courtesy. The guide to building an experiment repository nobody ignores covers the structure in detail; the short version is that a repository nobody reads is a cost, not an asset.
The blocker hiding behind all of them: judging the program by win rate
One belief keeps every blocker above alive: the assumption that a good experimentation program mostly wins. It does not, and it is not supposed to.
Ronny Kohavi, Diane Tang and Ya Xu, in “Trustworthy Online Controlled Experiments” (Cambridge University Press, 2020), report that even among well designed and well executed ideas at Microsoft, only about one third actually improved the target metric, roughly a third stayed flat, and the rest made things worse. That is not evidence of a weak team. It is the expected pattern when human intuition tries to predict real user behavior.
An organization that judges its program by win rate will always conclude the program is failing, and will start pressuring for shorter tests, smaller samples and friendlier readings, which is how every blocker above gets reinforced at once. The alternative framing is simple and defensible: the value of the program is the sum of the wins shipped and the losses prevented, and the second half is invisible unless someone reports it deliberately.
| What gets measured | What it rewards | Failure mode |
|---|---|---|
| Win rate | Testing safe, obvious changes | Program looks successful and learns nothing |
| Tests completed per quarter | Volume | Underpowered tests run to hit a count |
| Decisions made with evidence | Using the result, whichever way it points | Requires reporting losses, which needs governance |
| Revenue at risk protected | Catching the changes that would have hurt | Needs the loss estimate recorded at decision time |
Make this automatic with Donnu
Almost every blocker in this article gets worse when the mechanics of testing are expensive. If building a variation takes a sprint, nobody tests; if reading the result requires a spreadsheet, everybody reads the number they prefer. Donnu removes that friction: it runs the variation with a snippet that does not consume sprint capacity, and returns a verdict with the confidence interval attached, holding back the winner call until the variation has at least 200 visitors and 7 days on air, so the conversation about what the result supports starts from the same number for everyone.
Start a 14-day free trial and take one blocker off the list this week. For the full framework around it, see how to build an experimentation culture.
References
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
- Kohavi, R. The Origin of HiPPO: Highest Paid Person’s Opinion. LinkedIn Pulse. linkedin.com/pulse/origin-hippo.
- Thomke, S. Building a Culture of Experimentation. Harvard Business Review, March-April 2020. hbr.org/2020/03/building-a-culture-of-experimentation.
- Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.
- Kohavi, R. ExP Platform: accelerating innovation through trustworthy experimentation. exp-platform.com.
Read next:
Frequently asked questions
- What are the most common organizational blockers to A/B testing?
- Seven show up over and over: not enough traffic to size a test, the belief that testing slows delivery down, results being reversed by seniority, no single owner for the primary metric, an engineering queue that turns every variation into a sprint item, serial approval gates from legal or brand, and a team that cannot remember what was already tested. Six of the seven are process problems rather than statistics problems, which is why buying a better tool almost never fixes them.
- How do you handle the "we do not have enough traffic to test" objection?
- Treat it as an arithmetic question, not an opinion. Compute the sample per variation for the flow in question and see what the traffic can actually detect in a window you accept. On a 3.1% baseline with 9,200 weekly visitors, detecting a 15% relative gain takes 36 days, while a 25% relative gain takes 14 days. The honest move is not to abandon rigor, it is to change what you promise to measure: test bigger changes, pick a metric higher in the funnel, or accept a longer window.
- Does A/B testing slow a team down?
- Only when it is used as a gate on all work instead of a filter on high-stakes changes. A team that tests everything queues its releases behind statistics; a team that tests nothing pays for its mistakes in production. The workable rule is to test where being wrong is expensive or hard to reverse, and ship the rest, with tests running in parallel on independent surfaces rather than one at a time.
- What should you do when an executive overrides a test result?
- Separate the two legitimate cases from the illegitimate one. A business veto for reasons outside the metric (a legal constraint, a brand commitment, a strategy the test did not measure) is legitimate and should be recorded as such. Distrust of the measurement is also legitimate and is answered with an A/A test or a sample ratio mismatch check. Reversing a valid result simply because someone senior disagrees is the case that damages the program, because it teaches the team that running the test was theater.
- Why does a low win rate make executives lose confidence in experimentation?
- Because they are measuring the program by the wrong number. According to Kohavi, Tang and Xu in "Trustworthy Online Controlled Experiments", only about one third of well designed ideas tested at Microsoft actually improved the target metric. A win rate near a third is the expected pattern, not a failure, and the value of the other two thirds is the changes that never reached users. A program judged by win rate alone will always look like it is underperforming.
- How do you keep a team from testing the same refuted hypothesis twice?
- With a repository that is searchable by page and by hypothesis, not by test name, and with one habit: consulting it is the first step of writing a new hypothesis, not a step nobody takes. The failure mode is not the absence of documentation, it is documentation organized in a way that answers "what did we run in Q3" instead of "what do we already know about this checkout step".