How to Build an Experimentation Culture (2026)

Experimentation culture is the organizational system that makes a company test relevant decisions by default, not through the isolated initiative of someone enthusiastic. It has a regular cadence, defined roles, a learning repository and executive sponsorship that protects the process instead of vetoing uncomfortable results. That is very different from “running an A/B test now and then”, which is what most companies actually do.

The distinction matters because the A/B testing tool was never the bottleneck. Any team can install a sample size calculator and a significance calculator in an afternoon. The bottleneck is organizational: who decides what gets tested first, what happens when an executive disagrees with the result, what to do with a test that lost, and how to know whether the program is accelerating or just spinning. This guide is about that layer, not about the statistics of a single test (for that, see the complete A/B testing guide and the statistical significance guide).

It is worth saying up front what this guide is not. It is not a generic “be more data driven” manifesto, a phrase any company can sign at zero cost. It is a set of concrete, verifiable decisions: there either is or is not a written prioritized backlog, there either is or is not a person with a formal mandate to block a badly designed test, there either is or is not a record of losing tests that any new team member can consult before proposing the same idea again. Each of those questions has an objective yes or no answer, and that objectivity is what separates culture from intention.

What separates a mature experimentation culture from “testing now and then”

The difference is not the number of tests run in one isolated year. It is three axes that hold with or without a specific person on duty.

Cadence. In a mature culture there is an explicit minimum expectation: for example, every relevant product flow has at least one test running or queued each cycle. In a reactive operation, the rhythm depends on someone remembering. When that person goes on vacation, changes teams or changes priorities, experimentation simply stops. That is the most reliable sign that what exists is individual enthusiasm, not culture. A simple way to test this in your own company: ask what would happen to experimentation if the person who talks about testing the most left tomorrow. If the answer is “it would probably stop”, the program depends on a person, not on a system.

Governance. A mature culture has ready answers for boring questions: who prioritizes what enters the queue, who has the authority to block a badly designed test before it ships, and under what conditions a result can be contested after the fact. In a reactive operation, those questions only surface once a conflict has already happened, usually between a test result and a senior opinion, and at that moment the rule is invented in a hurry, under pressure, which almost always favors whoever has more power in the room instead of whoever has more evidence.

Who decides. This is the most sensitive point and the one that appears least in statistics manuals. In a mature program, most rollout decisions are routine and belong to whoever already has the mandate for them (normally the owner of the experimentation backlog); only high-risk or high-reversal-cost decisions escalate to an executive sponsor, and even then the discussion is about business context, not about reinterpreting the statistics by hand. In a reactive operation, every test becomes a political negotiation about who “wins” the conversation, and the statistical result becomes just one argument among others instead of the criterion that decides.

The flow below shows how those roles fit together, from hypothesis to rollout:

Executive sponsorship: what actually motivates leadership

The most common mistake when asking leadership for support is selling experimentation by volume: “we want to run more tests”. That sounds like process expense, not gain. The argument that tends to work is different: experimentation reduces the risk of an expensive wrong decision, it does not increase the amount of work.

Every executive has lived through the painful version of this: a months-long redesign, approved with conviction, that shipped worse than the previous version, or a pricing change that seemed obvious and sank conversion. Experimentation culture is the institutional answer to that pain, not a product team initiative. The pitch that works usually has three pieces:

• Cost of being wrong without a test. What would it cost, in revenue or reputation, to ship the wrong version to 100% of the base at once, without checking first with a slice of traffic?
• Cost of the test. Running the test costs a fraction of traffic for a few weeks, not the whole project budget. It is cheap insurance against an expensive mistake.
• What changes in the decision. The executive keeps deciding what matters (which problem to attack, which risk is worth taking); the test only swaps “I think” for “the data show”, on the part that can be measured before a full launch.

It is worth reinforcing what executive sponsorship should not be: individual approval of each test, or the power to reverse a result because it contradicts an intuition. When that happens, the entire program loses credibility quickly, because the team learns that the result only matters when it agrees with whoever is in charge. We return to that point in the section on organizational blockers.

In practice, the conversation that works in a budget meeting is short: instead of asking for “budget for an A/B testing tool”, ask for a mandate. Something like “we want no high-risk change to the payment flow going to 100% of users without first passing through a smaller slice of traffic”, followed by the approximate cost of running that slice for two or three weeks. That is a sentence about risk, budget and timeline, three things every executive already handles daily, and not a sentence about statistics, which tends to lose the room halfway through.

Roles and responsibilities in an experimentation program

Mature programs clearly distribute four functions that, in a reactive operation, tend to pile up (badly) on a single person: who prioritizes what enters the queue, who executes technically, who audits the statistics with technical veto power, and who decides the business rollout. Mixing those functions is a common source of bias: the same person who had the idea should not be the only one validating whether the result is statistically sound.

Role	Prioritizes the backlog	Executes the implementation	Audits the statistics	Decides the rollout
Executive sponsor	No	No	No	Ratifies high-risk strategic decisions
Backlog owner (PM or Growth lead)	Yes	No	No	Decides routine rollouts
Statistical guardian (data analyst)	No	No	Yes, with technical veto	Does not decide, but can block publication
Engineering	No	Yes	No	No
Design/UX	Advisory (proposes hypothesis and variation)	Advisory	No	No

The role most often missing in reactive operations is the statistical guardian with real technical veto: someone with formal authority to block the publication of a result when the sample is insufficient, when there was uncontrolled peeking, or when the SRM check points to a broken traffic split. Without that veto, the pressure to ship always beats the statistics, because nobody has an explicit mandate to say no.

Separating those four roles also resolves a silent bias: the person who had the idea for the test is, statistically, the least suited to validate alone whether their own result is sound. Not out of bad faith, but because everyone reads their own result hoping it confirms their own hypothesis. Small teams usually object that they do not have enough people to split four roles across four humans, and that is fair, but the point is not four people: it is four explicit hats, even if the same person wears two of them at different moments of the process, as long as the statistical audit happens at a moment separate from defending your own idea.

How to measure the experimentation velocity of a team

Experimentation velocity is how many tests, with real statistical rigor, a team can run per month. It is the capacity metric of a program, equivalent to what Optimizely calls “experiment velocity”: according to the company, velocity works as a proxy for program maturity, and five factors tend to raise it: a culture that accepts learning from losing tests, collaboration across teams (no silos duplicating the same test), automation of repetitive processes, alignment of the backlog with strategic priorities, and data infrastructure without an analysis bottleneck (Optimizely, “Accelerating Growth Through Experiment Velocity”). The same five points, inverted, are the main obstacles: resource constraints, risk aversion, disconnected teams, process complexity and data bottlenecks.

Velocity has two very distinct components, and mixing them is the most common error when diagnosing why a team tests so little:

1. The statistical floor. How much traffic the flow has, and how many visitors per variation the effect you want to detect demands. That is pure arithmetic: given fixed traffic, there is a maximum number of tests per month the traffic sustains, no matter how well organized the team is.
2. Organizational friction. Approval queues, manual setup time, rework from lack of standards. None of that is statistical: it is 100% governance, and it is where most of the velocity gain actually lives, because the statistical floor is a ceiling that is hard to move without more traffic.

That distinction matters especially for self-serve SaaS products, where much of the growth comes from changing the product itself (onboarding, activation, plan upgrade) instead of media campaigns. The growth experimentation playbook for SaaS covers exactly that scenario: product teams usually have less traffic per flow than a large ecommerce site, which makes organizational friction proportionally more expensive, because the statistical floor already eats most of the available calendar.

Worked example: the same traffic, twice the tests per year

To avoid inventing numbers by ear, let us compute with the same math this guide calculator uses. Scenario: the signup page of a SaaS receives 40,000 visitors per month, converts 6% into trials today, and the team wants to detect a 15% relative improvement (the same two-proportion math covered in the statistical significance guide), with the market standard of 95% confidence and 80% power.

Run the sample size this scenario demands yourself (baseline 6%, relative MDE 15%):

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

The result is 11,693 visitors per variation, or 23,386 in total for the test. With 40,000 visitors per month (about 1,333 per day), traffic alone already limits each round to 18 days, above the two-week floor recommended to cover complete behavior cycles. That is the statistical floor: with this traffic and this target effect, the theoretical maximum is about 20 tests per year (30 days per month divided by 18 days per test, multiplied by 12), even in a perfectly efficient operation.

Now the organizational part enters. In a reactive operation, without formal cadence, that same test usually loses extra time outside the statistics: waiting for a senior approval, rework because the variation did not follow an already validated technical standard, and the dead time between “the last test ended” and “someone remembered to start the next one”. An illustrative friction of 21 additional days (for comparison only, it is not a universal number) is already enough to cut real capacity in half:

Metric	Before (reactive operation)	After (formal cadence)
Sample per variation (same statistics)	11,693	11,693
Days dictated by traffic	18	18
Additional organizational friction	21 days	2 days
Total cycle per test	39 days	20 days
Tests per month	0.77	1.50
Tests per year	about 9	about 18
% of the statistical ceiling (20/year) used	about 46%	about 90%

Notice what did not change: traffic is the same, the effect the team wants to detect is the same, and the sample math is identical in both scenarios. The entire difference between 9 and 18 tests per year comes from cutting organizational friction, not from getting more visitors nor from relaxing statistical rigor. That is why experimentation culture is, in practice, more about removing process friction than about advanced statistics.

The learning repository: why losing tests matter as much as winners

One of the most cited findings in the literature of experimentation at scale is uncomfortable: most tested ideas do not improve the metric they were supposed to improve. Ronny Kohavi, Diane Tang and Ya Xu, in “Trustworthy Online Controlled Experiments” (Cambridge University Press, 2020), report that even among well designed and well executed ideas at Microsoft, only about one third actually improved the target metric, another third stayed statistically flat, and the rest made things worse. That is not a sign of an incompetent team: it is the expected pattern when human intuition tries to predict real user behavior.

Booking.com pushes that logic to the limit: according to reporting on its operation, the company runs something close to 25,000 tests per year, and most do not produce a statistically significant gain (Stefan Thomke, “Building a Culture of Experimentation”, Harvard Business Review, March-April 2020). The central point of the Thomke article is that at Booking.com this is not treated as failure: it is treated as signal, and every test, winning, losing or inconclusive, removes one assumption that was previously made in the dark.

Without a central repository, that knowledge simply evaporates. The hypothesis “simplifying checkout increases conversion” can be tested, refuted, and tested again two years later by someone who never knew the earlier result, spending traffic and time to reach the same conclusion. A minimum repository needs to record at least these fields:

Field	What to record
Date	When the test finished running
Hypothesis	The original testable sentence, not rewritten after the result
Primary metric	The number that decided the test
Result	Won, lost or inconclusive
Effect size	Observed lift and confidence interval
Report link	Where the raw data and the full reading live
Learning	What this teaches about the user, even when the test lost

The most frequently ignored field is the last one. “Lost” without explanation teaches nothing; “lost because users never read that block of text, not because the text was wrong” is a learning reusable in other hypotheses. Writing the hypothesis before the test, in a format that cannot be quietly rewritten afterwards, is what makes that field possible at all: see how to write an A/B test hypothesis for the format this repository expects.

The most common organizational blockers

Four patterns show up repeatedly in programs that never leave the reactive stage, and none of them is about statistics. What the four have in common is that they all protect someone from a short-term discomfort (disagreeing with a result, admitting an idea did not work, risking a bigger proposal) at the cost of the whole team long-term learning.

HiPPO. The acronym, meaning “highest paid person’s opinion”, was popularized by Avinash Kaushik and Ronny Kohavi around 2006, when the two worked at Intuit and Microsoft respectively (Kohavi, “The Origin of HiPPO: Highest Paid Person’s Opinion”, LinkedIn Pulse). The problem is not the executive having an opinion, it is the executive reversing a published result simply because they disagree with it. In a single decision, that teaches the team that running the test was theater.

Fear of “losing” a test. When a test result is read as a reflection of personal competence, the team rationally stops proposing risky changes. It is the opposite of what Thomke describes at Booking.com, where failing is the statistically expected behavior, not a shameful exception.

Cosmetic tests only. A backlog full of “change the button color” and “swap one word in the headline” is risk aversion disguised as rigor. Small changes tend to have small effects, and small effects require enormous samples to detect, as shown in the statistical significance guide; the team ends up spinning, producing many inconclusive tests.

Institutionalized peeking. “It is almost significant, let us ship it” is the peeking problem, except practiced as house policy rather than as an isolated error.

Blocker	How it shows up day to day	Fix
HiPPO	An executive reverses a result because they did not like it	Define before each test who decides, and under what conditions a business veto is acceptable
Fear of losing a test	The team only proposes very low risk changes	Treat a losing test as documented learning, never as a career demerit
Cosmetic tests only	Backlog full of “change the button color”	Prioritize by impact potential, not by ease of implementation
Institutionalized peeking	“It is almost significant, let us ship”	Sample and duration fixed up front; decision only after they are reached
No learning repository	The same hypothesis is tested again two years later	Mandatory repository for every test, winning or losing
No named guardian	Nobody has standing to block a broken test	Name one person with technical veto and back them publicly the first time they use it

The maturity ladder: from reactive to fully integrated

Several experimentation maturity models, with slightly different stage names, converge on the same logic: the more mature the organization, the less the decision depends on one person and the more it depends on a system. Conversion.com describes five stages in that direction, from reactive to optimized, each defined by cadence, governance and who holds decision authority (Conversion.com, “The Conversion Maturity Model: Benchmark Your Experimentation Program”). The ladder below adapts that logic:

Level	Cadence	Governance	Who decides
1. Reactive	Sporadic tests, no calendar	No written rule; depends on someone remembering	The most senior person present, case by case
2. Emerging	A few tests per quarter	One or two enthusiasts defend the process, without formal mandate	Still concentrated, but evidence is at least contested
3. Structured	Monthly cadence expected	Prioritized backlog (ICE or PIE, for example), defined roles, repository exists	Backlog owner decides routine; executive consulted on bigger cases
4. Integrated	Weekly cadence, multiple teams in parallel	Statistical guardian with recognized technical veto; velocity metrics tracked	Routine rollout decentralized; HiPPO stops reversing published results
5. Optimized	Continuous experimentation, part of the standard workflow	Every relevant product decision is born as a testable hypothesis	The system decides most of it; humans define what is worth testing, not the verdict

Few companies reach level 5 across every front at once. It is common to be at level 3 on cadence and still at level 1 on governance (the backlog exists, but the executive still reverses results when they want to). The value of the ladder is not to label the whole company, it is to point at which specific axis is holding progress back.

To use the ladder as a diagnostic, answer three questions separately for cadence, governance and decision, instead of trying to fit the whole company into a single level. How many relevant tests ran last quarter, and did that depend on a specific person remembering? Is there anything written defining who can block a badly designed test, or has it never been needed because nobody ever tried? And the last time a result contradicted a senior expectation, did the result or the expectation win? The answers rarely point to the same level on all three axes, and it is exactly that divergence that shows where to start.

A 90 day plan for the first three levels

The scaffolding of levels 1 to 3 requires no new technology, which is why it fits in a quarter. What follows is a sequence that works when it is followed in order, because each step removes the excuse that blocks the next one.

Weeks	Move	Done looks like
1 to 2	Write the current state honestly	One page listing tests run in the last two quarters, who decided each rollout, and where the results live today
3 to 4	Name the four hats	A named backlog owner, a named statistical guardian, and one sentence about which decisions escalate to the sponsor
5 to 6	Open the repository	A single shared document or table with the seven fields above, backfilled with every test anyone can still remember
7 to 9	Prioritize a real queue	A backlog scored with PIE or ICE, sized against real traffic, with a duration column that makes infeasible items visible
10 to 12	Run the cadence once, publicly	Two tests completed under the new process, both written into the repository, including one that did not win

The last row carries the whole plan. A program only becomes credible when a losing test is documented and published without anyone paying a political price for it. Until that happens, the process exists on paper and the team is still waiting to find out whether it is real. If your traffic makes two completed tests in six weeks impossible, that is a signal in itself, and the constraints of that situation are covered in CRO for low-traffic sites.

What to measure about the program itself

A program that measures only “tests run per month” gets exactly what it measures: more tests, not more learning. Four indicators together give a fairer picture, and none of them is hard to collect once the repository exists.

Indicator	How to compute it	What it reveals when it is bad
Cycle time per test	Average days from approved hypothesis to recorded decision	High friction, if it is far above the statistical floor
Share of the statistical ceiling used	Tests run divided by the theoretical maximum the traffic supports	How much capacity is lost to process instead of to traffic
Share of tests with a pre-registered hypothesis	Tests with a written hypothesis before launch, over total tests	Whether the backlog is producing questions or just changes
Share of results recorded, including losses	Tests in the repository over tests actually run	Whether losing tests are quietly disappearing

The fourth one is the most diagnostic and the least measured. If the repository holds far more winners than the roughly one-in-three rate the literature predicts, the losses are not missing from reality, they are missing from the record, and that is a governance problem rather than a data problem.

Make this automatic with Donnu

Everything this guide covered that can be automated in statistics (sizing the right sample, checking SRM, avoiding peeking, computing significance without fooling yourself) Donnu already does by default, with honest Bayesian reports and a client-side snippet that never blocks the customer page. What Donnu does not replace, because it is your company decision, is the organizational layer of this guide: who prioritizes the backlog, who holds the technical veto mandate, and what your executive does when the result is unwelcome. Automating the statistics frees exactly the time those roles need in order to exist for real, instead of being swallowed by manual spreadsheets and arguments about p-values.

If your company is still at the reactive stage, the first practical step is not buying a new tool: it is writing, in one sentence, who decides the rollout of a routine test and what happens when the result contradicts the intuition of whoever is in charge. After that, the tool comes in to take the manual work out of the way. Start a 14 day free trial once the organizational layer already has that answer ready.

References

• Kohavi, R., Tang, D. & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
• Kohavi, R. The Origin of HiPPO: Highest Paid Person’s Opinion. LinkedIn Pulse. linkedin.com/pulse/origin-hippo.
• Thomke, S. Building a Culture of Experimentation. Harvard Business Review, March-April 2020. hbr.org.
• Conversion.com. The Conversion Maturity Model: Benchmark Your Experimentation Program. conversion.com/blog.
• Optimizely. Accelerating Growth Through Experiment Velocity. optimizely.com/insights.
• Kohavi, R. & Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.

---

Read also: What is A/B testing? The complete guide · A/B testing statistical significance · How to write an A/B test hypothesis · Growth experimentation for SaaS · Leia em português