CRO

Experiment Documentation: A Repository Nobody Ignores

How to build an experiment documentation repository people actually use: the fields to record, how to make it searchable and what a duplicate test costs.

by Equipe Donnu12 min readUpdated on August 6, 2026
Flat illustration of a wall of archive drawers with one drawer open showing stacked index cards and a magnifying glass beside it

📚 This article is part of the guide How to Build an Experimentation Culture (2026).

An experiment repository is the searchable record of every test the team has run, with the hypothesis, the numbers, the decision and the interpretation of each one. Almost every experimentation team starts one; most abandon it within two quarters. The failure is rarely discipline, it is design: repositories get built as write-only archives, indexed by test ID and populated only with wins, which makes them useless exactly when someone needs them. This article covers the fields worth recording, how to index entries so people find them by the question they actually have, why losses are the highest-value records, and a worked example that prices a duplicate test in days of lost capacity. It is part of the guide on how to build an experimentation culture.

Why most experiment documentation repositories get ignored

There are three distinct failure modes, and they need different fixes. Diagnosing which one you have matters more than picking a tool.

Write-only. Entries go in and nothing comes out. Nobody ever cites a past experiment in a planning meeting, and nobody is punished for not doing so. The symptom is easy to test: ask when the repository was last read, not last written. The fix is process, covered further down.

Unsearchable. The content exists but retrieval fails. Entries are titled “EXP-114” or “Q3 checkout test”, indexed by date, and finding whether anyone ever tested removing the coupon field means reading fifty documents. The fix is indexing by question, page and metric.

Winners-only. The repository is a highlight reel. Losses and ties never got written up, so the archive teaches that everything the team tried worked, which is both false and dangerous: the same losing idea comes back next quarter with nothing on record to stop it. The fix is a documentation rule that does not depend on the outcome.

The three ways an experiment repository failsA write-only repository receives entries but is never read. An unsearchable repository is read but retrieval fails. A winners-only repository is searchable but holds roughly one third of what the team learned, since losses and ties were never written up.Write-onlyentries go innothing comes outUnsearchableindexed by ID and date,not by question or pageWinners onlywins recordedties missinglosses missingEach failure has a different fix. Adopting a new tool addresses none of them on its own.Diagnose which one you have before rebuilding anything.
Three failure modes, three fixes: a reading ritual, an index built on questions, and a documentation rule independent of the result.

What to record for each experiment

The entry schema below is deliberately short. Every field earns its place by answering a question that comes up later, and anything that does not is friction that will eventually kill the habit.

Field What goes in The later question it answers
Title as a question “Does removing the coupon field increase checkout completion?” “Has anyone tested this?”
Decision (first line) Shipped, reverted, or inconclusive “What happened?” without opening the entry
Hypothesis Because we observed X, we believe Y causes Z, measured by W “What were we actually predicting?”
Page and audience Exact URL pattern, device, country, new or returning “Does this apply to my case?”
Primary metric Exactly one, chosen before running “Was the metric picked after the fact?”
Guardrails Metrics that must not degrade “Did the win cost us something elsewhere?”
Planned sample and duration Sample per variation and the window fixed up front “Was this test properly sized?”
Observed numbers Visitors and conversions per variation “Can I recheck this result myself?”
p-value and confidence interval Not just the verdict, the interval “How big could the effect really be?”
Interpretation One paragraph: what we now believe and why “What did we learn beyond the verdict?”
Links Screenshots, dashboard, code change, roadmap item “Where is the evidence?”

Two fields deserve defending because they are the first to get cut. Observed numbers matters because a verdict without raw counts cannot be rechecked, and results do get rechecked, usually when someone suspects a tracking bug months later. Interpretation matters because the verdict alone ages badly: “variation B won, p equals 0.03” tells a new team member nothing about which belief changed. If you keep only one prose field, keep this one.

The confidence interval is the field that most changes how the entry reads later. A win of “+4% relative, interval from +0.2% to +7.8%” and a win of “+4% relative, interval from +3.5% to +4.5%” carry the same verdict and completely different information about what to expect after shipping. The reasoning behind that is covered in the guide to A/B testing statistical significance.

The entry template, ready to copy

Question: Does removing the coupon field increase checkout completion?
Decision: Reverted (variation trended worse, interval includes zero, guardrail degraded)

Hypothesis: Because session recordings show users leaving checkout to
hunt for coupon codes, we believe removing the visible coupon field will
increase checkout completion, measured as completed orders / checkout starts.

Scope: /checkout, all devices, all countries, 2026-05-04 to 2026-05-18
Primary metric: checkout completion rate
Guardrails: revenue per visitor, support contacts about discounts

Planned: 24,193 per variation, 14 days, 95% confidence, 80% power
Observed: A 24,301 visitors / 1,458 conversions (6.00%)
          B 24,190 visitors / 1,353 conversions (5.59%)
p-value: 0.055   CI on the difference: -0.82pp to +0.01pp

Interpretation: Removing the field did not reduce distraction, it removed
a perceived discount opportunity. Support contacts about discounts also rose.
Do not re-test removal; the open question is whether collapsing the field
behind a link performs differently.

The last line of the interpretation is the part that pays for the whole exercise. It closes one question and opens a sharper one, which is the difference between an archive and a research program.

Make it findable by the question, not the ID

Retrieval is where repositories die. The test is simple: can someone who has been at the company three weeks find out, in under two minutes, whether the team has ever tested the product page gallery? If not, the index is wrong.

Three indexing rules cover most of it:

A useful addition once the archive passes roughly thirty entries: a single summary page listing, per page of the site, the questions already answered and the ones still open. That page becomes the thing people actually read before proposing a test, and it feeds the experimentation roadmap template directly, because an open question with evidence behind it is a much stronger roadmap candidate than a fresh opinion.

Losses and ties are the highest-value entries

A repository that records only winners documents a minority of what the team learned. According to data published by Ronny Kohavi on experiments at Microsoft, roughly one third of tested ideas improve the target metric, one third change nothing and one third make it worse (Kohavi, Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years, KDD 2015). On that proportion, a winners-only archive is missing about two entries out of every three.

The two missing categories carry different value:

A practical rule that survives contact with reality: documentation is done by whoever proposed the test, within two working days of the decision, regardless of the outcome, and the roadmap item is not closed until the entry exists. Tying the entry to the closing of the item is what keeps it from depending on anyone’s mood.

Worked example: what a duplicate test actually costs

Arguments about documentation overhead end quickly once the cost of not having it is expressed in days. Here is the calculation for a plausible ecommerce team.

The team tests on a product flow receiving 120,000 visitors per month, converting at 3.0%, and sizes tests to detect a 15% relative lift with 95% confidence and 80% power.

Check the first step with your own baseline:

Sample size calculator
-Visitors per variation
-Total (2 variations)
-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

Now suppose three of those 26 tests re-answered a question the team had already answered and forgotten. That is 3 times 14, or 42 days of testing capacity, which is 11.5% of the testing year, spent to learn nothing new. Expressed the other way: the repository has to save a bit more than one duplicate test per year to pay for every hour anyone spends writing entries.

Annual testing capacity lost to duplicate testsA team with 26 tests of annual capacity that re-runs three already-answered questions spends 42 of its 365 testing days, about 11.5 percent of the year, producing no new information.26 tests of annual capacity, at 14 days each42 d323 days producing new information3 duplicate tests11.5% of the testing yearspent re-answering questions the team had already answered and could not find.The repository only has to prevent one duplicate per year to be worth more than it costs.Capacity computed at 120,000 monthly visitors on the flow, 3.0% baseline, 15% relative MDE.
Documentation overhead is cheap next to the cost of testing capacity. The comparison only becomes obvious once capacity is expressed in days rather than in enthusiasm.

Your own capacity, and therefore your own duplicate cost, comes out of the same math:

A/B test velocity calculator
-Tests per month
-Tests per year
-Days per test

Sample per variation at 95% confidence and 80% power (two-sided), from your rate and MDE. Tweak the fields and watch capacity update live.

There is a second cost that rarely gets counted. A duplicate test does not only consume days, it consumes the slot of the test that would have run instead. In a queue where items are ordered by expected value, the duplicate displaces the highest-value item still waiting, so the real loss is the difference between what the duplicate taught (nothing) and what the displaced item would have taught.

Governance that survives a busy quarter

Rule Why it holds up What breaks without it
The proposer writes the entry They hold the context nobody else has Entries get written by whoever has time, badly
Two working days after the decision Memory of the interpretation is still fresh Entries become verdict-only summaries
The roadmap item closes only with the entry Ties documentation to work already tracked Documentation becomes optional, then absent
Same template for wins, losses and ties Removes any outcome-dependent friction The archive silently becomes a highlight reel
Monthly read, not just monthly write Creates demand, which sustains supply Repository turns write-only within a quarter
One entry, one stable link Makes citing an experiment as easy as pasting Knowledge lives in chat threads that expire

The monthly read deserves the most attention because it is the least intuitive. Teams try to fix repositories by making writing easier; the durable fix is making reading routine. A ten-minute slot in the monthly planning meeting where someone answers “what does the archive already say about this quarter’s queue” converts the repository from a chore into a tool, and once it is a tool, the writing takes care of itself.

Make this automatic with Donnu

Most of a repository entry is data your testing tool already holds: the audience, the planned sample, the observed counts per variation, the p-value and the interval. Donnu keeps that record per experiment with the numbers frozen as they were read, so writing the entry becomes a matter of adding the hypothesis and the interpretation instead of reconstructing the numbers from a dashboard that has since moved on. What the team owns is the thinking; what the tool owns is the evidence.

Start a 14-day free trial and keep the evidence attached to every test from the first one. To organize the queue that this archive should feed, see the experimentation roadmap template.

References

Read also:

Frequently asked questions

What is an experiment repository?
It is the searchable record of every experiment a team has run, with the hypothesis, the setup, the observed numbers, the decision taken and what was learned. It is different from a roadmap, which looks forward and holds the queue of tests to run. The repository looks backward and exists so that a question already answered by data does not get re-argued, or worse, re-tested.
What should be recorded for each experiment?
At minimum: a question-shaped title, the hypothesis in evidence, change, effect and metric form, the exact audience and page, the primary metric and guardrails, the planned sample and duration, the observed numbers per variation with the p-value and the confidence interval, the decision taken, and one paragraph of interpretation. The interpretation is the field most often skipped and the one that makes the entry usable a year later.
Should losing and inconclusive tests be documented?
Yes, and they are the entries with the highest long-term value. If roughly two thirds of tested ideas do not improve the target metric, a repository holding only winners is documenting about one third of what the team learned. Losses stop the same idea from returning every quarter, and ties record which effect sizes the traffic was not able to detect.
How do you keep an experiment repository from being ignored?
Make it searchable by the question a person actually has, not by test ID or date. In practice: title each entry as the question it answers, tag it by page, page element, metric and audience, and put the decision in the first line so someone scanning results does not have to open the entry to know the outcome. A repository people have to read in full to use is a repository people stop opening.
How much does a duplicate test cost?
The cost is real capacity, measured in days. A team with a 14-day test duration and roughly 26 tests of annual capacity that re-runs three already-answered questions burns 42 days, or about 11.5% of the testing year, and gets zero new information for it. That is the number worth quoting internally when someone argues that documentation is overhead.
Where should the repository live?
Wherever the team already works and searches: the existing wiki, the knowledge base or a shared database tool. The tool matters far less than two properties: one entry per experiment with a stable link, and a search that returns results by page and metric. A repository living in a place people have to be reminded to visit will lose to whatever tool they already have open.

Keep reading