Triggered Analysis and Dilution: Two Numbers, One Test

Triggered analysis restricts the result to the users who actually reached the point where your variant could change something, and it is usually the only way to see an effect that genuinely exists. The price is that the triggered number is not the business number: to get overall impact you have to dilute it by the share of users who triggered, and that arithmetic tends to cut the result by a large factor. This guide covers what triggering actually means, the arithmetic linking segment effect to overall effect, a worked example where the same change reads significant when triggered and inconclusive when diluted, and the traps that turn triggered analysis into a machine for manufacturing wins. It is part of our complete guide to A/B testing and pairs directly with primary metric and OEC.

What triggering means and why it changes the denominator

Almost no change reaches everybody. You rewrite the invalid coupon error message, and most visitors never type a coupon. You swap the recommendation module on the product page, and half your sessions enter straight at the cart. In both cases the experiment technically randomized the whole population, but the variant only existed for part of it.

A user triggered the experiment when they reached the point in the flow where control and treatment become different things. Kohavi and colleagues use the word in that literal sense in the KDD 2014 rules of thumb. Describing a Bing experiment that truncated the results page for navigational queries, they note parenthetically that triggered means at least one of the user queries showed a page with a deep links block. That experiment ran with over 8 million triggered users, and the abandonment rate did not change to a statistically significant degree, with a p-value of 0.92. A second experiment in the same paper ran with over 3 million triggered users, extending the page to 20 results: revenue fell 1.8 percent, page load time slowed by 30 milliseconds and pagination dropped 18 percent, yet abandonment still did not move, with a p-value of 0.93. That change was not shipped.

The arithmetic point is simple and unforgiving. Every user who did not trigger has a treatment effect of exactly zero, because they saw the same thing in both arms. They are not neutral data. They sit in the denominator, contribute variance, and contribute no signal whatsoever. Diluting your experiment with non-triggered users is the cheapest way to turn a real effect into an inconclusive readout.

The dilution rule, in one line of arithmetic

Converting between the two numbers is direct and rests on no statistical assumption at all:

absolute lift overall = absolute lift among triggered users x share of users who triggered

A 1.5 percentage point lift on a trigger that fires for 15 percent of the base becomes 0.225 percentage points overall. This is the rule Kohavi and colleagues state in rule 2 of the KDD 2014 paper, under the heading that changes rarely have a big positive impact on key metrics. In their words, metrics should be diluted by their segment size, and a 10 percent improvement to a 1 percent segment has an overall impact of approximately 0.1 percent. They are careful to flag that “approximately”: if the segment metrics differ from the average, the impact will be different.

That caveat is the detail almost every team gets wrong, so it earns its own paragraph. The simple multiplicative rule holds exactly for the absolute lift. For the relative lift, the dilution factor is not the share of users, it is the share of the total metric that the triggered segment accounts for. A segment converting well above average weighs more in the metric than it weighs in the headcount.

Situation	Share that triggers	Effect measured on trigger	Diluted effect overall
Coupon error message	3 percent	plus 4.0 percentage points	plus 0.12 percentage points
Recommendation module on product page	15 percent	plus 1.5 percentage points	plus 0.225 percentage points
New shipping step in checkout	40 percent	plus 0.8 percentage points	plus 0.32 percentage points
Global header change	100 percent	plus 0.2 percentage points	plus 0.2 percentage points

That last row is the case where the distinction vanishes. When the trigger is the home page, triggered and overall are the same thing and the whole discussion disappears. It is why teams that only test the top of the funnel never had to think about this, and why the first serious checkout test is usually the moment the numbers stop adding up.

The other side of the rule, and what makes it useful in planning rather than only in postmortems, is that it runs backwards. If the business needs 0.2 percentage points overall to justify the change, and your trigger fires for 15 percent of users, you need roughly 1.33 percentage points among triggered users. That is the effect your design’s minimum detectable effect has to be able to resolve, and it is the number that belongs in the hypothesis before the test starts.

Worked example: one change, two verdicts

A store tests a recommendation module that only appears on product pages in one category. The experiment randomizes 80,000 users per variant. Of those, 12,000 per variant get far enough to see the module, so 15 percent trigger. Paste the numbers into the calculator below to check each line:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

First, triggered analysis, restricted to users who saw the module:

Triggered users only	Users	Conversions	Rate
A, control	12,000	2,400	20.000 percent
B, variant	12,000	2,580	21.500 percent

The calculator returns plus 1.500 percentage points, plus 7.50 percent relative, z = 2.865, p-value 0.00417, with a 95 percent interval of plus 0.474 to plus 2.526 percentage points. Significant, comfortably.

Now the same change measured across everyone. The 68,000 users per variant who never triggered convert at 3.00 percent in both arms, contributing 2,040 conversions on each side:

All randomized users	Users	Conversions	Rate
A, control	80,000	4,440	5.550 percent
B, variant	80,000	4,620	5.775 percent

The calculator returns plus 0.225 percentage points, plus 4.05 percent relative, z = 1.947, p-value 0.05153, with an interval of minus 0.001 to plus 0.451 percentage points. Not significant, and only barely not.

Check both halves of the dilution rule against those numbers. The absolute lift: 0.15 times 1.500 gives exactly 0.225 percentage points. The relative lift, which is where the “approximately” bites: 7.50 percent times the share of the total metric the segment accounts for, which is 2,400 divided by 4,440, or 54.05 percent, gives 4.05 percent. It matches. Note that the relative dilution factor is 54 percent, not the 15 percent headcount share, precisely because the triggered segment converts at 20 percent against 3 percent for everyone else.

The sample cost closes the story. Detecting a 7.5 percent relative effect on a 20 percent base, at 95 percent confidence and 80 percent power, takes 11,472 users per variant. Detecting the same effect once diluted, that is 4.05 percent relative on a 5.55 percent base, takes 165,644 users per variant. Dilution multiplied the required sample by more than 14, without the product change getting one millimetre smaller.

The trigger point doubles as a sanity check

There is a nearly free check hiding inside the idea of triggering, and it catches instrumentation bugs better than almost anything else. Before the trigger point, both arms saw the same thing. Therefore no metric measured before that point should differ to a statistically significant degree.

Kohavi records exactly this reasoning in his paper on unexpected results in SIGKDD Explorations. Testing whether a link should open in a new window, he notes that clicking the link is the triggering point, so there should not be a statistically significant difference until after the click, since that is the first place anything differs between arms. When the percentage of users who clicked the link came out significantly higher in treatment, the team knew that was impossible by construction and went looking. The cause was instrumental: clicks are commonly instrumented with an asynchronous request for a 1 by 1 pixel image, a mechanism well known to be lossy, and opening the destination in a new window measurably improved that beacon’s reliability in non-Internet Explorer browsers. The feature was still positive after the correction, just far less positive than the first readout claimed.

This is worth hardening into routine. For every experiment with a narrow trigger, measure and compare across arms: the count of triggered users, the trigger rate, and any purely pre-trigger metric. A statistically significant difference in any of the three is not a finding, it is a defect, and it belongs to the same family of checks as sample ratio mismatch.

Where triggered analysis misleads

Triggering is a power tool, and every power tool cuts both ways. Four failure modes show up repeatedly.

The trigger depends on the treatment. This is the serious one. If the criterion defining who lands in the triggered set is itself affected by the variant, the two sets stop being comparable and you have built selection bias. The canonical version is conditioning on “users who clicked the new button”, since control has no new button. Triggering must depend only on information fixed before the variant could act. Kohavi and colleagues describe, in their rule about avoiding complex designs, a case where an eligibility check ran before the experiment was even triggered and carried a bug: a user who had seen the new feature once was put on a whitelist that removed them from the experiment entirely. The results looked like treatment users churning after a single visit, and it took days to unpick.

The triggered number is reported alone. A report showing only the segment effect produces a roadmap whose sum never appears in revenue. If every piece is denominated in segment currency, the promised total can be several times the possible total.

The trigger gets narrowed for convenience. There is always a temptation to shave the segment thinner until the p-value gives way. That is the same mechanism as the peeking problem, operating on the slicing dimension instead of the time dimension. The trigger criterion is part of the design and has to be written down before the data arrives.

Guardrails get measured on the trigger only. Collateral damage rarely respects your slice. A change that improves the triggered segment and degrades navigation for people passing nearby has to be visible on the full population. Guardrail metrics stay on the wide denominator, always.

How to set this up in practice

A short runbook, ordered by how much rework each step saves:

1. Before running, write the trigger condition as one sentence any teammate could evaluate by looking at a log, and confirm it uses no information generated after exposure.
2. Estimate the trigger rate from historical data. If you do not know what share will trigger, you cannot size the test.
3. Convert the business target into a triggered target by dividing by the trigger rate, and only then compute sample size.
4. Instrument the trigger as its own event, not as something inferred later in a query. A trigger reconstructed in SQL is a trigger whose definition changes with every analyst.
5. When analysing, run the pre-trigger sanity check before you look at the result.
6. When reporting, present both numbers side by side, in this order: triggered effect, trigger rate, diluted effect. Never one without the other two.

Common mistakes

• Calling a segment a trigger. A trigger is the point where the experience starts to differ between arms. Slicing by country, device or traffic source is segment analysis, with all the multiple comparison problems that carries, and has nothing to do with triggering.
• Diluting the relative lift by the user share. Absolute lift dilutes by the share of users; relative lift dilutes by the share of the metric. Using the headcount share in both places understates the relative effect whenever the segment converts above average, which is the normal case.
• Comparing one test’s triggered effect with another test’s diluted effect. Backlog prioritisation built on mismatched denominators ranks the roadmap in the wrong order. Standardise everything to diluted impact before comparing.
• Forgetting that non-triggered users still cost sample. They do not disappear from the experiment; they sit there consuming traffic and calendar. On a smaller site that is frequently the difference between concluding and not concluding, a point we work through in CRO for low traffic sites.

Make this automatic with Donnu

The tedious part here is not understanding the dilution rule, it is remembering to apply it on every readout, every day, with the right trigger rate for that experiment. It is exactly the kind of arithmetic that evaporates when a report is assembled by hand.

In Donnu, the trigger event is declared alongside the experiment, so every result arrives with both readings already computed and labelled: the effect on the population that saw the change, and the diluted effect on the full base, with the trigger rate sitting between them. The pre-trigger sanity check runs on its own and raises a flag before you read the result, rather than after the decision has been made. If you would rather do the arithmetic yourself, the significance calculator accepts either denominator.

References

• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the rule that metrics should be diluted by segment size, the example of a 10 percent improvement on a 1 percent segment yielding roughly 0.1 percent overall with the explicit caveat that the value changes if segment metrics differ from the average, the 0.1 to 1.0 percent range for successful Bing experiments once diluted, the definition of a triggered user, the two experiments with over 8 million and over 3 million triggered users returning p-values of 0.92 and 0.93 for abandonment, and the eligibility check case where a whitelist removed users from the experiment. exp-platform.com.
• Kohavi, R. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations, 12(2), 2010. Source of the reasoning that no metric should differ before the triggering point, the new window link case where the percentage of users clicking came out significantly different, and the explanation that click instrumentation via a 1 by 1 pixel image is lossy and had its reliability changed by the treatment itself. kdd.org.
• Crook, T., Frasca, B., Kohavi, R. and Longbotham, R. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. KDD 2009. Source of the treatment of unrepresentative populations and the principle that noise spread evenly across arms costs power without invalidating the result, while a skew concentrated in one arm creates bias. exp-platform.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapters on triggered analysis, dilution and metric selection. Companion material at experimentguide.com.

Read also: Primary metric and OEC · Guardrail metrics · Sample ratio mismatch · Minimum detectable effect · P-value calculator · Leia em português