Randomization Unit: User, Session or Pageview

The randomization unit is the level at which assignment to control or treatment happens, and it settles two things at once: what kind of effect your test can detect, and how much statistical sample you actually hold. Switching units is not an implementation detail, it is a trade between power and validity, and the version that produces the prettiest p-value is almost always the wrong one. This guide covers the difference between randomization unit and analysis unit, a worked example where the same experiment reports z of 3.680 or 6.374 depending on which unit you read it in, the design effect that explains the gap, and a decision rule for choosing among user, session, pageview and cluster. It is part of our complete guide to A/B testing and continues the reasoning from ratio metrics in A/B testing.

Every test carries two units

Every online experiment has two different units, and most confused arguments about results come from treating them as one.

Deng, Knoblich and Lu define both precisely in their KDD 2018 paper on the Delta method in metric analytics. The randomization unit is the granularity level where sampling or randomization is performed. The analysis unit is the aggregation level of metric computation. Analysis is straightforward, the authors note, when the two agree, for example when randomizing by user while also computing average revenue per user.

Trouble starts when they diverge, which is more common than it sounds. The authors report that for the same experiment there are usually metrics with different analysis units: most user-randomized experiments run by Microsoft’s experimentation platform contain both user-level and page-level metrics.

There is one hard constraint on the relationship, and it is the first thing to understand before picking any unit. The randomization unit cannot be more granular than the analysis unit. The authors spell out why: if it were, the analysis unit would contain observations under both treatment and control, nullifying the purpose of differentiating the two groups. Randomize by pageview and then report “conversion rate per user” and nearly every user will have seen both sides, so the measured gap between arms shrinks by construction, not because the effect is absent.

Worked example: one test, two readings

A shop runs a test randomized by user. That is 100,000 users per arm, and each user averages 3 sessions over the period. The metric of interest is “converted at least once”.

Paste the numbers below into the calculator to reproduce the arithmetic line by line.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

First, the correct reading, with the analysis unit matching the randomization unit:

Read per user	Users	Conversions	Rate
Control	100,000	12,000	12.000 percent
Treatment	100,000	12,540	12.540 percent

On screen, the calculator shows rates of 12.00 and 12.54 percent, a lift of plus 4.5 percent, a p-value of 0.0002 and a 95 percent interval of plus 0.3 to plus 0.8 percentage points, with B as the winner. The interface rounds its display and does not show the z score, so the engine’s exact figures, the ones used throughout this article, are: plus 0.540 percentage points, plus 4.500 percent relative, z = 3.680, a p-value of 0.00023 and an interval of plus 0.252 to plus 0.828 percentage points. A solid, unspectacular result.

Now the same experiment read per session. With 3 sessions per user, the dashboard shows 300,000 sessions per arm, and the conversion rate per session is identical:

Read per session	Sessions	Conversions	Rate
Control	300,000	36,000	12.000 percent
Treatment	300,000	37,620	12.540 percent

The calculator shows the same rates and the same plus 4.5 percent lift, and the p-value drops into the band it displays as less than 0.0001. At full precision: the same plus 4.500 percent relative lift, z = 6.374, a p-value of 0.00000000018 and an interval of plus 0.374 to plus 0.706 percentage points. Nothing about the experiment changed. The only move was swapping the metric denominator from user to session.

The ratio between the two z values is exactly the square root of 3, the number of sessions per user: 6.374 divided by 3.680 gives 1.7321. That is not a coincidence, it is the signature of a standard error computed as if every session were a fresh independent observation.

The design effect: how much sample is real

Sessions from the same user are not independent. Someone who buys on one visit tends to buy on another; someone who never converts tends to never convert. The quantity that captures this is the intra-cluster correlation, defined in Deng, Knoblich and Lu as the contribution of between-cluster variance to total variance.

For equal cluster sizes the authors record the closed-form variance, and out of it comes the factor known as the design effect: 1 plus the intra-cluster correlation times the cluster size minus 1. They lay out both extremes clearly. When within-cluster variance is zero, every observation in a cluster is identical, the intra-cluster correlation equals 1, and variance depends only on the number of clusters. When between-cluster variance is zero, the observations really are independent, the correlation equals 0, and the formula collapses to the simple case.

Applied to our test, with clusters of 3 sessions per user:

Intra-cluster correlation	Design effect	Corrected z	p-value	Effective sessions per arm
0.00	1.000	6.374	0.00000000018	300,000
0.25	1.500	5.205	0.00000019	200,000
0.50	2.000	4.507	0.0000066	150,000
0.60	2.200	4.298	0.000017	136,364
1.00	3.000	3.680	0.00023	100,000

Look at the last row: with intra-cluster correlation of 1 the corrected z lands back on exactly the 3.680 of the per-user reading, and effective sample lands back on exactly the 100,000 users. The formula degrades to the right answer, which is a useful sanity check.

And look at what it is really saying. Those 300,000 sessions were never worth 300,000 units. They are worth somewhere between 100,000 and 300,000, depending on how much of the behavior belongs to the user and how much belongs to the visit. An extra session from the same visitor is not fresh sample, it is partially repeated information.

An honest caveat from the authors themselves: this closed form, while theoretically sound, has only limited practical value, because it assumes every cluster shares the same size and the same distribution, which never holds in production. Users make wildly different numbers of sessions. Their proposed answer is the Delta method applied directly to the metric definition as a ratio of two averages of randomization-unit quantities, the same logic behind our treatment of ratio metrics.

It is worth knowing how large the error is when you ignore this. In the simulation published in that paper, with a thousand clusters of heterogeneous sizes, the true standard deviation of the estimator was 0.00895 while the naive standard error came out at 0.00522, roughly 42 percent below the truth. The Delta method returned 0.00908, close to the real value.

How to choose the randomization unit

The choice is not a matter of technical taste, it is a sequence of constraints. Walk them in order.

The options in detail:

Randomization unit	When it is right	What you give up
User (cookie, account, device)	Default. Any change a visitor could notice across visits	Nothing, beyond needing a stable identifier
Session	The change lives inside one visit and leaves no memory, or no stable identifier exists	Cross-visit consistency, and any effect that accumulates
Pageview	Almost never, in product testing	Consistency inside a single visit; it contaminates the reading
Account or organization	B2B products where colleagues cannot see different screens	Power, because the account becomes the cluster
Geographic region	Interventions that cannot be delivered per user, such as offline media	A great deal of power, see geo experiments
Time block	Marketplaces with supply shared between arms	Power, plus the need to handle time dependence, see switchback experiments

Two rows deserve a footnote. Deng, Knoblich and Lu cite as real reasons for cluster randomization both enterprise policy prohibiting users within the same organization from getting different experiences, and the need to reduce bias in the presence of network interference, which is precisely the problem covered in interference between variants.

What moving up the hierarchy costs in sample

Moving the randomization unit up costs power, and you can price it before you start. With a 12 percent baseline and a minimum detectable effect of 4.5 percent relative, the sample size engine returns 57,948 users per variant at 95 percent confidence and 80 percent power.

Randomizing by session instead would nominally require that same count in sessions, which looks 3 times cheaper. But those correlated sessions are not worth one unit each. Applying the design effect to the requirement:

Scenario	Units needed per variant	Equivalent in users
Randomize by user	57,948 users	57,948
Randomize by session, intra-cluster correlation 0 (unreal)	57,948 sessions	19,316
Randomize by session, intra-cluster correlation 0.25	86,922 sessions	28,974
Randomize by session, intra-cluster correlation 0.50	115,896 sessions	38,632

The discount is real, but it is smaller than the count ratio and it shrinks fast as correlation climbs. More importantly, the discount only holds if the effect you want to measure fits inside a single session. If it does not, you are not saving sample, you are measuring something else.

When session randomization is legitimate

There is an honest case, and it deserves naming so session assignment does not read as automatic malpractice:

• The change leaves no memory. Result ordering inside one search, the layout of a recommendation block, the behavior of a filter. The visitor returns tomorrow with no reason to find a different screen strange.
• No stable identifier exists. Logged-out site, mostly new traffic, heavy cookie blocking. In that case the “user” you think you are randomizing is already a session in disguise, and saying so out loud is more honest than pretending otherwise.
• The decision window is short. If conversion happens within the same visit in nearly every case, cross-visit accumulation is small by nature.

Outside those cases the price is steep. Session randomization means the same visitor sees control on Monday and treatment on Wednesday, and any effect that depends on recognition, habit or learning gets diluted across arms. It also means contamination: part of arm A’s experience already sits in the head of the person being measured in arm B. The close relative of this problem shows up in the novelty effect in A/B testing.

Checklist before you launch

1. Is the randomization unit written down anywhere? If nobody wrote it, nobody will check it when the results come in.
2. Does the primary metric use that same unit in its denominator? If not, the confidence interval needs the Delta method or a design-effect correction.
3. Is the randomization unit equal to or coarser than the analysis unit? Never finer.
4. Is the unit identifier stable for the whole test? A rotating cookie silently converts user randomization into session randomization.
5. Does the split per arm match the plan in the assigned unit? Check it with the SRM checker, in the right unit.
6. Does the report show which unit each number was computed in? Two metrics in different units on the same table need labels.
7. Was the power calculation done in the assigned unit? Not in the most numerous one.

Common mistakes

• Choosing the unit by available counts. “We have more sessions than users, so let’s randomize by session” is a decision made for apparent power against validity.
• Randomizing by user and reporting per-page rates without correction. This is the single most common case, and it is exactly where the naive standard error came out 42 percent below the truth in the Deng, Knoblich and Lu simulation.
• Switching units mid-test. Migrating from cookie to logged-in account during the experiment reshuffles who sits in which arm.
• Treating device as user without saying so. The same person on phone and desktop is two units, and the effect measured is per device, not per person.
• Reading a cluster minimum as an observation minimum. Twenty B2B accounts with many seats inside are still twenty randomized units.
• Ignoring size heterogeneity. The authors explicitly recommend making cluster sizes homogeneous, because the variance of cluster size feeds directly into the variance of the metric and erodes power to detect small effects.

Make this automatic with Donnu

The expensive mistake here is not picking the wrong unit at the start. It is picking one unit at the start and reading the result in another at the end, with nothing in the report flagging the swap.

In Donnu, the randomization unit is part of the experiment declaration, and every metric in the report carries the label of the unit its denominator was computed in. When the denominator is finer than the assigned unit, the dependence warning sits next to the confidence interval instead of hiding in a footnote. And if you want to redo any of the arithmetic by hand, the p-value calculator and the sample size calculator take raw counts in whichever unit you choose.

References

• Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Source of the definitions of randomization unit as the granularity level where sampling or randomization is performed and analysis unit as the aggregation level of metric computation, of the note that analysis is straightforward when the two agree, of the constraint that the randomization unit cannot be more granular than the analysis unit lest the analysis unit contain observations from both arms, of the report that most user-randomized experiments on Microsoft’s platform contain both user-level and page-level metrics, of the closed-form variance for equal cluster sizes and the factor 1 plus intra-cluster correlation times size minus 1, of the two extreme cases at correlation 0 and 1, of the caveat that this closed form has limited practical value because it assumes equal clusters, of the practical reasons for cluster randomization (enterprise policy and network interference), of the recommendation to keep cluster sizes homogeneous, and of the simulation figures with true standard deviation 0.00895, naive standard error 0.00522 and Delta method 0.00908. arxiv.org/abs/1803.06336.
• Bakshy, E. and Eckles, D. Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods. KDD 2013. Source of the finding that procedures ignoring dependence are anti-conservative, producing Type I error rates above nominal and 95 percent intervals that cover the true value less than 95 percent of the time. arxiv.org/abs/1304.7406.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Reference on organizational metrics rather than feature metrics, on diluting effects by segment size, and on the volume of false positives generated by running many experiments across uncorrelated metrics. exp-platform.com.
• Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. General reference on choosing the randomization unit, experience consistency, and the consequences of analyzing in a unit other than the one assigned. Companion material at experimentguide.com.

Read next: Ratio metrics in A/B testing · Interference between variants · Switchback experiments · Sample ratio mismatch · Triggered analysis and dilution · P-value calculator · Leia em português