Quantile Metrics in A/B Testing: The Median and the p90

When the metric is a percentile rather than a mean, the standard standard-error formula stops holding, because it assumes each observation is independent and pageviews from the same user are not. The price is measurable: in the simulation in this article, the naive reading of a quantile produced a significant result in 31.20 percent of A/A tests at p50 and in 28.13 percent at p90, against a nominal 5 percent target. That is not a theoretical detail, it is the difference between shipping a performance win that exists and shipping one that never did. This guide covers why the average fails for performance, what exactly breaks in the arithmetic, how to fix it with a bootstrap at the right unit, and what the cheap proxy is when a bootstrap is not viable. It is part of our complete guide to A/B testing and complements ratio metrics and sample size for revenue and continuous metrics.

Quantile metrics: why performance is not measured by the average

The LinkedIn experimentation team opens their paper on the subject with a hypothetical that settles the argument in two sentences. Imagine two websites with exactly the same average page load time of 0.5 seconds. Website A loads all pages in 0.5 seconds. Website B loads 10 percent of pages in 5 seconds and the remaining 90 percent in 0 seconds.

The average is identical. The experience is not: website A is perceived as fast because each page loads within a blink of an eye, and website B is perceived as slow because users frequently need to wait 5 seconds before a page loads. The authors’ conclusion is direct: to optimize the speed experience you need to reduce the loading time of the slowest page loads, not reduce the average by making the fast pages even faster.

Running the quantiles of that same example, the difference appears immediately:

website	mean	p50	p95
A: every page in 0.5 s	0.50 s	0.50 s	0.50 s
B: 10% in 5 s, 90% in 0 s	0.50 s	0.00 s	5.00 s

The p50 and the p95 completely separate the two sites the average declares tied. (The p90 of website B falls exactly on the boundary between the fast and the slow group, so it is precisely the percentile that fails to distinguish them in this particular hypothetical: picking the right percentile is part of the job.) According to LinkedIn, the industry standard for measuring page load time is the quantile, with p90 monitoring tail performance as the ultimate metric to optimize for and p50 monitoring overall performance. Before their platform supported quantiles, average load time was used as a surrogate for p50, and there was no good surrogate at all for p90.

What exactly breaks in the arithmetic

There is a classic asymptotic formula for the standard error of a sample quantile. It works, and works well, under one condition: the observations must be independent and identically distributed.

In a performance A/B test that condition is false by construction. You randomize users, and each user generates several pageviews. As the LinkedIn authors put it, page load times from the same member are likely positively correlated, because page views from a member with a fast device and fast network are likely to all be faster, and vice versa.

The result is known and large. LinkedIn measured against the bootstrap, which they treat as ground truth because it is unbiased, and found a median underestimation of 74 percent in the standard deviation of the quantile. The consequence they publish is direct: when the estimated p-value is 0.05, the true p-value is actually 0.61, which inflates the false positive rate by 12 times and exposes the experimenter to 61 percent false positives when the nominal rate is 5 percent.

The price, measured in a reproducible simulation

We reproduced the phenomenon in a controlled scenario, so we could show both calculations side by side. The design: 6,000 users per arm; each user has a persistent effect of their own (their device and network) plus per-pageview noise; the number of pageviews per user is random, averaging close to 5. That produced 26,755 pageviews in control and 26,729 in the variant.

First, the A/A test. We generated both arms from exactly the same distribution and read the quantile with the asymptotic formula under independence. Any significant result here is a false positive:

quantile	real false positive rate of the naive reading	nominal target	A/A replications
p50	31.20%	5%	3,000
p90	28.13%	5%	3,000

Six times the promised level at the median. The magnitude landed below the 61 percent LinkedIn observes on real data, which is expected: the within-user correlation in our simulated scenario is weaker than that of real page load times. The sign and the order of magnitude are the same.

Then the actual A/B test, with a small shift introduced in the variant:

quantile	observed difference	naive standard error	naive p-value	standard error from a user-level bootstrap	correct p-value	understatement
p50	plus 0.0485 s	0.01092	0.00001	0.02058	0.01856	47.0%
p90	plus 0.1259 s	0.03656	0.00057	0.05878	0.03216	37.8%

In both cases the effect is real and both calculations agree on the verdict. What changes is the declared confidence. At p50, the naive calculation reports a p-value of 0.00001, the kind of number that in a meeting becomes “essentially impossible to be chance”. The correct calculation reports 0.01856, which is significant but modest. The correct 95 percent interval runs from 0.0086 to 0.0899 seconds, while the naive one runs from 0.0271 to 0.0698. The naive reading not only claims more certainty than it has, it claims a more precise effect than it measured.

The right way: bootstrap at the randomization unit

The correction is not sophisticated, it is conceptual: resample at the same unit you randomized. As LinkedIn describes the procedure, the resampling needs to happen at the member level to preserve the dependency structure, because page load times of the same member are not necessarily independent, but members are independent of each other.

In practice:

// Bootstrap of a quantile at the randomized user level.
// users = array of arrays; each element holds ALL events from one user.
function bootstrapQuantile(users, q, B, rand) {
  const n = users.length;
  const estimates = new Array(B);
  for (let b = 0; b < B; b++) {
    const sample = [];
    for (let i = 0; i < n; i++) {
      const u = users[Math.floor(rand() * n)];      // a whole user, with replacement
      for (const x of u) sample.push(x);            // all of their events together
    }
    sample.sort((a, c) => a - c);
    const pos = q * (sample.length - 1);
    const lo = Math.floor(pos), hi = Math.ceil(pos);
    estimates[b] = sample[lo] + (sample[hi] - sample[lo]) * (pos - lo);
  }
  estimates.sort((a, c) => a - c);
  return {
    p025: estimates[Math.floor(0.025 * (B - 1))],
    p975: estimates[Math.floor(0.975 * (B - 1))],
  };
}

The common and very expensive mistake is resampling individual pageviews with replacement instead of users. That breaks the dependence exactly the way the i.i.d. formula does, and hands back precisely the too-small standard error you were trying to avoid. A bootstrap fixes nothing on its own: what fixes it is resampling at the right unit. It is the same principle we described in randomization unit.

For the p-value, the natural companion is a permutation test, shuffling whole users between the two arms. The difference between quantiles has no comfortable closed form, and the pairing of permutation for the p-value with bootstrap for the interval covers both sides.

The trap of the distribution-free interval

There is an elegant and old construction for the confidence interval of a quantile that assumes neither normality nor any shape: the interval from order statistics. You compute which positions of the sorted sample bracket the quantile at the desired confidence, using the binomial distribution, and read the values at those positions. Spotify records that this kind of exact, distribution-free interval for population quantiles has been known for a long time and can be constructed using only order statistics.

Applied to the control arm of our example:

quantile	95% interval from order statistics	positions used
p50	from 1.3989 to 1.4260 s	13,217 and 13,539 of 26,755
p90	from 3.4618 to 3.5503 s	23,983 and 24,177 of 26,755

The half width of the p50 interval here is 0.0136 seconds, almost exactly 1.96 times the naive standard error of 0.00717 seconds. In other words: the distribution-free interval reproduces the naive interval, because it also assumes independence. It drops the normality assumption and leaves intact the assumption that is actually wrong in your case.

It is worth recording the practical limitation Spotify points out alongside it: these order-statistic intervals unlock the one-sample case for massive samples, but they do not extend directly to the two-sample case, the difference in quantiles, which is exactly what an A/B test needs.

When a bootstrap does not fit the budget

The bootstrap is expensive. Spotify quantifies it: the complexity of the Poisson bootstrap algorithm is on the order of the product of the estimator cost and the number of resamples, and since quantile estimators are based on order statistics, the per-resample cost is already linear in the sample size. Multiply by a thousand resamples and by hundreds of millions of observations and the arithmetic stops working.

The two published solutions attack the cost, not the validity:

• LinkedIn, a closed-form asymptotic expression. They derive the asymptotic distribution of the sample quantile without the i.i.d. assumption, requiring only that page load times from different members be independent, which is true whenever the member is the randomization unit. The result, validated on 242 real experiments across different analysis populations, date ranges, platforms and quantiles: over 500 times faster than the bootstrap, with only a 2 percent chance of the standard deviation estimate differing from the bootstrap’s and, when it differs, a difference below 7 percent. The resulting real false positive rate is at most 5.1 percent when the nominal one is 5 percent.
• Spotify, a resampling-free Poisson bootstrap. They use the properties of the Poisson distribution and of order-statistic based quantile estimators to derive algorithms that produce the same interval without running the resamples, and without additional assumptions. The authors record that this unlocked bootstrap confidence intervals for quantiles and for differences in quantiles in A/B tests with hundreds of millions of observations.

If you are not operating at LinkedIn or Spotify scale, the good news is that a plain bootstrap solves your case. A few thousand users and a thousand resamples run in seconds, as they did to generate the tables in this article.

The binary proxy, when none of that fits

The cheapest escape of all swaps the quantile for a per-user indicator: instead of “p90 page load time”, measure “user who had at least one page load above 3 seconds”. That solves the dependence problem at the root, because there is now one observation per randomized user, and it hands the problem back to the two-proportion calculation every tool performs.

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

In our scenario, 1,796 of the 6,000 control users had at least one page load above 3 seconds (29.933 percent), against 1,925 of the 6,000 in the variant (32.083 percent). Pasting those four numbers into the calculator above: z of 2.5460, a p-value of 0.01090, a difference of 2.1500 percentage points and a 95 percent interval from 0.4953 to 3.8047 percentage points.

What you gain: a correct calculation, closed form, no bootstrap and no risk of understating the standard error. What you lose: the effect size in milliseconds. The proxy says the share of affected users rose by 2.15 points, and it does not say the p90 rose by 126 milliseconds. For a guardrail metric, where the question is binary (“did it get bad enough to abort?”), the proxy is usually sufficient, and that is how it appears in our piece on guardrail metrics. For the success metric of a performance project, where the question is “how much did it improve”, it does not serve.

One trap with the proxy: the threshold has to be chosen before you look at the data. Testing 2, 3 and 5 seconds and reporting whichever came out significant is a multiple metrics problem dressed up as a technical choice.

Checklist

1. Pick the quantile by what the metric needs to show, not by habit. p50 for overall performance, p90 or p95 for the tail.
2. Never read a quantile with the i.i.d. asymptotic formula if the randomization unit is the user and the measurement unit is the pageview.
3. Bootstrap by resampling whole users, with all of their events together. A thousand resamples already give a stable estimate.
4. Distrust the order statistic interval. Distribution free is not dependence free.
5. Run an A/A before trusting your quantile pipeline. If it comes out significant far more than 5 percent of the time, the arithmetic is wrong, not the product.
6. If you cannot bootstrap, use the binary per-user proxy, with the threshold declared before you look at the data.
7. When you publish, say which calculation you used. A p90 with a p-value and no declared method is a number with no provenance.

Make this automatic with Donnu

The reason almost no A/B testing tool reports quantiles is not lack of interest, it is architecture: to compute a quantile correctly you need to keep events attached to the user who generated them, and most platforms aggregate everything into counters at collection time. Once aggregated, the necessary information no longer exists.

Donnu keeps the event attached to the randomized user, which is the precondition for resampling at the right unit and for the binary per-user proxy to come out correct without workarounds. If your current project is a performance one and your tool only reports averages, the immediate and cheap step is to build the binary per-user indicator and run the significance calculator on it, rather than comparing averages that hide the tail.

References

• Liu, M., Sun, X., Varshney, M. and Xu, Y. Large-Scale Online Experimentation with Quantile Metrics. arXiv 1903.08762, 2019. Source of the two-website example with a 0.5 second average and opposite experiences; of the record that the industry standard for page load time is the quantile, with p90 monitoring the tail and p50 overall performance, and that before this work there was no good surrogate for p90; of the finding that the bootstrap is statistically valid but takes days, while the asymptotic estimate under independence is scalable and underestimates the variance by an order of magnitude; of the median 74 percent underestimation of the standard deviation, with an estimated p-value of 0.05 corresponding to a true p-value of 0.61, inflating false positives by 12 times and reaching 61 percent when the nominal rate is 5 percent; of the requirement that bootstrap resampling happen at the member level to preserve dependency; and of the proposed method’s results (over 500 times faster than the bootstrap, 2 percent chance of differing, difference below 7 percent when it differs, real false positive rate of at most 5.1 percent, validated on 242 real experiments). arxiv.org.
• Schultzberg, M. and Ankargren, S. Resampling-free bootstrap inference for quantiles. arXiv 2202.10992, Spotify Experimentation Platform Team, March 2022. Source of the framing that the computationally intensive nature of the bootstrap made inference infeasible in large-scale experiments; of the Poisson bootstrap complexity being on the order of the product of estimator cost and number of resamples, with quantile estimators linear in sample size because they rest on order statistics; of the record that exact, distribution-free intervals for population quantiles can be constructed using only order statistics but do not extend directly to the two-sample difference-in-quantiles case; and of the result that the proposed resampling-free algorithms unlocked interval computation for quantiles and differences in quantiles in A/B tests with hundreds of millions of observations. arxiv.org.
• Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014. Source of the framing that experimentation methodologies typically rely on means assumed to be normally distributed, that many metrics of interest in online experiments are skewed and therefore require a higher lower bound on sample size, and of the recommendation to use bootstrapping techniques for skewed distributions with small samples. exp-platform.com.
• Deng, A., Knoblich, U. and Lu, J. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. KDD 2018. Source of the context for why experimentation platforms at scale pursue analytical variance estimators for metrics that are not simple means, and why the cost of the bootstrap is the practical obstacle to adopting it in production. arxiv.org.

Read next: Permutation tests · Ratio metrics · Count metrics · Randomization unit · Guardrail metrics · Significance calculator · Leia em português