Mobile vs Web A/B Testing: What Is Actually Different
Mobile vs web A/B testing: release cycle, randomization unit, sample size, propagation latency and the guardrails that only exist inside an app.
📚 This article is part of the guide Mobile App A/B Testing: The Complete Guide.
Mobile and web A/B tests use exactly the same statistics and almost nothing else in common. The two-proportion test, the p-value, the confidence interval and the sample size formula are identical; what changes is the release cycle, the randomization unit, the propagation delay between assigning a variation and rendering it, the volume of traffic available on the screens that matter and the guardrails that only exist inside an app. This article is a child of the complete guide to A/B testing in mobile apps and focuses on one question: what a team that already tests well on the web has to unlearn before testing inside an app.
The short version: on the web the hard part is usually deciding what to test, because shipping a variation costs minutes. On mobile the hard part is the machinery around the test, because a variation that is not behind a remote flag can take weeks to reach everybody and cannot be turned off when it goes wrong.
Mobile vs web A/B testing: the six differences that change the outcome
| Dimension | Web | Mobile app |
|---|---|---|
| Shipping a variation | Immediate, a snippet or a deploy | Depends on a release, store review and user adoption, unless it is behind remote config |
| Killing a bad variation | Immediate, flip the flag or roll back | Immediate only with remote config; otherwise a new submission and update cycle |
| Randomization unit | Cookie or browser storage, easily lost | Device or installation id, sometimes the logged-in account |
| Assignment to render delay | Milliseconds | Seconds to days, depending on config fetch and app launch |
| Traffic on the deciding screen | Often tens of thousands per week | Often a few thousand per week |
| Extra guardrails | Load time, error rate | Crash rate, store rating, app size, battery and data usage |
Every row in that table has the same practical consequence: on mobile you get fewer, slower and more expensive experiments, so each one has to be chosen and sized more carefully than its web equivalent.
Difference one: shipping and killing a variation
On the web, a variation is a deploy or a snippet rule, and rolling back is another deploy. On mobile, a variation baked into the binary has to pass store review and then wait for users to update, which means the test population grows slowly and unevenly. Worse, it cannot be stopped: a variation that hurts conversion or breaks a screen on a particular device family keeps running until a new version ships and gets adopted.
This is why remote configuration is not a nice-to-have on mobile, it is the precondition for testing at all. The variation ships inside the binary but stays inactive; a server-side flag decides who sees it, and the same flag turns it off in minutes. Firebase Remote Config and its A/B Testing layer are the most common implementation on both platforms, and every serious alternative works the same way.
One consequence to plan for: the flag is fetched, cached and applied on the app’s own schedule, not on yours.
Difference two: assignment to render latency
On the web, assignment and render happen in the same page load. On mobile there is a gap. The app fetches remote configuration, caches it, and often applies the new value only on the next launch, so a user can be assigned to variation B and keep seeing variation A for hours or days. Devices in aggressive battery saving mode, or apps left in the background for a week, stretch that further.
Two practical rules follow. First, treat the opening days of a mobile test as a propagation window rather than valid sample; if your platform lets you, start counting exposure from the first event that proves the variation actually rendered, not from the assignment. Second, never compare a mobile test’s day-one numbers with a web test’s day-one numbers, because the mobile ones include a mixture of users who have not received the change yet.
Difference three: the randomization unit
A web test randomizes on a cookie or a browser storage key, which is fragile: cleared storage, private windows and a second device all produce a new identity for the same person. A mobile test randomizes on a device or installation identifier, which is far more stable within a device and just as unstable across devices: the same person on a phone and a tablet can sit on both sides of the experiment.
| Situation | Recommended randomization unit | Why |
|---|---|---|
| Logged-out app, single device per user | Device or installation id | Stable, simple, no identity needed |
| Logged-in product used on several devices | User account id | Keeps the same person in the same variation everywhere |
| Test that changes something before login | Device id, with login users analyzed separately | Mixing the two units in one reading contaminates the split |
| Test whose effect only appears after login | User account id, excluding logged-out sessions | Including sessions that could not convert dilutes the effect and inflates the sample needed |
The rule of thumb: pick the unit that matches the level at which the effect happens, and never mix two units in the same reading. This connects directly to SRM, because a change of unit halfway through a test shows up as a split ratio that drifts over time.
Difference four: sample size and the arithmetic of patience
This is where the difference stops being conceptual and starts costing weeks. The formula does not change, only the denominator does. Adjust the numbers below for your own baseline and traffic:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
The same test, two platforms
Take a conversion rate of 3% on the deciding screen and a target of detecting a 10% relative improvement (from 3.0% to 3.3%), at 95% confidence and 80% power. The math returns 53,211 users per variation, on the web and inside the app alike, because the statistics do not care where the user is.
What changes is how long it takes to accumulate them:
- Web, 40,000 visitors per week on the tested page: about 19 days.
- App, 6,000 users per week reaching the tested screen: about 125 days, more than four months, which is not a test, it is a season.
The honest mobile answer is not to run it anyway and peek. It is to change the question: aim at a 20% relative improvement instead (from 3.0% to 3.6%), and the requirement drops to 13,914 per variation, which at the same 6,000 users per week takes about 33 days. A bigger target effect demands a bigger change, which in practice means testing a redesigned screen instead of a reworded button.
Difference five: what you can and cannot test
Some experiments simply do not exist on the other side of the divide.
- Only on the web: anything that depends on instant deployment, such as testing several headline variations in a single week, and anything driven by search or referral traffic landing directly on the tested page.
- Only in the app: store listing tests (Apple Product Page Optimization, Google Play Store listing experiments), push notification tests, and anything tied to native permission prompts, such as when to ask for notification or tracking permission.
- On both, but with different rules: pricing. On the web you control the checkout and the price display. In the app, Apple App Store Review Guideline 3.1.2 requires the amount charged to be the most prominent and legible pricing element on the purchase screen, which rules out a whole family of price-anchoring layouts that would pass on a web paywall. The mobile paywall testing guide covers those limits in detail.
Difference six: guardrails that only exist in an app
A web test watches load time and error rate. A mobile test has to watch three more, and all three can turn a statistical winner into a business loss.
- Crash rate and app not responding rate per variation. A variation that loads a heavier screen can crash on older devices while converting better on newer ones. Read stability per variation, and per device tier when the volume allows.
- Store rating and review sentiment. A monetization change can win on conversion and cost you a fraction of a star, and rating feeds install volume. That is a slow, compounding loss which no conversion dashboard will surface.
- App size, battery and data usage. Users on entry-level devices and constrained data plans leave silently. Their absence does not appear as a lower conversion rate, it appears as a smaller denominator, which is exactly the kind of loss an aggregate reading hides.
The mistake of pooling mobile and web A/B testing into one result
The most expensive mistake teams make when they run both surfaces is treating them as one experiment with one sample. The two populations have different baseline rates, arrive through different channels and convert on different timelines. Pooling them sets up the classic conditions for Simpson paradox, where each group points one way and the aggregate points the other. Run two experiments, size each one on its own baseline and traffic, and compare the two readings as two independent pieces of evidence. If they agree, your confidence in the mechanism goes up. If they disagree, you have learned something specific about the surface, which is more useful than a blended number that describes neither.
The same discipline applies to SRM. A 10,000-user test that should split 50/50 and observes 5,300 against 4,700 produces a goodness-of-fit chi-square of 36 with 1 degree of freedom, far above the 6.63 threshold at 1%, with a p-value near two in a billion. On the web that usually means a redirect or a bot filter. On mobile it usually means an app version or a config cache, and it invalidates the reading either way until the cause is found.
A worked example: reading a web result honestly
A web checkout converts at 3.00%. The team ships a new payment step layout and runs the test to 26,000 visitors per variation, closing with 780 orders in control and 866 in the variation (3.33%). Running those four numbers through the same two-proportion math this blog uses everywhere: z = 2.15, p-value ≈ 0.0312, with a 95% confidence interval on the difference from +0.03 to +0.63 percentage points, and an observed relative lift of +11.0%.
The result is significant, and the interval is a useful reminder of how imprecise a significant result can be: the true gain could be as small as 0.03 points, which is almost nothing, or as large as 0.63 points. On the web, running longer to tighten that interval costs a few days. Inside an app with 6,000 weekly users on the same screen, tightening the same interval would cost months, which is the real reason mobile programs should prefer fewer, bolder experiments over many marginal ones. The statistical significance guide explains how to read each of those numbers.
Make this automatic with Donnu
If your product lives on both surfaces, the honest split of work looks like this: inside the app, you need remote config, device-level randomization and stability guardrails; on the web, you need a snippet that does not slow the page, automatic sample sizing and statistics that do not invent certainty.
Donnu covers the web and client-side half of that picture: a light snippet, automatic sample sizing and honest Bayesian statistics, with no framework shipped to the visitor. It does not run experiments inside a native iOS or Android binary, and this guide does not pretend otherwise. If your journey includes a landing page, a web checkout or a browser-accessible logged-in area, start a 14-day free trial and apply the same rigor to that half.
Read also: A/B Testing in Mobile Apps: The Complete Guide · Mobile Paywall A/B Testing · Client-Side vs Server-Side A/B Testing
References
- Firebase. A/B Testing with Firebase Remote Config. Official documentation on remote configuration, variant assignment and rollout on iOS and Android. firebase.google.com/docs/ab-testing.
- Apple Developer. App Review Guidelines (section 3.1.2, Subscriptions). Rules on the prominence of the amount charged on the purchase screen. developer.apple.com/app-store/review/guidelines.
- Apple. Product Page Optimization. Store listing testing in App Store Connect, an experiment type with no web equivalent. developer.apple.com/app-store/product-page-optimization.
- Google Play Console. Store listing experiments. Native store listing A/B testing, including text as well as visuals. play.google.com/console/about/store-listing-experiments.
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Supporting material at experimentguide.com.
- Google Play Console Help. Release app updates with staged rollouts. Official documentation on staged release adoption, which is what makes a version-gated variation reach users unevenly. support.google.com/googleplay/android-developer/answer/6346149.
Frequently asked questions
- Is the statistics of a mobile A/B test different from a web A/B test?
- No. The math is identical: the same two-proportion test, the same p-value, the same confidence interval, the same sample size formula. What changes is everything around the math, and that is what changes the result. Mobile has a slower release cycle, a randomization unit tied to the device rather than the browser, a delay between assigning a variation and rendering it, and far less traffic on the screens that matter. Same statistics, harder conditions.
- Why do mobile A/B tests take so much longer?
- Because the denominator is smaller and the propagation is slower. A landing page can see tens of thousands of visitors a week; the paywall screen of a mid-sized app may see a few thousand users a week. Using the math on this blog, detecting a 10% relative improvement over a 3% baseline needs about 53,211 users per variation. At 40,000 weekly visitors on the web, that is roughly 19 days. At 6,000 weekly users inside an app, the same test would need about 125 days, which is why mobile teams usually raise the minimum detectable effect instead of waiting.
- Can I run the same test on my app and my website and pool the results?
- No, and it is one of the most common mistakes. The two populations behave differently, they enter the funnel through different channels, and their baseline conversion rates are usually different. Pooling them creates the conditions for Simpson paradox, where the aggregate direction contradicts the direction in each individual group. Run them as two separate experiments with their own sample size, and compare the readings afterwards as two pieces of evidence, not as one dataset.
- What is the randomization unit in a mobile app test?
- Normally the device or an installation identifier, sometimes a logged-in user account when one exists. That is more stable than a web cookie, which is deleted or partitioned frequently, but it also creates a different problem: a user with a phone and a tablet can land in different variations, and a reinstall can produce a new identifier for the same person. When the same user must always see the same variation across devices, the randomization unit has to be the logged-in account, and users who are not logged in should be excluded from the analysis rather than mixed in.
- Do I still need remote config if my app can ship weekly?
- Yes, for two independent reasons. First, shipping is not the same as adoption: a release only reaches users who actually update, so a version-gated variation always produces an uneven split. Second, a test that lives in the binary cannot be turned off. Without remote configuration, killing a harmful variation requires a new submission, a new review and a new update cycle, while the variation keeps running on every device out there.
- Which guardrails exist on mobile that do not exist on the web?
- Three that matter. Crash rate and app not responding rate per variation, because a variation can be statistically winning while degrading stability on a subset of devices. Store rating and review sentiment, because a monetization change can lift conversion and drag the rating down, which then depresses install volume. App size and battery or data usage, because a heavier variation loses users on entry-level devices and constrained networks, and those losses do not appear in the conversion rate of the users who stayed.