Mobile App A/B Testing: The Complete Guide

Mobile app A/B testing is the same core idea as web A/B testing, splitting the audience at random between two versions and measuring which one converts better, but the execution changes in ways that decide whether the test gives you a trustworthy answer or stalls halfway. There is no instant deploy: every UI or flow change has to travel through a remote feature flag, because shipping a new store build for each variant would be slow and impractical. The randomisation unit is normally the device or the signed-in user, not the session. And the installed base never updates all at once, which creates a specific way of polluting your sample that almost no generic A/B testing guide covers.

This is the hub article for the mobile testing cluster on this blog. It covers what changes between iOS, Android and the web, the three mechanisms that get confused with each other (client-side, server-side and store listing tests), what to test on each screen, the applied statistics with a fully worked example, a neutral tool comparison, and the mistakes that most often void a mobile result. If you are still building the fundamentals, start with the complete guide to A/B testing; if your focus is on the web side, the conversion rate optimization guide covers that terrain in depth. Two child articles go deeper on specific screens: mobile onboarding A/B testing and app store listing A/B testing.

What mobile app A/B testing actually is

A mobile A/B test takes a screen, a flow or a message inside the app (or attached to it, like a push notification) with known behaviour, and tests a variation of it against the current version, with real users, at the same time. The statistical concept is identical to a website test. The difference lies in how the variant reaches the user, and in how you guarantee that the same person always sees the same side despite opening the app dozens of times over several weeks rather than loading a page once.

The mobile-specific vocabulary

• Install ID (device ID) or authenticated user ID: the stable key used to decide which side of the test a user belongs to, replacing the session cookie common on the web.
• Remote feature flag: a server-controlled switch that turns on or off a block of code that already ships inside the published binary, without requiring a new store release.
• Remote Config: the service (Firebase Remote Config is one of the most widely used) that delivers those flag values to the app at runtime.
• Staged rollout / phased release: a store mechanism that releases a new app version to a growing slice of the base over several days. It exists for release safety, not for experimentation. The two are easy to confuse because both “show different things to different people”, but a staged rollout has no control group and computes no significance; it only limits the blast radius of a bad build.
• Minimum eligible version: the binary version from which the variant code exists at all. Users below it must be excluded from the denominator, not merely from exposure.
• Install cohort: the group of users who installed the app within the same time window, useful for isolating the effect of a change on people who arrived after it shipped.

How mobile testing differs from web testing

Four differences change how you design, run and read a test inside an app compared with a website.

Randomisation unit. On the web, many tools still think in sessions (the browser cookie). In an app, the unit almost always has to be the device (a stable install ID) or the authenticated user, because the same person may open the app dozens of times across weeks and must land on the same side every time. If assignment happens per app session, the same user can flip between A and B across launches, and their conversion belongs to neither side.

Store release cycle instead of instant deploy. A website publishes a change in minutes. An app has to submit a new build to the App Store or Google Play, wait for review (hours to a few days, varying by platform and period) and then wait for the installed base to update organically, which takes weeks. That is why practically every serious mobile testing programme depends on remote feature flags: the variant already exists inside the approved binary, and the server decides in real time which branch each user sees, without a new release per test.

Propagation latency. Even with a feature flag, part of your base is in airplane mode, has not opened the app for days, or sits on a device with aggressive battery policies that delay syncing with the remote config service. The test is switched on for everyone at once, but real exposure spreads over hours or days.

App version fragmentation. Unlike the web, where everyone loads the newest page on every visit, an app has several binary versions running live at the same time: people who updated yesterday, people who updated a month ago, and people who have not updated in six months because automatic updates are off. If your test depends on code that only exists in recent versions, users stuck on old builds sit outside real randomisation while still generating product events that can leak into an aggregate metric.

Three mechanisms that get confused: client-side, server-side and store listing

“Mobile A/B testing” actually covers three quite different mechanisms, and it is common to mix them up because all three compare two versions of something related to the app.

• Client-side (remote feature flag). The variant lives inside the already-published binary. A feature flag SDK (Firebase Remote Config, LaunchDarkly, Statsig and others) fetches configuration from the server on launch or sync, and the app itself decides what to render. This is the standard model for testing interface, copy, screen order and small flow changes inside the app.
• Server-side. The decision is made in the backend, and the app simply asks “what should I show this user” and renders the answer. This is the more robust model for business rules (pricing, offer eligibility, recommendation logic) and for any experiment that also has to stay consistent across app, website and transactional email, because the same backend decision feeds all three.
• Store listing tests. Technically this is not a test “in your app”: it is an experiment run by the store itself on the listing page, before installation. Google offers it as Store listing experiments in Google Play Console; Apple offers Product Page Optimization in App Store Connect. Both test icon, screenshots and preview video against real store search and browse traffic. Google also allows testing the listing text (the app descriptions), which Apple’s Product Page Optimization does not cover, since it is restricted to icon, screenshots and video. The metric you read in both cases is listing view to install conversion, never an in-app event.

Side by side

Criterion	Client-side	Server-side	Store listing
Where the decision is made	Inside the app, reading a remote flag	In the backend, the app only renders	Inside the store (Google Play / App Store)
Typically tests	Interface, copy, screen order, small flows	Price, eligibility, business rules, recommendations	Icon, screenshots and preview video (both stores); app descriptions on Google Play only
Cross-channel consistency	App only	High, the same decision can feed app, web and email	Not applicable, it is only the store listing
Who operates the tool	Product or mobile team, feature flag SDK	Backend or growth team	Google Play Console / App Store Connect
Typical metric	In-app conversion (activation, upgrade)	Business conversion (revenue, retention)	Listing view to install rate
Depends on store review	No, runs on an already-approved binary	No	Not review as such, it runs inside the platform’s own flow

What to test on each screen

Not every screen deserves the same testing effort. These concentrate most of the upside, in rough order of impact.

Screen or moment	What people usually test	Specific caution
Onboarding / tutorial	Number of screens, skippable or not, asking for permissions before or after showing value	The biggest activation lever; drop-off here never reappears further down the funnel
Paywall / upgrade	Price position, plan anchoring, trigger for showing it (time, feature, usage)	Directly governs revenue; sensitive to app version and to store payment rules
Push notification	Send time, copy, behavioural segmentation	Careful with time zones on a global base; measure the downstream action, not the isolated tap
Navigation / information architecture	Tab bar versus side menu, item order, section naming	Structural change, hard to revert without confusing returning users
Pricing	Plans, trial length, entry discount	Same caution as any pricing test: long-term effects on lifetime value are slow to surface
In-app checkout	Payment flow, number of steps, accepted methods	In-app purchase rules from Apple and Google on commission and payment systems change often, driven by platform and regulatory decisions; confirm the current rule before designing the test

On that last point: in-app purchase policies at Apple and Google have already gone through material changes in commission and in the obligation to use the store’s own payment system, across different markets and periods. Treat this as something that moves, not as a fixed rule: check the current official App Store Connect and Google Play Console documentation before designing any in-app checkout test, instead of replicating what was true a year ago.

Push notification testing shares its logic with email marketing (the same reasoning about subject line and send time applies), so if your messaging programme covers both, apply the same statistical discipline to each channel with its own separate sample. The complete statistical guide to A/B testing email marketing covers that side in depth.

Writing a mobile test hypothesis

The same hypothesis discipline that applies to any A/B test applies here, except the observation that triggers it usually comes from product analytics (an event instrumented in the app), not from a website heatmap. The format is unchanged: because I observed [product data], I believe that [change to the screen or flag] will produce [effect], measured by [product metric].

A concrete mobile example: “because 38% of users abandon onboarding at the notification permission screen (from the instrumented activation funnel in the app), I believe that moving that request to after the user completes their first valuable action will reduce abandonment, measured by onboarding completion rate at D1”. That sentence already carries the primary metric (onboarding completion at D1), the expected direction (up) and the origin (a real product event, not a team hunch). Weak mobile hypotheses tend to sound like “what if we made the app prettier”, with no metric and no direction. Discard that shape before spending a development cycle on it. The guide to writing an A/B test hypothesis has the full template.

Primary metric and mobile-specific guardrails

All the logic of primary, secondary and guardrail metrics from classic A/B testing applies unchanged, but mobile adds guardrails that barely exist (or matter much less) in a website test.

Guardrail	Why it matters in mobile	Warning sign
Crash rate	A variant can convert better and still crash the app more often on a specific device or OS version	Crash rate rises in one variant only
Store rating and sentiment	An aggressive flow change (paywall too early, insistent permission prompts) can lift a short-term metric while damaging the public rating	Average rating drops, or negative reviews start naming the tested change
Uninstall rate	Conversion gains bought with perceived friction show up here before they show up in subscription churn	Uninstalls rise in the “winning” variant
Cold start / load time	Any feature flag or remote config SDK adds a network call; implemented badly, it delays the first screen	Time to first interactive screen degrades in the flagged variant

Ignoring these guardrails is like celebrating a website test that lifted conversion while quietly doubling cancellations: the number that decided the test went up, and the business went down.

The statistics of mobile: why the sample takes longer

The maths behind a mobile A/B test is exactly the maths behind a web A/B test (the same normal approximation for two proportions), but three factors make it harder in practice.

1. Smaller eligible traffic. A marketing site can see tens of thousands of visitors a week on a single page. An app usually has a fraction of that on a deep funnel screen (paywall, checkout), because the audience has already passed through an installation filter to get there.
2. App update cycles. If the test depends on a new binary, part of your target audience is simply not eligible yet because they have not updated. The denominator of “everyone who should enter the test” grows at the pace of organic updates.
3. Contamination from old versions. Users on an old build, with neither the variant code nor the flag, keep generating product events (opens, conversions) that can land in an aggregate metric if the analysis pipeline does not filter by minimum eligible version, distorting the reading before anyone notices.

Size your own mobile test

Set the conversion rate of the screen, the minimum effect you want to detect and your real weekly volume (not website traffic, the eligible user volume on that specific screen):

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

A worked example with real numbers

Take a common mobile scenario: an activation screen (the user completes the core action within the first 7 days, the “D7” milestone) with a baseline rate of 22%, testing an onboarding change the team expects to improve by +15% relative (from 22% to roughly 25.3%). At 95% confidence and 80% power, the same defaults used by every calculator on this blog, the sample size formula returns 2,602 users per variant.

If the app gets 2,500 new eligible installs per week (a realistic figure for a mid-sized app, and far below the traffic of a marketing landing page), the test needs 15 days to fill both variants. About two weeks of waiting, even with a generous +15% target effect. That arithmetic is precisely why mobile tests on deep funnel screens take longer than landing page tests: it is not sloppiness, it is a smaller eligible volume by default.

Now look at what happens when the team gets impatient and reads the result at 900 users per variant (roughly a third of what is needed), with the same observed proportions:

	Full sample (2,602 per variant)	Partial sample (900 per variant)
Rate A (control)	21.98% (572 / 2,602)	22.00% (198 / 900)
Rate B (variant)	25.60% (666 / 2,602)	25.67% (231 / 900)
Observed relative lift	+16.4%	+16.7%
p-value	0.0022	0.0679
Confidence interval of the difference	+1.30 to +5.92 points	−0.27 to +7.60 points
Verdict	Significant, B wins	Inconclusive

The observed improvement is almost identical in both cases (the variant really does look about 16% better), but the verdict flips completely. With the full sample, the p-value lands at 0.0022 and the confidence interval of the difference never crosses zero (it sits between +1.30 and +5.92 percentage points), a solid result. With a third of the sample, the same real effect produces a p-value of 0.068, above the 0.05 threshold, and the interval crosses zero (−0.27 to +7.60 points), meaning “no difference at all” is still plausible. That does not mean the partial test “failed”: it means there was not enough sample for the same real effect to become statistical evidence. This is exactly why you compute N before running instead of deciding by eye halfway through, and it is the same lesson the peeking problem article covers in depth.

SRM in mobile: the old app version is the prime suspect

Sample Ratio Mismatch happens when the observed split between variants drifts away from the configured one because of a collection defect rather than chance. In mobile, two causes dominate the suspect list:

• Forced update bug. An “update to continue” screen that appears for one variant only pushes users on older versions disproportionately toward the other side, and people who update early tend to have a different engagement profile from people who delay.
• Variant-isolated crash. If variant B has a bug that kills the app before the exposure event fires, the app “disappears” from B’s count without even registering as abandonment, artificially inflating A’s share of the observed total.

A concrete example: a test configured at 50/50 ends up recording 7,150 users in variant A against 4,850 in B, out of 12,000 total. The chi-square goodness-of-fit check returns chi-square ≈ 440.8 with a p-value below 0.0001, far under the 1% alert threshold, meaning a split of that magnitude is statistically impossible to obtain by chance from a 50/50 configuration. Every conversion number in that test is suspect until the cause of the divergence is found and fixed, even if the gap between A and B looks clean and favourable at first glance.

How long to run a mobile test

The general rule does not change: run for at least one to two full weeks even if the calculated sample arrives earlier, so the test covers complete weekday and weekend cycles. Mobile adds two factors that stretch that timeline in practice. The first is flag propagation: if the test depends on a newer app version, the real clock for “enough eligible sample” only starts once a meaningful slice of the base has updated, not on the day you flipped the switch in the dashboard. The second is app-specific cyclicality: weekend usage in consumer apps (games, entertainment, social) tends to differ sharply from weekday usage in productivity or finance apps, and closing a mobile test after three or four days captures a biased slice of your base just as it would on a website.

If the test involves push notifications, the time zone caution mentioned earlier also affects duration: a test that “runs for 14 days” on server time may in practice expose part of the base to fewer complete send cycles, if the trigger always fires at a fixed server hour and a slice of the audience lives in time zones where that hour falls in the middle of the night.

Pre-flight checklist (copyable asset)

Before switching on a test in production, confirm every item below. It is the mobile version of the A/A test and the execution care any honest A/B test demands, adapted to the parts that only exist in apps.

Item	Confirm before switching on
Randomisation unit	Stable device ID or user ID, never an app session
Minimum eligible version	Users below it are excluded from the denominator, not only from exposure
Flag tested on both platforms	The flag renders correctly on iOS and Android before the test goes live
SRM check configured per platform	Chi-square runs separately for iOS and Android, on top of the aggregate
Guardrails instrumented	Crash rate, uninstalls and (where relevant) store rating monitored from day one
Sample and duration computed in advance	N per variant and timeline set with the calculator above, not “let’s see how it goes”
Time zones handled (if push is involved)	Sending and reading segmented by user time zone, not server hour
Prior A/A test (if this is your first test in the tool)	Two identical variants show no significant difference and no SRM, confirming collection is clean

Treat this list as a quality gate, not bureaucracy. Every line exists because one of the causes of an invalidated mobile test has already shown up in a real experimentation programme.

Mobile A/B testing tools (neutral view)

There is no single right tool. The deciding criteria are always the same: client-side or server-side, cost per event or user, and integration with the mobile analytics you already run (Firebase Analytics, Amplitude, Mixpanel and similar).

Tool	Model	Strength	Consider when
Firebase A/B Testing + Remote Config	Client-side	Free up to a high volume, native integration with Firebase Analytics and Cloud Messaging	Your app already lives in the Firebase ecosystem and the test is in-app interface or flow
Optimizely Feature Experimentation (formerly Full Stack)	Client-side and server-side	Mature, supports multiple SDKs (mobile, server, web) under the same experiment	You need one experiment coordinated across mobile and backend
LaunchDarkly	Client-side and server-side (flags first)	Strong focus on release flags and progressive rollout, with experimentation as an added layer	The main goal is safe release control and testing is a bonus on top
Statsig	Client-side and server-side	Statistics and feature flags integrated from the start, good product metric support	You want experimentation and product analytics in one tool
PostHog	Client-side and server-side (open source)	Product analytics, feature flags and experiments in one suite, with a self-hosted option	You already run PostHog for analytics and want to unify experimentation
VWO Mobile	Client-side	Comes from the web CRO world, familiar to teams already testing landing pages	Your marketing team already uses VWO on the web and wants the same mental model in the app
AB Tasty App	Client-side	Visual editor aimed at marketing, less engineering-dependent for simple variations	The people creating tests are marketing or product, not only engineering

One final criterion beyond the table: in most of these tools cost scales with event or unique user volume, not with the number of tests running. An app with a few million events a month usually fits inside free or entry tiers at Firebase, Statsig or PostHog; high event volume is where the pricing conversation changes character between options, and it is worth asking for a real quote instead of assuming from the public tier page. For a broader view of the category, see the neutral comparison of CRO tools.

Donnu A/B today is focused on the web and client-side side: a lightweight snippet that never blocks the page, and honest Bayesian statistics. It does not cover native mobile apps (iOS and Android), and this guide does not claim otherwise. If your product is a hybrid app or has a meaningful web layer (web view onboarding, web checkout, a landing page that feeds installs), Donnu already handles that slice with the same statistical rigour this guide argues for across the whole app.

Mobile-specific mistakes

• Testing with unbalanced device populations without checking SRM per platform. iOS and Android tend to have different user profiles, behaviour and conversion rates. Running the SRM check only on the aggregate can hide an imbalance that exists inside one platform.
• Ignoring behavioural differences between iOS and Android. A strong aggregate win can be driven entirely by one platform while hiding a loss on the other. Whenever volume allows, read the result segmented by platform before generalising the decision to the whole app.
• Ignoring time zones in push notification tests. A global base is not awake at the same time. Sending and measuring push at a fixed server hour, without adjusting for user time zone, mixes people at peak attention with people asleep, distorting both the test and the experience.
• Ignoring outdated app versions as a source of noise. As covered in the statistics section, users stuck on an old build without the variant code keep generating events that can contaminate aggregate metrics unless the pipeline filters by minimum eligible version.
• Running the test on the conversion metric alone, with no guardrail. A more aggressive paywall can lift upgrade conversion in the short term while raising cancellations and negative reviews weeks later. Without guardrails instrumented from day one, that effect only surfaces when it is expensive to reverse.
• Trusting a small sample because of launch pressure. It is tempting to declare a winner the moment the dashboard shows any improvement, especially when the team already wants to move on to the next feature. The worked example above shows how the same real effect reads as “inconclusive” or “significant” purely as a function of sample size.

When a mobile A/B test is the wrong tool

Forcing a test that will never close is worse than not testing at all.

Scenario	Why it does not work	What to do instead
App with a few hundred new installs a week	The sample calculation easily asks for months on a rare conversion screen	Apply known mobile UX heuristics and remove obvious friction without demanding statistical significance; the low-traffic CRO article covers the same trade-off for websites
Obvious, risk-free fix	A crash, an unresponsive button, wrong copy: these are bugs, not hypotheses	Fix it directly, without running a test to “confirm” the obvious
The question is “why”, not “how much”	An A/B test measures the effect of a change, it does not explain the behaviour behind it	Use qualitative research: user interviews, session recordings, usability tests
The change depends on a new binary most of the base does not have yet	Until the update propagates, the eligible sample is too small for any reliable reading	Wait for propagation to reach a reasonable floor, or use a feature flag so this test does not depend on a new release

Do this automatically with Donnu

This guide covered what genuinely changes when A/B testing leaves the browser and enters the app: randomisation per device, remote feature flags instead of instant deploy, samples that take longer to fill, and an SRM that usually starts life as an update bug or an old build in the field. It is the same statistical rigour this blog defends everywhere, applied to terrain with more operational friction.

Donnu solves the web and client-side part of that discipline today: a lightweight snippet that never blocks the page, automatic sample sizing and honest Bayesian statistics. If your product already mixes app and web (a landing page feeding installs, a web view checkout or onboarding, a logged-in area reachable from both browser and app), the natural next step is bringing the same statistical discipline to your site with a 14-day free trial, while this mobile cluster grows with child articles covering each screen in detail.

---

Read also: Mobile Onboarding A/B Testing · App Store Listing A/B Testing · Feature Flags vs A/B Testing · Leia em português

References

• Firebase. A/B Testing. Google’s official documentation on testing UI, feature and campaign changes with Remote Config and Cloud Messaging. firebase.google.com/docs/ab-testing.
• Firebase. Remote Config. Official documentation on the remote configuration service that changes app behaviour and appearance without a new release. firebase.google.com/docs/remote-config.
• Google Play Console. Store listing experiments. Official page on Google Play’s native listing A/B test, run against real store traffic. play.google.com/console/about/store-listing-experiments.
• Apple. Product Page Optimization. Official documentation on the App Store Connect feature that tests up to three variants of icon, screenshots and preview video against the original page. developer.apple.com/app-store/product-page-optimization.
• Microsoft Research. Diagnosing Sample Ratio Mismatch in A/B Testing. microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing.