Mobile Onboarding A/B Testing: What Improves Retention
Mobile onboarding a/b test done right: the real activation metric, what permission timing costs you, and the statistical trap that fools most apps.
📚 This article is part of the guide Mobile App A/B Testing: The Complete Guide.
A mobile onboarding A/B test only produces a trustworthy answer when the metric deciding it is real activation, the completion of a specific core action inside a fixed day window, rather than a vanity metric like “tutorial screens viewed”. This article is part of the complete guide to mobile app A/B testing and covers the single most common and most badly measured mobile experiment: onboarding. You will see which metric to use, what the small screen and the system permission prompts change about conversion, the bias that inflates results without anyone noticing, and the statistical trap unique to apps with several versions live in the field.
The real activation metric: D1, D7, D30 and the core action
The most common trap in mobile onboarding testing is optimising what is easy to count (screens viewed, buttons tapped, percentage of tour completed) instead of what actually predicts whether the user comes back. Mobile growth practice works with a retention curve anchored on three milestones: D1 (the user reopens the app the next day), D7 (they return and repeat the core action within a week) and D30 (the habit has consolidated). None of the three is “watched the whole tutorial”. Every product has its own core action: for a finance app it might be connecting an account; for fitness, logging the first workout; for messaging, holding the first real conversation. Define yours before testing anything in the welcome flow.
Market benchmarks help calibrate expectations, never to set a target. UXCam’s survey of mobile retention reports these bands for strong performers, the 75th percentile of each category:
| Category | D1 | D7 | D30 |
|---|---|---|---|
| Social | 50% to 60% | 25% to 30% | 15% to 20% |
| Fintech | 35% to 45% | 18% to 25% | 10% to 15% |
| Productivity | 40% to 50% | 22% to 28% | 12% to 18% |
| Games | 40% to 50% | 12% to 18% | 5% to 8% |
| Ecommerce | 25% to 30% | 8% to 12% | 3% to 6% |
Read those as the top quartile, not as the average. According to the same UXCam survey, D30 below 5% is the norm across most app categories rather than the exception, with the median sitting around 4%. If your app sits above that, the issue is not the general curve, it is how far your onboarding pushes people up or down it, and only an A/B test with the right metric can measure that.
Small screen, big decisions: what changes from web onboarding
In web onboarding the user has a large screen, several tabs open and the option to come back later. On mobile, every element on screen competes for far scarcer attention: physical space is smaller, sessions are shorter and a distraction (another app’s notification, a phone call) is one tap away. That changes what is worth testing.
- One element per screen carries more weight. A full signup form on a mobile screen usually converts worse than the same form split into short steps, because every extra visible field competes directly for the user’s thumb.
- The operating system permission prompt lands in the middle of the flow. This is the most concrete and most measurable difference between mobile and web onboarding: asking for push, location or camera access is a native interruption the browser does not have, and when it appears directly affects how many people finish onboarding.
- App updates are not instant. On the web, publishing a change affects everyone on the next page load. On mobile, a new onboarding variant only reaches people on an app version that supports it, which creates a statistical trap of its own, covered further down.
The decisive point: when to ask for system permission
Asking for push, location or camera access on the first onboarding screen is the most repeated mobile UX mistake, and its cost is measurable rather than a vague sense of “friction”. Android’s official documentation on permission best practices recommends requesting a permission in the context of use, at the moment the user is about to use the feature that needs it, and not generically at app start: users accept more often when they understand why they are being asked, and refuse more often when the request arrives without context. The same pattern shows up on iOS: priming screens (an explanation of the value before the official system prompt) tend to raise acceptance, while pushing the prompt on first launch tends to lower it.
The funnel below marks where that request typically lands in a self-serve onboarding flow, and the size of the drop that appears right there:
Treat the moment of the permission request as any other testable variable: variant A asks on first launch, variant B asks only when the user taps a feature that requires it. Measure the effect on onboarding completion and on real activation, because the permission itself is not the goal, it is only a means.
Survivorship bias attacks mobile onboarding the same way
The same statistical mistake that invalidates web SaaS onboarding tests shows up here, and the sibling article on SaaS onboarding A/B testing already describes the general logic: comparing activation only among people who finished onboarding, ignoring everyone who dropped out midway, is comparing two groups that have already been filtered in different ways. In mobile this tends to be more severe, because the permission screen and the small screen together produce more early abandonment than most web flows.
Reusing the numbers from the funnel above: 1,000 people installed the app, 640 completed onboarding and 220 performed the core action within 7 days. Measure activation only among finishers (the common mistake) and the number looks great. Measure it against everyone who entered and the honest number is much smaller:
In practice, define the activation window (D7, for example) counted from install, and count every install that entered the test in the denominator, including the ones that never reopened the app. It is less comfortable to look at, and it is the only number that represents what actually happened.
The mobile-only trap: stuck on an old version, and the SRM nobody catches
There is a statistical problem that barely exists on the web and is common in mobile: part of your users get stuck on an older app build, either because the onboarding change only exists from a newer store release, or because the device has not fetched updated remote config values yet. According to Firebase Remote Config documentation, the default minimum fetch interval in production is 12 hours: an app can keep running on cached configuration for up to half a day after you publish a change, and a user who rarely opens the app can stay on the old build far longer than that.
The practical effect: if the test split depends on the device having received the new variant, whoever is stuck on the old version never enters the experiment the way they should, and the traffic left for each side stops matching the 50/50 you configured. That is exactly what Sample Ratio Mismatch (SRM) detects: a goodness-of-fit test between the observed and expected split. Microsoft’s research on diagnosing SRM in A/B tests treats the problem as serious enough to block reading any result until it is ruled out, and the user stuck on an old app build is the most common mobile flavour of that same defect: part of the audience never had a real chance of landing on the expected side.
Here is how it shows up numerically. A test configured for 50/50 (6,000 expected per side out of 12,000 total) but with a slice of users stuck on the old build produces a very different split:
The practical fix: segment the SRM check by app version, not only on the aggregate. If the imbalance disappears once you exclude old builds from the calculation, the problem is confirmed, and the answer is forcing an update (or excluding incompatible versions from the split), never “ignore it and read the activation result anyway”.
Mobile onboarding versus SaaS web onboarding
The statistics behind an onboarding A/B test are the same in any channel: sample sized before running, full denominator against survivorship bias, and no significance reading without checking split health first. The SaaS onboarding A/B testing article covers exactly that discipline for web products and is worth reading alongside this one. What changes in mobile are the execution details: the variant depends on which app version the user has installed (the web has no such concept, every visit already loads the newest code), and delivering a change through remote config has real propagation latency until the device fetches the new configuration, instead of happening instantly on the next page load. Ignoring those two differences is the most common way to import a textbook web A/B test and have it fail silently in mobile.
Guided tour, free sandbox or progressive onboarding
The three most common mobile onboarding structures solve different problems, and none of them wins every time:
This table summarises what to test at each stage of mobile onboarding and where the risk of fooling yourself lives:
| Stage | What to test | Common hypothesis | Risk of fooling yourself |
|---|---|---|---|
| System permission | When to ask for push, location or camera (first launch vs after a valuable action) | Asking with context raises acceptance | Measuring only opt-in rate treats the permission as the win; measure the effect on activation, not on acceptance |
| Flow structure | Guided tour vs free sandbox vs progressive onboarding | Less cognitive load on a small screen raises completion | Finishing the tour is not activating; measure the core action, not screens viewed |
| Screen density | How many elements per screen (one CTA vs several) | Emptier screens reduce abandonment | A screen emptied too far can hide information people needed, which only shows up in retention |
| Re-engagement push | Post-install notification (timing and trigger) for people who did not return | The right reminder recovers usage | It only reaches people who accepted push; comparing without adjusting the denominator reintroduces survivorship bias |
| Variant delivery | Rollout via new store release vs remote config | Remote config reaches everyone faster | Local cache and users stuck on old builds distort the 50/50 (check SRM per version) |
Sizing the test: how many new users you need
Suppose your app activates 22% of new signups within 7 days today (the validated core action, in the D7 window) and you want to detect a 15% relative improvement from a new placement of the permission screen, taking activation to roughly 25.3%. At 95% confidence and 80% power, the market default parameters, adjust the calculator below for your own scenario:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Running those numbers through the same maths the calculator uses (sampleSizePerVariant), the answer is 2,602 signups per variant (5,204 in total). With 2,500 new installs a week, the test takes about 15 days to gather that sample, enough to cover at least two full weeks of behaviour including weekends.
Here is why skipping that calculation is the most common way to throw away a good onboarding idea. Simulating the same real effect used above (22% against roughly 25.3%, using significance()) at two different sample volumes:
| Scenario | N per variant | Activated (A / B) | p-value | Verdict |
|---|---|---|---|---|
| Sized sample | 2,602 | 572 / 658 | ≈ 0.0050 | Significant, B wins |
| Insufficient sample | 900 | 198 / 228 | ≈ 0.0962 | Inconclusive, same real effect |
The difference between “it worked” and “it did not work” did not come from the real effect changing, it is the same activation gain in both rows. The only thing that changed was sample size. Early-stage mobile onboarding teams with a few hundred installs a week are especially vulnerable to this trap: they run the test for 10 days, read “not significant” and discard an idea that actually worked, only under-powered.
Before running any onboarding test, write the hypothesis first: the guide to writing an A/B test hypothesis has the exact format and stops you from “discovering” a winner among dozens of secondary metrics purely by chance.
Do this automatically with Donnu
Declaring a mobile onboarding winner without correcting survivorship bias, or without checking SRM per app version, is the most common way to celebrate a result that does not exist. Donnu A/B applies the same significance engine used in this article to every variant, always against the full denominator of everyone who entered the test, and the lightweight snippet never blocks your product’s onboarding flow. Before declaring any winner, size the test with the calculator above and confirm split health segmented by app version, not only in aggregate.
Start a 14-day free trial and size your next onboarding test with real rigour instead of the volume that “looks like enough”.
Read also: Mobile App A/B Testing: The Complete Guide · A/B Testing SaaS Onboarding · How to Write an A/B Test Hypothesis · Leia em português
References
- Android Developers. App permissions best practices (timing and context of the permission request). developer.android.com/training/permissions/usage-notes.
- UXCam. Mobile App Retention Benchmarks by Industry. uxcam.com/blog/mobile-app-retention-benchmarks.
- AppsFlyer. 5 ways to increase your ATT opt-in rates (priming before the system prompt). appsflyer.com/blog/tips-strategy/apps-boost-att-opt-in.
- Microsoft Research. Diagnosing Sample Ratio Mismatch in A/B Testing. microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing.
- Firebase. Get started with Remote Config on Android (default fetch and cache interval). firebase.google.com/docs/remote-config/android/get-started.
Frequently asked questions
- What is the real activation metric for mobile onboarding?
- It is the completion of a specific core action of your product inside a fixed day window, such as D7 or D30, counted from install rather than from the end of the tutorial. "Onboarding screens viewed" or "tour completed" are vanity metrics: they describe whether someone walked through an introduction, not whether they came back. Validate your core action by cross-referencing who performed it with who is still active at D7 and D30, before testing any onboarding variation.
- Does asking for push, location or camera permission during onboarding hurt conversion?
- Yes, and the effect is measurable rather than merely felt. Android permission best practice recommends requesting a permission in the context of use rather than at app start, because users accept more often when they understand why they are being asked. Reports from mobile measurement vendors on iOS tracking opt-in point the same way: a priming screen that explains the value before the system prompt tends to raise acceptance, while pushing the request on first launch tends to lower it. The exact timing is testable, so treat it as one more variant in your onboarding test, not as a fixed technical detail.
- Why can a mobile onboarding test look significant and still be wrong?
- The most common reason is survivorship bias: measuring activation only among people who finished onboarding instead of everyone who entered the test. If one variant drives more people away early (a badly placed permission screen, for instance), the people left over are naturally more engaged, and the rate "among finishers" looks artificially better even when real activation across everyone is flat or worse. The correct comparison always uses the full denominator: everyone who installed and entered the test.
- What breaks the 50/50 split when testing onboarding in a mobile app?
- The most mobile-specific cause is a user stuck on an older app build who never receives the new variant, either because the change only exists from a newer store release or because the local remote config cache has not fetched the updated values yet. That pushes disproportionate traffic to one side and breaks the split you configured. Run a Sample Ratio Mismatch check segmented by app version before trusting any activation result.
- Guided tour, free sandbox or progressive onboarding: which converts better?
- There is no universal winner, which is exactly why this is a test rather than a design preference. Products with a single obvious path to core value tend to win with a guided tour. Products whose value depends on the user context (their own data, integrations, varied use cases) tend to win with a free sandbox or progressive onboarding that only asks for what is needed as usage advances. Measure by real activation in the fixed day window, never by flow completion rate.