App Store A/B Testing (ASO): What You Can and Cannot Test
App store a/b testing aso explained: what Google Play and Apple let you test, how each measures a winner, and how to size a store listing test properly.
📚 This article is part of the guide Mobile App A/B Testing: The Complete Guide.
App store A/B testing is a test the platform runs on your listing page, before anyone installs, and it is measured on one number: listing view to install conversion. It is not the same mechanism as an A/B test inside your app, it does not use your code, and no third-party tool can run it for you. This article is part of the complete guide to mobile app A/B testing and covers what Google Play and the App Store actually let you test, how each one decides a winner, how to size a store test with real numbers, and the traps that make a “winning” listing fail to move installs after launch.
Where a store listing test sits in the funnel
Everything an app team can test splits cleanly into two zones, separated by the install itself.
What each app store lets you test
The two platforms are close in spirit and different in coverage. Both run the experiment against real store traffic, split it themselves, and report conversion for each variant. What you are allowed to vary is where they part ways.
| Google Play (Store listing experiments) | App Store (Product Page Optimization) | |
|---|---|---|
| Visual assets testable | App icon, feature graphic, screenshots, preview video | App icon, screenshots, app previews |
| Listing text testable | Yes, the app descriptions | No, Product Page Optimization does not cover name, subtitle or description |
| Where it is configured | Google Play Console | App Store Connect |
| Traffic used | Real Google Play search and browse traffic | Real App Store traffic, with a configurable share allocated to the test |
| Maximum duration | Google documents an automatic stop after 6 months | Apple documents up to 90 days |
| Reported metric | Install clicks per variant, reported with a confidence interval | Conversion rate for each treatment against the original |
| Localisation | Experiments are configured per store listing localisation | Treatments can be localised per market |
Two consequences follow from that table. First, if your hypothesis is about wording (“does calling it a planner instead of a to-do list raise installs?”), Google Play can answer it natively and the App Store cannot: on iOS you would have to change the metadata and compare periods, which is not a controlled experiment. Second, an icon test is the one hypothesis both stores answer the same way, which is why icon is the most common first test on both platforms.
There is one more Apple feature people confuse with a test: custom product pages, which are alternative versions of your page with their own URLs, used to match a specific acquisition campaign to a specific message. They are a targeting tool, not an experiment: there is no random split and no control group. Use them to align paid traffic with the page it lands on, and use Product Page Optimization when you want a controlled comparison.
What you cannot test in the store
The list of things outside the built-in experiments is worth stating plainly, because a lot of ASO advice glosses over it:
- Price and in-app purchase tiers. Not part of either store experiment. Pricing experiments belong to your backend or to a promotional strategy, and they carry their own fairness and communication considerations.
- Keyword metadata as a controlled test. You can change keywords and watch what happens, but neither store splits traffic between two keyword sets, so what you get is a before-and-after comparison contaminated by seasonality, competitor moves and ranking drift.
- Category and age rating. Configuration, not creative.
- The app itself. Anything after the install is a feature flag test, not a store test.
- Cross-store comparisons. A variant that wins on Google Play tells you nothing statistically valid about the App Store. Different audiences, different page layout, different traffic mix. Run each test where it belongs.
How the stores decide a winner, and why you should still do the maths
Both platforms report their own confidence estimate, and both are honest that a result inside the uncertainty band is not a result. That is genuinely useful, and it is also where most teams stop thinking, which is a mistake for two reasons.
The first is that a store’s built-in verdict answers “is this difference distinguishable from noise”, not “how long will I wait, and is that wait worth it”. Only a sample size calculation answers the second question before you commit weeks of store traffic to a test.
The second is that store traffic is not yours to increase on demand. On a website you can point ads at a page to accelerate a test. On a store listing, the experiment consumes organic impressions at whatever rate the store sends them, which means an underpowered listing test does not just fail, it fails slowly.
Size the test before you launch it
Set your current listing conversion rate, the minimum improvement worth shipping, and your weekly listing views:
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
A worked example with real numbers
Take a listing converting at 28% of store page views into installs, a plausible figure for an established app with a clear category fit. Three target effects, at 95% confidence and 80% power, produce very different commitments:
| Target improvement | New rate | Sample per variant | At 10,000 listing views/week | At 50,000 views/week |
|---|---|---|---|---|
| +5% relative | 29.4% | 16,388 | 23 days | 5 days |
| +10% relative | 30.8% | 4,155 | 6 days | 2 days |
| +15% relative | 32.2% | 1,872 | 3 days | 1 day |
Read that table before designing the creative, not after. Chasing a +5% relative lift on a listing that sees 10,000 views a week means committing more than three weeks of store traffic to one comparison; chasing a +15% lift with a genuinely different icon concept closes in days. This is the same trade-off between ambition and patience that applies to any test, and the low-traffic testing article covers the general version of it.
Now the result side. Suppose the test ran to 30,000 views per variant and produced 8,400 installs on the control (28.0%) against 8,820 on the treatment (29.4%), exactly the +5% relative lift from the first row. Running those through the same significance engine used across this blog gives z = 3.79, p-value ≈ 0.00015, with a confidence interval on the difference of +0.68 to +2.12 percentage points. The interval never touches zero, so the treatment genuinely converts better. Note how modest the honest claim is: the improvement is somewhere between roughly two thirds of a point and two points of conversion, not “the new icon boosted installs by 5%” stated as a certainty.
The traps specific to store tests
Seasonality eats small effects. Store traffic mix changes with holidays, platform editorial features, competitor launches and paid campaigns you are running elsewhere. A test that runs across a category-wide spike is comparing two variants under conditions neither will face afterwards. Prefer a stable window, and treat a result found only during an anomalous period as provisional.
Traffic source mix is not constant. Someone who arrives from a store search for your brand name behaves differently from someone browsing a category chart, who behaves differently again from someone tapping an ad link. If your test runs while a paid campaign is pushing atypical traffic to the page, the winning variant is the one that suits that mix, not necessarily the one that suits your normal mix.
Localisation is not translation. A screenshot set that wins in one market frequently loses in another, because the visual conventions, the competitive set and the reasons people install differ. Run the test per localisation instead of generalising one market’s winner to all of them.
Installs are not the goal. This is the most expensive trap on the list. An aggressive listing, one that promises more than the product delivers, reliably raises install conversion and just as reliably lowers day-7 retention, because it recruits people the product was never going to satisfy. That is why the guardrail below matters more than the primary metric.
The guardrail: retention of the cohort the test recruited
Both stores report install conversion and stop there, which is reasonable, since they cannot see inside your app. You can. Tag the install cohort by test variant in your own analytics and compare day-1 and day-7 retention between them, alongside the store’s conversion verdict.
The rule to adopt: ship the variant that wins on retained users, not on installs. If variant B raises install conversion by 5% relative and drops day-7 retention by more than that in relative terms, B is a worse listing that happens to look better on the store dashboard. This is the same reasoning as guardrail metrics in any experiment, covered in the common A/B testing mistakes article.
Pre-flight checklist for an app store listing test
| Item | Confirm before launching |
|---|---|
| Hypothesis written down | A specific claim about why the new asset should convert better, not “let’s try a fresher icon” |
| Effect size chosen in advance | The minimum improvement worth shipping, translated into a sample and a timeline with the calculator above |
| One variable at a time (when possible) | Icon and screenshots changed together tell you the bundle won, not which part did |
| Stable window | No overlapping platform feature, seasonal spike or unusual paid campaign inflating the page |
| Per localisation | Test configured for the market you intend to change, not one market generalised to all |
| Retention guardrail instrumented | Install cohort tagged by variant in your own analytics before the test starts |
| Post-launch verification | After shipping the winner, watch conversion for a full cycle to confirm the effect survives outside the test |
Combining store tests with in-app tests
The sequence that works is boring and effective: fix the biggest leak first. If your listing converts well and your onboarding leaks, a better icon just sends more people into a broken funnel. If your onboarding is solid and almost nobody reaches it, listing creative is the right place to spend the next month. Measure both, decide with the arithmetic, and run one test at a time in each zone rather than four overlapping tests whose effects you can no longer separate.
For the in-app half of that pairing, mobile onboarding A/B testing covers the activation metric and the survivorship bias that ruins most onboarding results.
Do this automatically with Donnu
Store listing tests can only be run by the platforms themselves, and this article does not pretend otherwise. What travels between the two zones is the discipline: size the test before running it, read the confidence interval rather than the headline lift, and always keep a guardrail on the metric downstream of the one you are optimising.
Donnu A/B applies exactly that discipline to the web side of your funnel: the landing page that feeds installs, the web view checkout, the logged-in area reachable from a browser. Lightweight snippet, automatic sample sizing, honest Bayesian statistics, no invented certainty. Start a 14-day free trial and stop shipping winners that only exist inside the noise.
Read also: Mobile App A/B Testing: The Complete Guide · Mobile Onboarding A/B Testing · Common A/B Testing Mistakes
References
- Google Play Console. Store listing experiments. Official page on Google Play’s native listing experiment, run against real store traffic. play.google.com/console/about/store-listing-experiments.
- Apple. Product Page Optimization. Official documentation on the App Store Connect feature that tests variants of icon, screenshots and app previews against the original page. developer.apple.com/app-store/product-page-optimization.
- Apple. Custom product pages. Official documentation on alternative product pages with their own URLs, used for campaign targeting rather than experimentation. developer.apple.com/app-store/custom-product-pages.
- Google Play Console Help. Run A/B tests on your store listing. Setup and reporting reference for Play Console experiments, including target metrics, localised experiments and the six month automatic stop. support.google.com/googleplay/android-developer/answer/12053285.
- Sensor Tower. App Store Optimization: visibility and conversions. Industry reference on ASO practice, including testing visual listing elements to raise conversion. sensortower.com/blog/app-store-optimization-visibility-conversions.
Frequently asked questions
- What exactly can I A/B test on an app store listing?
- On Google Play, Store listing experiments cover the visual assets (icon, feature graphic, screenshots and preview video) and the app descriptions for a given localisation. On the App Store, Product Page Optimization covers the visual assets only: app icon, screenshots and app previews. Apple does not offer an A/B test for the app name, subtitle or description through Product Page Optimization. Anything outside those lists (price, category, keywords in the metadata field) is not testable through the built-in store experiment.
- Is a store listing test the same as an A/B test inside my app?
- No. A store listing test runs before installation, on the store page itself, and it is executed and measured by the platform, not by your code. The metric is listing view to install conversion. An in-app A/B test runs after installation, through a remote feature flag or a backend decision, and measures product events like activation, upgrade or retention. They answer different questions and are usually owned by different people.
- How long should a store listing test run?
- Long enough to reach the sample your baseline conversion rate and target effect require, and at least one full week to cover the weekday and weekend cycle. Apple documents that a Product Page Optimization test can run for up to 90 days. In practice the binding constraint is store traffic: a listing with modest impression volume that is chasing a small relative improvement can need tens of thousands of views per variant, which the sample size calculator in this article makes concrete.
- Why did my store test win and my installs not go up afterwards?
- Three common reasons. First, the effect was real but small, and the shipped version is now competing against seasonality that moved more than the test did. Second, the test ran on a traffic mix (search, browse, external referral) that is not the mix you get after launch, so the winning asset performs differently on a different audience. Third, and most damaging, the winning asset oversells: it raises installs and lowers retention. Always read a store test alongside a downstream retention guardrail rather than on install conversion alone.
- Can I use my own A/B testing tool on the store listing?
- No. The store page is rendered by Google Play and the App Store, not by your servers, so no third-party snippet or SDK can split that traffic. The only way to test a listing is through the store native experiment (Store listing experiments or Product Page Optimization). External tools can help you produce and pre-screen creative concepts, but the live test itself only runs inside the platform.