Feature Flags vs A/B Testing: The Real Difference

Feature flags and A/B tests answer different questions, even though they get lumped together constantly. A feature flag is a delivery mechanism: a switch in the code that decides, without a new deploy, who sees which version of a feature. An A/B test is a measurement method: it splits an audience into comparable groups and uses statistics to say whether a change genuinely improves a metric, or the observed difference is just chance. Confusing the two is a common, expensive mistake: teams flip a flag to 50% of traffic, eyeball the analytics dashboard, decide the number “feels” higher, and ship to everyone without ever running an actual test.

This guide keeps the two concepts separate: what each one does on its own, when to reach for only a flag, when to reach for only a test, and how to combine them, the pattern that LaunchDarkly, PostHog, Statsig, and Optimizely all describe in their own product documentation, without losing statistical rigor along the way.

Feature flags and A/B tests, in one sentence each

A feature flag (or feature toggle) is a conditional in the code that turns a behavior on or off at runtime, without requiring a new deploy. A widely cited reference article on the topic, published on Martin Fowler’s site, describes feature toggles as “a powerful technique, allowing teams to modify system behavior without changing code.” A flag answers: who sees what, right now?

An A/B test is a controlled experiment: it randomly splits traffic between a control version and one or more variations, measures a primary metric on each side, and applies a statistical test, a p-value, a confidence interval, or a bayesian probability, to decide whether the difference is real. A test answers: does this change actually work, or was it a coincidence?

	Feature flag	A/B test
Question it answers	Who sees what, right now?	Does this change actually work?
Nature	Delivery mechanism (deploy / release)	Measurement method (statistics)
Decided by	Business rule, risk, permission	Statistical significance or bayesian probability
Needs a calculated sample?	No	Yes
Can exist without the other?	Yes (rollout, kill switch, permissioning)	Yes, though usually delivered through some form of traffic split

The core difference: controlling exposure vs measuring effect

The most direct way to hold onto the distinction: a flag controls exposure, an A/B test measures effect. A flag can expose a feature to 5%, 50%, or 100% of users without that, on its own, saying anything about whether the feature is good. An A/B test can measure the effect of a change even with no flag infrastructure involved at all, the classic marketing A/B test swaps elements directly on a page with no flag layer underneath. These are two different axes, and a project can need one, the other, or both at once.

Deciding which one a change actually needs, before writing any code, is worth doing as a deliberate first step rather than an afterthought. The flowchart below walks through the two questions that matter: does the change carry real technical risk, and is its effect on a metric genuinely uncertain.

When to use only a feature flag

Use a feature flag with no A/B test behind it when the question is about delivery risk, not about effect on a metric. Optimizely’s own glossary entry on feature flags names the same three recurring cases:

• Progressive rollout (canary release). Release the feature to 1%, then 10%, then 50% of users, watching errors and performance at each step before advancing. The question here is “does this break anything?”, not “does this convert more?”
• Kill switch. A switch that turns a feature off instantly if something goes wrong in production, with no deploy rollback needed. Optimizely describes this as a final safety layer: if a feature misbehaves, you flip the switch and fall back to the stable version immediately, without waiting on a new build.
• Permissioning. Controlling who has access to a feature by plan, region, or segment, an enterprise-only feature, a closed beta for a specific group of customers. This kind of flag tends to live a long time, unlike rollout flags, which should be retired as soon as the feature stabilizes.

None of these three cases needs a calculated sample size, significance, or a control group. The decision is binary and operational: “is this safe?” or “can this person see it?”, not “does this change move the metric?”

When to use only an A/B test

In the opposite direction, use an A/B test without leaning on feature flag infrastructure when the question is about effect on a metric that matters, and the technical risk of the change itself is low. The classic case is marketing and CRO testing: a headline, a button label, a form layout, a displayed price. None of these changes threaten system stability; the only risk is conversion, and that is exactly what the test measures.

The signal that you need real statistical rigor, not just a flag, is this: the decision is expensive if you get it wrong (pricing, checkout, onboarding), the expected difference is small enough to be confused with noise, or you will need to defend the decision with numbers later. In those cases, jumping straight to “we set it to 50% and the chart looks better” is the exact mistake PostHog calls out in its own documentation: without a real experiment, you have not controlled for statistical significance, and the difference that looked real might just be noise.

If you are setting up this kind of test from scratch, it is worth sizing the sample before you launch it: see the step-by-step guide to running an A/B test for the full process and the sample size calculator.

When you need both

The more mature pattern, and the one LaunchDarkly, PostHog, and Statsig all describe in their own product documentation, is to use the flag as the delivery mechanism and place the A/B test as an experimentation layer on top of it. The flag controls who gets which variant; the A/B test, tied to that same split, measures whether the variant wins. In product practice, this usually follows the same stages as a progressive rollout, with a formal measurement step added in the middle before advancing to 100%:

This is exactly the model Statsig documents for combining feature gates and experiments: gates make good pre-filters for targeting the right audience, and experiments then quantify the lift across metrics for that audience, with allocation capped at 50/50 so the comparison stays fair. LaunchDarkly describes the same pattern in its own experimentation docs: you connect a flag to the metrics that matter for it, and LaunchDarkly surfaces confidence intervals showing how much evidence supports each variation, which is what then justifies widening the rollout to the rest of the base.

Worked example: e-commerce and SaaS

E-commerce. A store wants to replace its three-step checkout with a single-page version. The technical risk is real, a broken checkout is lost revenue instantly, so the change ships behind a feature flag, released first to 5% of traffic just to confirm nothing breaks. Once that is confirmed stable, the same flag starts feeding a real A/B test: 50% of visitors see the new checkout, 50% see the old one, with stable per-visitor assignment and purchase completion rate as the primary metric. Only once the test declares significance does the flag advance to 100%.

SaaS. A product team wants to test a new onboarding flow that asks for less information at signup. The technical risk is low, it does not touch billing or sensitive data, but the effect on the metric that matters (7-day activation) is genuinely uncertain and could go either way. Here the flag exists mostly for deployment convenience, engineering can merge incomplete code into the main branch without affecting production, rather than out of fear of breaking something; it is the A/B test, sized in advance from the weekly volume of new signups, that decides whether the new onboarding sticks.

Reading the result with rigor, even when it comes from a flag

Once a flag is delivering both versions and the test is running at a stable stage (the 50% mark in the checkout example above), the statistical read is the same as for any A/B test: compare visitors and conversions on each side. Reusing a worked example already validated in this blog’s statistics engine, with the control recording 210 conversions out of 4,200 visitors and the variation recording 273 out of 4,200:

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Running those numbers through the calculator above, the control’s rate is 5.0%, the variation’s is 6.5%, the two-sided p-value lands near 0.003, and the confidence interval on the difference does not cross zero. In other words: even though this data came from a flag sitting at 50% rollout, the difference is statistically significant, not noise, and only at that point does it make sense to advance the flag to 100%. For the full walkthrough of that math, see how to declare statistical significance without fooling yourself.

One detail worth reinforcing: data collected while the rollout percentage was still changing (from 1% to 10%, from 10% to 50%) should not enter that calculation. Every stage change can alter who is being exposed, teams running progressive rollouts often vary the targeting criteria at each phase, and mixing different cohorts into the same statistical analysis is a quiet way to contaminate the result. Treat each stable plateau as its own collection window.

Feature flags vs A/B testing, side by side

Criterion	Feature flag	A/B test
Goal	Control release risk	Measure effect on a metric
Decision unit	Business rule (plan, region, % rollout)	Statistics (p-value, CI, or bayesian probability)
Typical lifespan	Short for rollout (days to weeks); long for permissioning	Fixed duration, calculated from sample size and traffic
Who decides	Engineering / product, by risk criteria	The statistical result, defined before the test starts
Reference tools	LaunchDarkly, Split/Harness, Flagsmith, Unleash, Statsig	Donnu A/B, VWO, Optimizely, GrowthBook
Common mistake if used alone	Assuming “live in production” means “proven to work”	Running a high-risk technical change with no safety net behind it

Common mistakes when mixing the two

• Declaring a winner by watching the rollout, without a test. The flag reaches 50%, the number on the dashboard “looks” better, and the team advances to 100% without ever calculating significance. This is precisely the pattern PostHog calls out as the costliest mistake: the observed difference can be entirely noise.
• Mixing data from different rollout stages into the same analysis. Combining the week the rollout sat at 10% with the week it sat at 50% distorts the comparison, because the exposed audience changes at every stage boundary.
• Leaving the flag alive after the test is done. Rollout and experiment flags are transient by nature. Martin Fowler’s reference article flags this exact risk: forgotten toggles accumulate technical debt and make the codebase progressively harder to reason about. Once the test decides, win, loss, or discard, the flag should be removed, not left “just in case.”
• Running a high-risk technical change as an A/B test with no flag behind it. Testing a large checkout rewrite directly, with no kill switch available, removes your ability to revert instantly if something breaks in production.

Automate this on Donnu

Combining controlled rollout with an honest statistical read is manual work when each piece lives in a different tool, and the most common failure mode is exactly the one this guide walks through: a team ships past 50% because the dashboard “looked” better, never having calculated whether that difference was real. Donnu A/B owns the measurement half of that combination: you define the hypothesis and the traffic split, the lightweight snippet never blocks your product’s page, and the bayesian statistics engine declares a winner with the same rigor described in this guide, not a dashboard glance at a flag sitting at 50%.

Start a free 14-day trial and measure your next change with real statistics, not just a flag at 50%. For the statistical foundation behind every test, read the complete guide to A/B testing; to decide where the traffic split should technically happen, see client-side vs server-side A/B testing.

---

Also read: Feature flags: the complete guide · What is A/B testing? The complete guide · Client-side vs server-side A/B testing

References

• Fowler, M. (with Hodgson, P.). Feature Toggles (aka Feature Flags). martinfowler.com/articles/feature-toggles.html.
• Statsig. When to Use Feature Gates vs. Experiments. docs.statsig.com/guides/featureflags-or-experiments.
• Optimizely. Feature Flags. optimizely.com/optimization-glossary/feature-flags.
• PostHog. What is a Feature Flag? Feature Flags vs Remote Config vs A/B Testing. posthog.com/blog/what-is-a-feature-flag.
• LaunchDarkly. Experimentation. launchdarkly.com/docs/home/experimentation.