CRO

Money-Back Guarantee A/B Testing: Read Refunds, Not Orders

A money-back guarantee can lift orders and refunds at once. Judge the test on kept orders per visitor, read after the refund window closes.

Flat illustration of a shopping bag resting on a calendar, with a curved arrow returning a coin to an open hand, in deep green and teal on a soft mint green background

A money-back guarantee is a treatment that moves two numbers: it raises orders, because buying feels less risky, and it can raise refunds, because more people use the way out. Refunds arrive after the order, inside the guarantee window, so on the day the test “ends” most of them have not happened yet. The decision metric is kept orders per randomized visitor (orders minus refunded orders, over visitors), read only after the window has closed for the last order in the sample. This guide is part of the conversion rate optimization guide and applies the method from landing page optimization to one decision that sellers of courses, software and physical products face: should the guarantee be longer or looser? It walks through one illustrative test read three ways, shows why comparing a 7-day and a 30-day guarantee on day 28 favors the longer one by construction, and covers what the FTC, the EU and Brazil say about guarantees.

A guarantee test is not hard to launch. It is hard to read, because the cost of the treatment shows up weeks after the benefit.

What a money-back guarantee changes, and what this guide is not about

A guarantee is a promise about what happens after the purchase: if the buyer is not satisfied, they get their money back. It works on the moment of decision by moving risk from the buyer to the seller. That is why it can lift orders. It is also why it can lift refunds: the buyers it convinces are, almost by definition, the ones who were unsure, and some of them will take the offer.

Three neighboring topics on this blog look similar and are not the same test:

The academic picture supports the idea that the effect on refunds is not known in advance. Janakiraman, Syrdal and Freling’s meta-analysis of 21 papers on return policy leniency, published in the Journal of Retailing in 2016, concludes that “overall, leniency increases purchase more than return”. The same abstract adds that “money and effort leniency increase purchase”, that “scope leniency increases returns” and that “time and exchange leniency reduces returns”. In other words, the literature does not promise that a longer window raises refunds; on average it found the opposite for time. Your product, price and audience decide what happens on your page, and only a test read at the right moment can tell you.

Why a money-back guarantee test needs two clocks

Every guarantee test runs on two clocks. The first is the entry clock: how long you send traffic to the test and collect orders. The second is the guarantee clock: how long each order can still turn into a refund. The test is only finished when the second clock has run out for the last order the first clock collected.

That has a consequence most teams miss. If you stop reading when traffic stops, the refunds you have seen depend on the length of the guarantee. An arm with a 7-day guarantee has had time to show nearly all of its refunds. An arm with a 30-day guarantee has shown only the refunds from its oldest orders. The comparison is not neutral: it is tilted toward the longer guarantee, and the tilt is bigger the longer the guarantee is.

Timeline of a guarantee test: entry period, guarantee window and the two reading daysA horizontal timeline from day 0 to day 60. The entry period, when orders are collected, runs from day 0 to day 28. For the control with a 7-day guarantee, the window for the last order closes on day 35. For the variation with a 30-day guarantee, it closes on day 58, followed by a short processing margin to about day 60. A marker on day 28 is labeled naive reading, when traffic stops. A marker near day 60 is labeled honest reading, after every window has closed.The test clock is entry plus the longest guarantee windowday 0102028405058Entry: orders collected, days 0 to 287 daysA: last window closes on day 35B: 30-day window for the day 28 ordernaive readinghonest reading, about day 60Grey block: processing margin for refunds requested on the last day. Illustrative schedule.
Traffic stops on day 28, but the 30-day arm can still produce refunds until about day 60. Any reading before that compares a nearly complete arm with an incomplete one.

The rule is simple to state: the reading day is the last entry day plus the longest guarantee in the test plus a processing margin. With 28 days of traffic and a 30-day guarantee, that is about day 58 to 60. Plan the calendar, the report date and the stakeholders’ expectations around that day, not around the day traffic stops.

The illustrative test: 7 days against 30 days, unconditional

Everything below uses one illustrative scenario, not Donnu data and not customer data. A seller of an online course, or an online store with a single price product, sends about 15,000 visitors a week to its sales page. The test splits traffic 50/50 for 4 weeks (28 days), so each arm receives 30,000 visitors.

Everything else on the page is the same. Each visitor is randomized once and keeps the same arm, and every order and every refund is tied to the visitor’s arm through the order ID.

How big a test this page can afford

Size the test on the metric that will decide it: kept orders per visitor, with a baseline of 2.82 percent in this scenario. At 95 percent confidence, 80 percent power and a two-sided test:

relative MDE visitors per variation entry days at 15,000 a week full test clock with a 30-day guarantee
10% 56,718 53 53 days of entry plus 30 days of window
15% 25,789 25 25 days of entry plus 30 days of window

Reproduce the first row. With current conversion rate 2.82, minimum detectable effect 10 relative, confidence 95, power 80 and 15,000 visitors per week on a two-sided test, the calculator shows 56,718 visitors per variation and about 53 days. Change the effect to 15 and it shows 25,789 per variation and about 25 days.

Sample size calculator
-Visitors per variation
-Total (2 variations)
-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

Run it backward. With 30,000 visitors per variation and a 2.82 percent baseline, the smallest lift this test can reliably detect is about 13.4 percent relative. The MDE calculator does that backward calculation for any traffic level, and minimum detectable effect explains how to choose one. Keep that 13.4 percent in mind: it is the reason the honest reading below comes back inconclusive.

Remember that the calculator counts entry days only. The real calendar for the 15 percent row is 25 days of traffic plus 30 days of guarantee, and for the test we actually ran it is 28 plus 30.

Reading 1: orders say “winner”

The first reading counts orders and nothing else, on day 28.

arm visitors orders order rate
A, 7-day guarantee 30,000 900 3.00%
B, 30-day unconditional 30,000 1,035 3.45%

The relative lift is +15.0 percent, or +0.45 percentage points, with a z of 3.1197 from the stats engine, a p-value of 0.0018 and a 95 percent confidence interval of the difference from +0.17 to +0.73 points. Significant. This is the screenshot that gets shared in the team channel, and it says nothing about refunds.

Reading 2: refunds on day 28 say “even better”

A careful team subtracts refunds before declaring the result. On day 28, the refunds already recorded are 50 in A and 40 in B.

arm orders refunds seen by day 28 kept orders kept orders per visitor
A, 7-day guarantee 900 50 850 2.83%
B, 30-day unconditional 1,035 40 995 3.32%

The relative lift grows to +17.1 percent, with a p-value of 0.0006 and an interval from +0.21 to +0.76 points. B now looks better than it did on orders alone: fewer refunds and more orders. This is the trap. B has fewer refunds on day 28 not because its buyers are happier, but because most of them still have weeks left to ask for their money back. The reading looks more rigorous than Reading 1 and it is more biased.

Cumulative refunds by day for the 7-day and the 30-day guaranteeTwo cumulative curves on a chart of refunds from 0 to 120 against days since launch from 0 to 60. The 7-day arm A rises steadily to 50 refunds by day 28 and flattens at 54 around day 35. The 30-day arm B rises more slowly, reaching 40 refunds by day 28, then keeps climbing to 114 by day 58. A vertical dashed line at day 28 shows that A has shown about 93 percent of its final refunds at that point and B only about 35 percent.Cumulative refunds: what day 28 can and cannot see050100day 02858day 28, traffic stopsA: 54 final, about 93% seen by day 28B, 30 days: 114 finalabout 35% seen on day 28B: 40 on day 28A: 50 on day 28
On day 28 the 7-day arm has already shown 50 of its 54 refunds. The 30-day arm has shown 40 of 114. The curves cross after traffic stops, which is exactly the part a day 28 report never sees.

Reading 3: kept orders after every window closes say “inconclusive”

The honest reading waits until the 30-day guarantee has expired for the last order collected on day 28, plus processing, around day 58 to 60. Final refunds: 54 in A (6.0 percent of its orders) and 114 in B (11.0 percent of its orders).

arm orders final refunds kept orders kept orders per visitor
A, 7-day guarantee 900 54 846 2.82%
B, 30-day unconditional 1,035 114 921 3.07%

The relative lift is +8.9 percent, +0.25 percentage points, with a z of 1.8111 from the engine, a p-value of 0.0701 and an interval from minus 0.02 to +0.52 points. Not significant at 95 percent. The direction is positive, but the test does not support declaring B the winner. The true effect may well be around 9 percent, and a 30,000 per arm test can only reliably detect about 13.4 percent: the test was underpowered for the effect that actually remained after refunds.

Here are the three readings side by side.

reading when metric A B relative lift p-value 95% CI, pp verdict
1 day 28 orders per visitor 3.00% 3.45% +15.0% 0.0018 +0.17 to +0.73 significant, but blind to refunds
2 day 28 kept orders, truncated 2.83% 3.32% +17.1% 0.0006 +0.21 to +0.76 significant, and biased toward B
3 about day 60 kept orders, final 2.82% 3.07% +8.9% 0.0701 minus 0.02 to +0.52 inconclusive
Three readings of a guarantee test, 95 percent intervals of the differenceThree horizontal confidence intervals on a percentage point scale from minus 0.2 to plus 0.8, with a vertical line at zero. Reading 1, orders on day 28, runs from plus 0.17 to plus 0.73 with a point estimate of plus 0.45. Reading 2, kept orders on day 28, runs from plus 0.21 to plus 0.76 with a point estimate of plus 0.48. Reading 3, kept orders after every window closed, runs from minus 0.02 to plus 0.52 with a point estimate of plus 0.25 and crosses zero.95% CI of the difference, 30 days minus 7 days, in percentage points-0.20+0.2+0.4+0.6+0.8Reading 1, ordersday 28, no refunds+0.17 to +0.73, p = 0.0018Reading 2, kept ordersday 28, truncated+0.21 to +0.76, p = 0.0006Reading 3, kept ordersafter windows close-0.02 to +0.52, p = 0.0701Dots are point estimates: +0.45, +0.48 and +0.25 pp. Illustrative data, two-sided z-test.
The two day 28 readings sit well above zero. The final reading slides left and touches zero. Same test, same visitors: the only thing that changed is how long we waited.

Check the numbers in the calculator

Statistical significance calculator
Control (A)
Variation (B)
Control (A) · Rate-
Variation (B) · Rate-
Relative lift-
p-value-
95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Enter 30000 and 846 for Control (A) and 30000 and 921 for Variation (B), the final kept orders from Reading 3. The calculator shows rates of 2.82% and 3.07%, a relative change of +8.9%, a p-value of 0.0701 and a 95% CI of the difference of -0.0% … +0.5% (pp). Now enter the orders from Reading 1, 900 and 1035 on the same visitor counts: 3.00% and 3.45%, +15.0%, 0.0018 and +0.2% … +0.7% (pp). Finally the truncated Reading 2, 850 against 995: 2.83% and 3.32%, +17.1%, 0.0006 and +0.2% … +0.8% (pp). The screen rounds the interval to one decimal and does not display z; the z values and two-decimal intervals quoted above come from the same engine. More on reading these outputs in statistical significance for A/B tests.

Why the refund rate is not the decision metric

Once the window closes, a second temptation appears: compare the refund rate. In this test it went from 6.00 percent of orders in A to 11.01 percent in B (54 of 900 against 114 of 1,035). Put those numbers into the calculator with orders as “visitors” and the p-value comes out below 0.0001. A significant, alarming jump. A team that decides on the refund rate kills B.

That decision uses the wrong denominator. The refund rate is refunds divided by orders, and orders are exactly what the guarantee changed. B brought in 135 extra orders, and those marginal buyers are the hesitant ones, the people who needed the guarantee to buy. It is natural that more of them use it. A higher refund rate is compatible with more kept orders per visitor, and in this test that is what the point estimate shows: refunds per order almost doubled while kept orders per visitor still pointed up.

The cleaner rule follows from how the test was randomized:

The same argument is why the primary metric of any test should be defined before launch; the overall evaluation criterion walks through the general case.

Defining kept orders per visitor

A metric that can judge a guarantee test has four parts, and each one closes a specific leak.

part the shortcut (misleading) the definition to use
unit orders, or sessions of the sales page the visitor randomized to an arm
numerator orders orders that were not refunded, including chargebacks as refunds
attribution refunds by the day they happened each refund credited to the arm of its original order, through the order ID
reading day the day traffic stops last entry day plus the longest guarantee plus processing

Tie every refund to the original order. Refunds live in your payment platform or store backend, not in the testing tool. Keep the order ID next to the arm the buyer saw, so a refund on day 41 lands on the visitor who bought on day 20, in the right arm.

Count chargebacks as refunds. A buyer who disputes the charge with the bank instead of asking you is still an order you did not keep.

Close the window for everyone, then add margin. Requests made on the last day of the guarantee may be processed a few days later, and the money takes longer still to reach the buyer: Stripe tells merchants that the customer “sees the refund as a credit approximately 5-10 business days later, depending upon the bank”. What you count is the day the refund is issued, not the day it lands, but the request still needs time to be handled. Wait for the longest guarantee to expire for the last order, then give processing a few days before you pull the numbers.

Revenue moves with it when the price is single. With one price, kept revenue per visitor has the same relative lift as kept orders per visitor: +8.9 percent here. With several prices or partial refunds, compute kept revenue directly; sample size for revenue and continuous metrics explains why revenue needs a bigger sample than a rate.

Some costs do not come back with the refund. Stripe’s documentation states that “Stripe’s processing fees from the original transaction aren’t returned” when you refund. Add shipping, return handling, support time and, for a course or software, the access the buyer already consumed. None of these show up in kept orders per visitor, so a variant that only ties on kept orders is a loss in money. Weigh them in the decision rule, not after the fact.

Before trusting any of it, confirm the split is close to 50/50 with a sample ratio mismatch check, and resist opening the report every day while the windows are still open: the peeking problem applies with extra force when the late data is the costly data. The general version of this waiting rule, for any metric that matures over time, is in cohort maturity in A/B tests.

What to do with an inconclusive guarantee test

An inconclusive result like Reading 3 is a likely outcome at realistic traffic, because the effect left after refunds is smaller than the effect on orders, so it deserves a plan before it happens.

  1. Do not reopen the day 28 numbers to rescue a winner. They are biased toward B by construction; using them now is choosing the answer and then the evidence.
  2. Do not simply add days to the same test. Extending until the p-value crosses 0.05 is peeking with extra steps. If you want more power, plan a new test sized for it.
  3. Size the follow-up honestly. To detect a 10 percent relative change in kept orders, this page needs 56,718 visitors per variation: about 53 days of traffic plus 30 days of guarantee, close to three months of calendar before a decision.
  4. Let the costs break the tie. With a positive but inconclusive result, the fees, shipping and support that do not come back with a refund argue for keeping the control or testing a cheaper variant, such as a 14-day or a conditional guarantee.
  5. Write down what you learned. Refunds per order almost doubled. That is a real fact about your buyers even though the decision metric was inconclusive, and it shapes the next hypothesis.

What else to test besides the guarantee length

“7 days or 30 days” is only one of the levers. Each one acts on orders and on refunds differently, and each needs the same decision metric.

what you change example likely effect on orders likely effect on refunds reading day
Length 7, 14 or 30 days may rise with length, not necessarily in proportion can go either way; the meta-analysis found time leniency reduces returns on average after the longest window closes
Unconditional or conditional “no questions asked” against “if you completed the first module” unconditional removes more doubt conditions filter refunds but must be disclosed clearly after the window, plus time to verify conditions
Placement next to the price, near the buy button, only in the FAQ strongest near the decision point small direct effect; it changes who notices the promise after the window closes
Wording “satisfaction guaranteed” against “30 days to get your money back” concrete terms are easier to believe clear terms reduce disputes and chargebacks after the window closes
Checkout reminder a short line under the pay button can recover buyers who hesitate at payment little direct effect after the window closes
Extended or result-based “double your money back”, “results or refund” can lift orders strongly can lift refunds strongly and invites abuse after the window, and check the legal side first

A guarantee is a promise in advertising and, in many places, it sits on top of rights the buyer already has by law. What follows is context for test design, not legal advice; check with a lawyer for your market.

market source what it says what it means for the test
United States FTC, Guides for the Advertising of Warranties and Guarantees, 16 CFR 239.3 use “Satisfaction Guarantee” or “Money Back Guarantee” only if you refund the full purchase price on request, and disclose material conditions clearly a conditional variant must show its conditions where buyers will notice them
European Union Directive 2011/83/EU, articles 9, 13 and 16 14 days to withdraw from a distance contract without giving a reason; reimbursement within 14 days of being informed; exceptions such as digital content once supply has begun with consent a “7-day guarantee” sits below the statutory withdrawal right, so the real control is 14 days
Brazil Consumer Defense Code, article 49 7 days to withdraw from purchases made outside the commercial establishment, with amounts paid returned immediately a 7-day guarantee largely restates the law

United States. The FTC’s guides are direct. Section 239.3(a) says a seller “should use the terms ‘Satisfaction Guarantee,’ ‘Money Back Guarantee,’ ‘Free Trial Offer,’ or similar representations in advertising only if” it “refunds the full purchase price of the advertised product at the purchaser’s request.” Section 239.3(b) adds that the ad “should disclose, with such clarity and prominence as will be noticed and understood by prospective purchasers, any material limitations or conditions.” The guides give examples such as returning “the unused portion within 30 days for a full refund”. For a test, that means a conditional variant is not just wording: the conditions have to be visible, and the visibility itself may change the result.

European Union. Article 9 of the Consumer Rights Directive gives the consumer “a period of 14 days to withdraw from a distance or off-premises contract, without giving any reason”. Article 13 requires the trader to “reimburse all payments received from the consumer, including, if applicable, the costs of delivery” no later than 14 days after being informed. Article 16(m) excludes “digital content which is not supplied on a tangible medium if the performance has begun with the consumer’s prior express consent and his acknowledgment that he thereby loses his right of withdrawal”. If you sell physical products into the EU, the shortest guarantee you can honestly test is the legal 14 days, and the question becomes 14 against 30. For courses and downloads the rules depend on how access starts, which is a question for counsel.

Brazil. Article 49 of the Consumer Defense Code lets the buyer withdraw “no prazo de 7 dias” (within 7 days) from a purchase made outside the store, and the amounts paid “serão devolvidos, de imediato” (will be returned immediately). A Brazilian sales page that advertises a 7-day guarantee is mostly describing the law, which is fine as long as it does not present the minimum as a special favor.

Guardrails for a guarantee test

Kept orders per visitor is the decision. These guardrail metrics catch damage it cannot see; declare each one with a threshold before launch.

guardrail why it moves with a longer or looser guarantee how to read it
Refund rate per order more hesitant buyers enter, and more of them use the way out diagnostic, never the decision; explain it, do not optimize it
Chargebacks and disputes unclear terms push buyers to the bank instead of to you a rise costs fees and account standing, beyond the refund
Support contacts about refunds every refund request is a conversation a rise is a cost kept orders do not count
Time to refund slow handling turns refunds into disputes keep it stable between arms, or the comparison is not fair
Consumption before refund for courses and software, how much was used before asking a rise in “use then refund” is a sign of abuse

If the instinct after a guarantee test is to fight refunds with a retention offer, that is a separate experiment with its own traps; see cancel flow A/B testing.

A test plan you can copy

  1. Hypothesis written: which lever (length, conditions, placement, wording) and which direction you expect for orders and for refunds.
  2. Legal floor checked for every market the page sells to, and conditions visible on the page if the variant has any.
  3. Primary metric declared: kept orders per randomized visitor, chargebacks counted as refunds, each refund credited to the arm of its original order.
  4. Order ID linked to the arm so refunds from the payment platform can be matched later.
  5. Sample sized on kept orders with the sample size calculator, and the calendar set to entry days plus the longest guarantee plus processing.
  6. Guardrails declared with thresholds, including disputes and support contacts.
  7. Decision rule written: ship only if kept orders per visitor wins at the planned reading day and no guardrail crosses its threshold; on an inconclusive result, keep the control unless the costs say otherwise.
  8. No reading before the reading day, except to check that tracking and the split are healthy.

Make this automatic with Donnu

The trap in a guarantee test is the calendar. The report that looks finished on the day traffic stops has seen the benefit of the longer guarantee and only a fraction of its cost, so it leans toward the longer guarantee by construction.

Donnu runs A/B tests and Split URL tests on web pages. You can change the guarantee block, its wording or its placement in the visual editor directly on the live sales page, or send each arm to its own page with Split URL. Each visitor is randomized once and keeps the same variation for the whole test, and the conversion goal can be the order confirmation page, so orders are counted per randomized visitor from the first day. Reports are Bayesian, and the product is built PT-BR and LGPD first. Refunds live in your payment platform, so the part that stays with you is the discipline this guide describes: keep the order ID next to the variation, subtract refunds by the original order, and open the decision only when the longest guarantee has closed for the last order.

The sample size calculator and the significance calculator reproduce every number above, and the revenue per visitor calculator helps when your prices are not single. When you are ready, create your Donnu account and run your guarantee test on the metric that counts what you keep.

References

Read next: Landing page optimization · Conversion rate optimization guide · Trust badges A/B testing · Conversion lag in A/B tests · Guardrail metrics · Overall evaluation criterion · How to A/B test pricing · Leia em português

Frequently asked questions

Does a longer money-back guarantee increase sales?
It often increases orders, because it lowers the risk the buyer feels at the moment of purchase, but orders are only half of the answer. A longer guarantee can also increase refunds, and those refunds arrive after the order, inside the guarantee window. In the illustrative test in this guide, moving from a 7-day guarantee to a 30-day unconditional guarantee raises orders by 15.0 percent with a p-value of 0.0018, while kept orders per visitor, read after every window closed, rise only 8.9 percent with a p-value of 0.0701: inconclusive.
What metric should decide a money-back guarantee A/B test?
Kept orders per randomized visitor: orders minus the orders that were refunded, divided by the visitors assigned to each arm. Count each refund against the variation of the original order, and read the metric only after the guarantee window has closed for the last order in the sample, plus a few days for processing. With a single price, kept revenue per visitor moves by the same relative amount.
Why is the refund rate not the decision metric?
Because its denominator, orders, is changed by the treatment. A guarantee that brings in extra, more hesitant buyers can raise the refund rate per order and still raise kept orders per visitor. In the illustrative test the refund rate goes from 6.00 percent to 11.01 percent of orders, a significant jump, while kept orders per visitor point upward. Track the refund rate as a diagnostic, but decide on kept orders per visitor.
When can I read the results of a guarantee test?
Only after the longest guarantee in the test has expired for the last order collected, plus processing time. With 28 days of traffic and a 30-day guarantee, that is around day 58 to 60, not day 28. A reading on day 28 has seen almost all refunds from the 7-day arm and only a fraction of the 30-day arm, so it favors the longer guarantee by construction.
Is a 7-day money-back guarantee just the legal minimum?
It depends on where you sell. In Brazil, article 49 of the Consumer Defense Code gives 7 days to withdraw from purchases made outside a physical store, so a 7-day guarantee there largely repeats the law. In the European Union, the Consumer Rights Directive gives 14 days for distance contracts, with exceptions such as digital content once delivery started with the buyer consent. In the United States, the FTC guides say to use the words money back guarantee only if you refund the full purchase price on request. This is not legal advice.
What should I test about a money-back guarantee besides its length?
Unconditional against conditional terms, where the guarantee appears on the page, how it is worded, whether it is shown at checkout, and extended or result-based guarantees. Each one can move orders and refunds in different directions, so every variant needs the same metric: kept orders per visitor, read after the longest window closes.