CRO

CTA A/B Testing: What to Test Beyond Button Color

CTA A/B testing that moves signups, not just clicks: test the offer, the risk and the next step, size it to your traffic and skip the button color.

Flat illustration of a large pill-shaped teal button on a deep green panel, a hand cursor pressing it and a narrow funnel below dropping small dots, on a soft mint green background

CTA A/B testing pays off when you test what the button promises and what happens after it is clicked, not how it looks. The offer in the label, the risk the visitor perceives right below it and the step that follows the click move signups; color almost never does at medium traffic. And the single most common way to fool yourself is to decide the test on CTA clicks instead of signups or purchases. This guide is part of our landing page optimization method, which in turn sits inside the conversion rate optimization guide. It walks through the parts of a call to action worth testing, why button color rarely earns a slot, how much traffic each metric needs, and two illustrative tests on the same page: one that “wins” on clicks and changes nothing, and one that wins where it counts.

Button color is the most famous CTA test and the least useful one for most teams. It is easy to build, so it gets built. The cost is not the build. The cost is the four or eight weeks of traffic it consumes while a test that could have moved revenue waits in the queue.

What CTA A/B testing actually measures

A call to action (CTA) is the control that asks the visitor to take the page’s main step: start a trial, create an account, add to cart, book a demo. CTA A/B testing compares two or more versions of that control, and of what surrounds it, by splitting real traffic at random and measuring which version produces more of the outcome you care about.

That last phrase hides the whole problem. There are two outcomes on the table:

A CTA test that lifts the micro conversion and leaves the macro flat has not improved anything. It has only moved people one screen further before they leave. The overall evaluation criterion of a CTA test is therefore the macro conversion, declared before launch. Clicks stay on the report as a diagnostic: they explain why the macro moved, or why it did not.

For the rest of this guide we use one illustrative page, not customer data: a SaaS trial signup page with 10,000 visitors a week, a CTA click rate of 12% and a signup rate of 3% of visits. A four-week test gives 40,000 visits, 20,000 per variant.

The anatomy of a call to action: what is worth testing

A CTA is not just a button. It is a small system: a label that makes a promise, a line of microcopy that answers the visitor’s main worry, a position on the page, a visual treatment and, out of sight, the screen the click leads to. Each part changes a different thing in the visitor’s decision.

Anatomy of a call to action and what to test in each partA browser window with a headline, a large green button labeled Start my 14-day free trial, a microcopy line below it saying no credit card needed and cancel anytime, a dashed fold line, a row of proof logos, a second smaller button repeated after the proof and a card showing the next screen, which asks only for email and password. Five labels on the right name what to test: the label as offer and promise, the microcopy as perceived risk, placement above the fold, repetition after the proof, and whether the next step keeps the promise.Start my 14-day free trialNo credit card needed. Cancel anytime.Start my free trialNext screen: email and password, no card field1 Label: the offer and the promisecontrast: text 4.5:1, boundary 3:1 (WCAG)2 Microcopy: the perceived risk3 Placement: visible above the fold4 Repetition: again after the proof5 After the click: keep the promisethe part most CTA tests forget
Five parts, five different hypotheses. Only the first two live inside the button area; the fifth is not on the page at all, and it often decides whether a click becomes a signup.
CTA part what it changes in the decision example change metric that decides
Label (offer and promise) what the visitor believes they get “Get started” to “Start my 14-day free trial” signups per visit
Microcopy below the button the perceived cost or risk of clicking add “No credit card needed. Cancel anytime.” signups per visit
Placement and repetition whether the CTA is seen at the moment of decision repeat the CTA after the proof section signups per visit
Contrast, size, target area whether the control is perceivable and usable meet WCAG contrast and target size fix, do not test
What happens after the click whether the promise is kept on the next screen remove the card field from step 2 signups per visit, then activation
Color of the button almost nothing, once contrast is met green to orange rarely worth the traffic

The offer in the label: specific beats generic

The label is a promise. “Get started”, “Submit” and “Learn more” promise nothing in particular, which is exactly why they feel safe to write and why they underperform as decision aids. A specific label names the outcome, the price and the commitment: “Start my 14-day free trial”, “Get the pricing PDF”, “Book a 20-minute demo”.

Nielsen Norman Group studied this exact label. In “Get Started Stops Users” (Harley and Flaherty, 2017), they report that a generic Get Started call to action “attracts clicks, but also misleads users”, because visitors who wanted information clicked it and landed in surveys, quizzes and signup flows they were not ready for. Their conclusion is the thesis of this guide in one line: using a generic label “may indeed get more clicks on the CTA, but a link is a promise”, and if the promise is wrong you reach “superficial conversion goals” without keeping customers. They recommend stating “precisely what users should expect”.

Two consequences for testing:

Perceived risk right below the button

The line of microcopy under a CTA is where the visitor’s main objection gets answered or ignored. For a SaaS trial the objection is usually money (“will I be charged?”) and lock-in (“can I leave?”). For ecommerce it is shipping cost, returns and delivery time. For a demo request it is the fear of a sales call.

This is one of the CTA changes with a plausible chance of a large effect, with one condition: the microcopy must be true on the next screen. Writing “No credit card needed” above a signup step that still asks for a card is the fastest way to turn a click lift into abandonment, and it also breaks trust with the visitors who do sign up. The honest version of this test changes the offer itself, which is exactly what Test 2 below does.

Placement and repetition

A CTA can only convert someone who sees it at the moment they decide. On a short page that moment is near the top; on a long page that answers objections, the decision often happens after the proof section, so repeating the CTA there gives the visitor a way to act without scrolling back. These tests are legitimate, but they tend to shift clicks between buttons more than they create new signups, which is one more reason to judge them on the macro conversion. Count all CTA instances as one goal, or you will read cannibalization as growth.

Contrast, size and accessibility: requirements, not hypotheses

Contrast is where button color has a real job, and it is a pass or fail job. WCAG 2.2 gives the thresholds:

If your current button fails any of these, fix it and ship it: a test would only tell you, slowly, that people click what they can see and hit. Once these thresholds are met, the remaining differences between two passing colors are exactly the small effects the next section argues against chasing.

What happens after the click

Every CTA test is secretly a test of the next screen. The click is a step into a flow, and the flow decides whether the promise in the label is kept: how many fields step 2 asks for, whether a card is required, whether the visitor lands on the trial they were promised or on a plan picker. A CTA test with a real chance of moving signups often changes that flow and then tells the truth about it on the button. For the flow itself, see self-serve signup flow optimization.

Why button color is almost never worth a test

The button color test has a famous origin story, and it has a famous rebuttal from people who run experiments at enormous scale. In “Seven Rules of Thumb for Web Site Experimenters” (KDD 2014), Kohavi, Deng, Longbotham and Xu discuss a published result in which “Red Beats Green” for a “Get Started Now” button, and they write: “Since we do not see a lot of sites with red call-to-action buttons, we believe this is not a general result that replicates well.” Their rule is titled “Your Mileage WILL Vary”.

The same paper is honest about the other side. Bing ran font color experiments in 2013 that improved task success and monetization “to the tune of over $10M annually”, and the result was replicated with “a much larger sample of 32 million users”. But the authors also say such breakthroughs are rare, “perhaps one in 500 experiments” at Bing, and their second rule is titled “Changes Rarely have a Big Positive Impact to Key Metrics”: successful experiments there improve key metrics “by 0.1% to 1.0%”.

Put those two facts next to your traffic. Bing can detect a fraction of a percent because it has tens of millions of users per test. A page with 10,000 visitors a week cannot. For our illustrative page, detecting a 5 percent relative lift in signups needs 207,938 visitors per variant, or 292 days. A color test whose honest expected effect is a few percent is not a test you can finish; it is a test you abandon, or worse, one you stop early on a lucky day.

That does not mean color never matters. It means that, after contrast is met, color is a judgment call you ship, not a hypothesis you spend a month of traffic on.

How much traffic a CTA test needs, by metric

The choice of metric changes the sample size more than any other decision. Clicks happen four times as often as signups on our page (12% against 3%), and lower baselines need much larger samples to detect the same relative lift. At 95% confidence, 80% power, a two-sided test, two variants and 10,000 visitors a week:

relative MDE CTA click (baseline 12%) per variant days signup (baseline 3%) per variant days
5% 47,036 66 207,938 292
10% 12,004 17 53,211 75
15% 5,443 8 24,193 34
20% 3,122 5 13,914 20
Days needed to detect each lift, CTA clicks against signupsPaired horizontal bars for relative minimum detectable effects of 5, 10, 15 and 20 percent at 10,000 visitors a week. CTA clicks at a 12 percent baseline need 66, 17, 8 and 5 days. Signups at a 3 percent baseline need 292, 75, 34 and 20 days; the 292 day bar runs off the chart. A dashed line marks 28 days, four full weeks: on signups only the 20 percent lift fits inside it, while on clicks every lift except 5 percent does.Days to detect each lift at 10,000 visitors a weekCTA click (12%)signup (3%)0306090120 days28 days = 4 full weeksMDE 5%66292, off the chartMDE 10%1775MDE 15%834MDE 20%52095% confidence, 80% power, two-sided, two variants. Same page, same traffic, different metric.
Measuring clicks makes a CTA test look cheap. Measuring signups shows its real price: a 15 percent lift takes 34 days instead of 8.

This is the quiet reason click-based CTA tests are so popular: they finish. A click test at a 15 percent MDE is done in 8 days. The same question asked of signups needs 34 days. Teams that switch the metric to clicks are not choosing a better metric; they are choosing a shorter test that answers a different question.

Run the planning step yourself. With current conversion rate 3, minimum detectable effect 15 relative, confidence 95, power 80 and 10,000 visitors per week on a two-sided test, the calculator shows 24,193 visitors per variation, 48,386 in total for two variations and an estimated duration of 34 days.

Sample size calculator
-Visitors per variation
-Total (2 variations)
-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

Run it backward and you get the budget of a four-week test. With 20,000 visitors per variant, the smallest lift in signups this page can reliably detect is 15.9% relative, or 0.48 percentage points, from 3.00% to 3.48%. Anything you expect to move signups by less than that is not a candidate for a four-week test. The concept and the trade-offs are covered in minimum detectable effect, and the MDE calculator does the backward calculation for any traffic level.

The central trap: deciding by clicks

Here is a common version of that trap, shown with illustrative numbers.

Test 1 compares the generic label “Get started” (A) with the specific “Start my 14-day free trial” (B). Nothing else changes: the next screen still asks for a credit card, as it did before. After four weeks:

metric A: “Get started” B: “Start my 14-day free trial” relative lift p-value 95% CI of the difference
CTA clicks 2,400 / 20,000 (12.00%) 2,760 / 20,000 (13.80%) +15.0% below 0.0001 +1.14 to +2.46 pp
signups 600 / 20,000 (3.00%) 612 / 20,000 (3.06%) +2.0% 0.7263 -0.28 to +0.40 pp
signups per click 25.0% (600 / 2,400) 22.2% (612 / 2,760) lower

On clicks, B is a textbook winner: +15.0 percent, a z statistic of 5.37 and an interval that sits well away from zero. If you enter 20,000 and 2,400 against 20,000 and 2,760 in the significance calculator further down, it reports a p-value of “< 0.0001”. On signups, the test is inconclusive: +2.0 percent with an interval from -0.28 to +0.40 percentage points, which comfortably includes zero.

Click to signup funnel for Test 1, variant A against variant BTwo side by side funnels from 20,000 visits each. Variant A, Get started, has 2,400 CTA clicks, 12.00 percent, and 600 signups, 3.00 percent, so 25.0 percent of clickers sign up. Variant B, Start my 14-day free trial, has 2,760 clicks, 13.80 percent, and 612 signups, 3.06 percent, so 22.2 percent of clickers sign up. The extra 360 clicks in B produce only 12 extra signups.A: “Get started”B: “Start my 14-day free trial”Visits · 20,000Visits · 20,000CTA clicks · 2,400 (12.00%)CTA clicks · 2,760 (13.80%)Signups · 600Signups · 6123.00% of visits3.06% of visits25.0% of clickers sign up22.2% of clickers sign upClicks: +15.0%, p below 0.0001. Signups: +2.0%, p = 0.7263, inconclusive.360 extra clicks, 12 extra signups. The new label pulled clicks from people who quit at the card step.
The click metric reports a big, significant win. The signup metric reports nothing distinguishable from zero. Only one of them pays the bills.

The ratio in the last row explains what happened. B’s label promised a free trial, so more people clicked. The next screen still asked for a credit card, so the extra clickers, the ones who clicked because “free” sounded low risk, left at that step. Signups per click fell from 25.0 to 22.2 percent. The label raised expectations that the flow did not meet, which is precisely the failure NN/g describes.

Kohavi and colleagues have a rule for this pattern too: “Reducing Abandonment is Hard, Shifting Clicks is Easy”. They describe experiments in which a module gets good click-through and is declared good for users, and warn that if it only moved clicks around, “it is only useful if those clicks are better”, for example if they “lead to higher success or purchases”. Their summary: “local improvements are easy; global improvements are much harder.” A CTA click is the most local metric on the page.

What to do with Test 1:

  1. Do not ship B as a win. On the declared macro metric it is inconclusive, and the minimum detectable effect for this test was 15.9 percent; a true effect of 2 percent could exist and this test could never confirm it.
  2. Read the diagnostic. The click lift plus the per-click drop is a strong hint about where the real problem is: the card step, not the button.
  3. Write the next hypothesis from it. “Because the free trial label lifted clicks but not signups, and the click to signup rate fell at the card step, we believe removing the card requirement and saying so at the CTA will raise signups per visit.” The format comes from how to write an A/B test hypothesis.

One more caution about click metrics: a fresh, unusual label can attract curiosity clicks in its first days that fade later. That is the novelty effect, and it inflates micro metrics more than macro ones.

Test 2: change the offer, not the paint

Test 2 keeps the hypothesis honest by changing the product, not just the words. Variant B says “Create a free account, no credit card”, and the card field is actually removed from the next step. Variant A is the original page. Same traffic, same four weeks:

arm visitors signups signup rate
A (original: “Get started”, card required) 20,000 600 3.00%
B (“Create a free account, no credit card”, card removed) 20,000 702 3.51%

The relative lift is +17.0%, with a p-value of 0.0041 (z of 2.87) and a 95% confidence interval of the difference from +0.16 to +0.86 percentage points. The interval does not include zero, and the observed lift sits just above the 15.9 percent this test was sized to detect.

Statistical significance calculator
Control (A)
Variation (B)
Control (A) · Rate-
Variation (B) · Rate-
Relative lift-
p-value-
95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Enter 20000 and 600 for Control (A) and 20000 and 702 for Variation (B). The calculator shows a rate of 3.00% for A and 3.51% for B, a relative lift of +17.0%, a p-value of 0.0041, a 95% CI of the difference of +0.2% … +0.9% (pp) and the verdict Significant winner · B wins. The z statistic and the two-decimal interval quoted above come from the same formula; the screen rounds the interval to one decimal and does not display z. Plug in the click data from Test 1 to see the opposite lesson: the screen shows a p-value of “< 0.0001” for a change that did nothing for signups.

Confidence intervals of the difference for the two CTA testsThree horizontal intervals on a percentage point scale from minus 0.5 to plus 2.5, with a vertical line at zero. Test 1 on CTA clicks runs from plus 1.14 to plus 2.46 points, point estimate plus 1.80, far from zero. Test 1 on signups runs from minus 0.28 to plus 0.40 points, point estimate plus 0.06, crossing zero. Test 2 on signups runs from plus 0.16 to plus 0.86 points, point estimate plus 0.51, entirely above zero.95% CI of the difference, B minus A, in percentage points-0.50+0.5+1.0+1.5+2.0+2.5Test 1, CTA clicksmicro, not the decision+1.14 to +2.46Test 1, signupsinconclusive-0.28 to +0.40, crosses zeroTest 2, signupssignificant on the macro+0.16 to +0.86, above zeroDots are point estimates: +1.80, +0.06 and +0.51 pp. Illustrative data, two-sided z-test.
The widest, most impressive interval belongs to the metric that does not decide anything. The one that matters is narrower and closer to zero, and still clearly positive.

Before calling Test 2 final, check its guardrail metrics. Removing the card requirement is a classic trade: more signups, possibly fewer of them converting to paid when the trial ends. Declare trial to paid conversion (or first week activation) as a guardrail before launch and read it once those cohorts mature; trial to paid conversion A/B testing covers how to wait for that number without stalling every test.

What the winner is worth

Turn the lift into signups per month. At 10,000 visits a week the page gets about 43,333 visits a month (10,000 times 52, divided by 12):

estimate absolute lift extra signups per month (about 43,333 visits)
lower bound of the CI +0.16 pp about 70
point estimate +0.51 pp about 221

The point estimate is the number that goes on a slide, and it is the one to distrust. When you ship a variant because it cleared a significance bar, the observed effect is, on average, larger than the true one, because the tests that happened to draw a lucky high reading are the ones that pass. That is the winner’s curse. Plan on about 70 extra signups a month and treat anything above it as upside.

Testing three or more CTA texts at once

CTA copy is where teams are most tempted to run A/B/n tests: four labels, one control, may the best one win. The trap is arithmetic. Each comparison against the control is another chance for chance itself to cross the line. With three variations against a control at the usual 5 percent threshold, the probability of crowning at least one false winner when none of them works is about 14 percent, not 5.

The standard fix is a multiple comparison correction. The simplest, Bonferroni, divides the threshold by the number of comparisons, so with three variations each one must beat 0.0167 instead of 0.05. It costs sample: every comparison needs more evidence, and every extra arm splits the same weekly traffic further. On a page like ours, where a two-arm test on signups already needs four weeks for a 15.9 percent MDE, a four-arm test on signups is a quarter of a year. The full treatment, including when the extra arms are worth it, is in testing multiple variants without false positives, and the A/B/n significance calculator applies the correction for you.

A practical alternative for low and medium traffic: pick the two most different labels from your list, the ones that make different promises, and test those head to head on signups. Small wording tweaks between similar labels are the copy equivalent of button color.

A CTA A/B testing plan you can copy

Fill in every line before the variant goes live.

  1. Macro conversion defined: the exact event that counts (signup completed, purchase, demo booked), not the click.
  2. Click kept as a diagnostic: tracked, reported, never used to decide.
  3. Hypothesis written with the CTA part being changed (label, risk, placement, next step) and why.
  4. Promise checked against the next screen: whatever the label and microcopy say is true one click later.
  5. Accessibility baseline met: button text at 4.5:1, component boundary at 3:1, target at least 24 by 24 CSS pixels.
  6. MDE checked against traffic: the expected lift on the macro metric is at or above what the planned weeks can detect. Use the sample size calculator.
  7. Duration fixed in whole weeks and a stopping rule written: no decision before the planned sample. Evan Miller’s classic advice applies: “Decide on a sample size in advance and wait until the experiment is over.”
  8. Number of arms justified, with a correction declared if there are three or more.
  9. Guardrails declared: trial to paid conversion or activation for SaaS, refund rate or revenue per visitor for ecommerce.
  10. Value plan prepared with the lower bound of the interval.

Common mistakes in call to action testing

  1. Deciding on clicks. The click is the easiest metric to move and the least connected to revenue.
  2. Writing a promise the next step breaks. “Free” on the button and a card field on step 2 converts curiosity into abandonment.
  3. Testing color before contrast. If the button fails WCAG contrast, fix it; there is nothing to learn.
  4. Testing color at all on medium traffic. The expected effect is far below what four weeks can detect.
  5. Counting repeated CTAs as separate wins. A second button that steals clicks from the first is not growth.
  6. Stopping on a good day. Peeking inflates false positives far beyond 5 percent; in Evan Miller’s worst case example, checking after every observation gives a 26.1 percent false positive rate. See the peeking problem.
  7. Running five labels without a correction. The smallest p-value in an uncorrected table is not a winner.
  8. Forecasting with the point estimate. Plan with the lower bound; the winner’s curse makes the point estimate optimistic.

Make this automatic with Donnu

The hard part of CTA testing is not thinking of a better label. It is resisting the two shortcuts that make a test look finished: deciding on the click because it moves fast, and stopping the moment the dashboard turns green.

Donnu runs A/B tests and Split URL tests on web pages. You build the CTA variant in the visual editor directly on the live page, including the microcopy under the button, and when the real change lives in the next step, such as a signup screen without the card field, Split URL sends each arm to its own page. The conversion goal can be a click, a form submission or a page visit, so you can point it at the signup confirmation page instead of the button and let the test be decided on the macro conversion. Reports are Bayesian, and the product is built PT-BR and LGPD first.

What stays with you is the discipline in this guide: size the test on the metric that pays, write the stopping rule before launch and plan with the lower bound. The sample size calculator and the significance calculator reproduce every number above on your own data, free.

References

Read also: Landing page optimization · Conversion rate optimization guide · Overall evaluation criterion · Minimum detectable effect · Testing multiple variants · The winner’s curse · Self-serve signup flow optimization · Leia em português

Frequently asked questions

What should I test in a CTA besides button color?
Test the things that change the decision, not the paint: the offer and promise in the label, the perceived risk right below the button (card required or not, cancel anytime), how specific the copy is, where the button sits and whether it repeats after the proof, and above all what happens after the click. Contrast and size are accessibility requirements to meet first, not hypotheses: WCAG 2.2 asks for 4.5:1 for button text and 3:1 for the visual boundary that identifies the control.
Is button color worth A/B testing?
Almost never at medium traffic. Once contrast is met, a color change tends to produce a small effect, and small effects need huge samples: on a page with 10,000 visitors a week and a 3 percent signup rate, detecting a 5 percent relative lift needs 207,938 visitors per variant, or 292 days. Kohavi and colleagues note that a published "red beats green" button result is not one they expect to replicate well.
Should I judge a CTA test by clicks or by signups?
By signups or purchases, the macro conversion. Clicks are a diagnostic. In the illustrative test in this guide, "Start my 14-day free trial" beat "Get started" on clicks by 15.0 percent with a p-value below 0.0001, but signups moved only 2.0 percent with a p-value of 0.7263, inconclusive. Signups per click fell from 25.0 to 22.2 percent: the new label pulled extra clicks from people who quit at the next step.
How much traffic does a CTA A/B test need?
Far more when you measure the metric that matters. At 10,000 visitors a week, 95 percent confidence and 80 percent power, detecting a 15 percent relative lift on a 12 percent click rate takes 5,443 visitors per variant (8 days), while the same lift on a 3 percent signup rate takes 24,193 per variant (34 days). A 4-week test with 20,000 visitors per variant can detect about 15.9 percent relative on signups and nothing smaller.
Can I test three or four CTA texts at the same time?
Yes, but correct for multiple comparisons. With three variations against a control at a 5 percent threshold, the chance of at least one false winner when nothing works is about 14 percent. A Bonferroni correction asks each comparison to beat 0.0167 instead of 0.05, and every extra arm also splits your traffic further, so each comparison needs more calendar time.
How much is a CTA test winner worth?
Multiply monthly traffic by the absolute lift and plan with the lower bound of the confidence interval. In the illustrative example, a CTA that removed the credit card requirement lifted signups from 3.00 to 3.51 percent. At about 43,333 visits a month that is about 221 extra signups a month at the point estimate, but about 70 at the lower bound, and 70 is the number to plan with because of the winner's curse.