CTA A/B Testing: What to Test Beyond Button Color
CTA A/B testing that moves signups, not just clicks: test the offer, the risk and the next step, size it to your traffic and skip the button color.

📚 This article is part of the guide Conversion Rate Optimization (CRO): The Complete 2026 Guide.
CTA A/B testing pays off when you test what the button promises and what happens after it is clicked, not how it looks. The offer in the label, the risk the visitor perceives right below it and the step that follows the click move signups; color almost never does at medium traffic. And the single most common way to fool yourself is to decide the test on CTA clicks instead of signups or purchases. This guide is part of our landing page optimization method, which in turn sits inside the conversion rate optimization guide. It walks through the parts of a call to action worth testing, why button color rarely earns a slot, how much traffic each metric needs, and two illustrative tests on the same page: one that “wins” on clicks and changes nothing, and one that wins where it counts.
Button color is the most famous CTA test and the least useful one for most teams. It is easy to build, so it gets built. The cost is not the build. The cost is the four or eight weeks of traffic it consumes while a test that could have moved revenue waits in the queue.
What CTA A/B testing actually measures
A call to action (CTA) is the control that asks the visitor to take the page’s main step: start a trial, create an account, add to cart, book a demo. CTA A/B testing compares two or more versions of that control, and of what surrounds it, by splitting real traffic at random and measuring which version produces more of the outcome you care about.
That last phrase hides the whole problem. There are two outcomes on the table:
- The micro conversion is the click on the button itself. It happens fast, it happens often and it is easy to count.
- The macro conversion is the thing the page exists for: a completed signup, a purchase, a booked demo. It happens later, less often and sometimes on a different page.
A CTA test that lifts the micro conversion and leaves the macro flat has not improved anything. It has only moved people one screen further before they leave. The overall evaluation criterion of a CTA test is therefore the macro conversion, declared before launch. Clicks stay on the report as a diagnostic: they explain why the macro moved, or why it did not.
For the rest of this guide we use one illustrative page, not customer data: a SaaS trial signup page with 10,000 visitors a week, a CTA click rate of 12% and a signup rate of 3% of visits. A four-week test gives 40,000 visits, 20,000 per variant.
The anatomy of a call to action: what is worth testing
A CTA is not just a button. It is a small system: a label that makes a promise, a line of microcopy that answers the visitor’s main worry, a position on the page, a visual treatment and, out of sight, the screen the click leads to. Each part changes a different thing in the visitor’s decision.
| CTA part | what it changes in the decision | example change | metric that decides |
|---|---|---|---|
| Label (offer and promise) | what the visitor believes they get | “Get started” to “Start my 14-day free trial” | signups per visit |
| Microcopy below the button | the perceived cost or risk of clicking | add “No credit card needed. Cancel anytime.” | signups per visit |
| Placement and repetition | whether the CTA is seen at the moment of decision | repeat the CTA after the proof section | signups per visit |
| Contrast, size, target area | whether the control is perceivable and usable | meet WCAG contrast and target size | fix, do not test |
| What happens after the click | whether the promise is kept on the next screen | remove the card field from step 2 | signups per visit, then activation |
| Color of the button | almost nothing, once contrast is met | green to orange | rarely worth the traffic |
The offer in the label: specific beats generic
The label is a promise. “Get started”, “Submit” and “Learn more” promise nothing in particular, which is exactly why they feel safe to write and why they underperform as decision aids. A specific label names the outcome, the price and the commitment: “Start my 14-day free trial”, “Get the pricing PDF”, “Book a 20-minute demo”.
Nielsen Norman Group studied this exact label. In “Get Started Stops Users” (Harley and Flaherty, 2017), they report that a generic Get Started call to action “attracts clicks, but also misleads users”, because visitors who wanted information clicked it and landed in surveys, quizzes and signup flows they were not ready for. Their conclusion is the thesis of this guide in one line: using a generic label “may indeed get more clicks on the CTA, but a link is a promise”, and if the promise is wrong you reach “superficial conversion goals” without keeping customers. They recommend stating “precisely what users should expect”.
Two consequences for testing:
- A specific label can lower clicks and still raise signups, because it filters out people who were never going to complete the next step. Judging that test on clicks would kill the better variant.
- The label must match the promise that brought the visitor in. If the ad said “free for 14 days” and the button says “Request access”, you have created a mismatch before the visitor reads a word of the page. That is its own test family, covered in ad to landing page message match.
Perceived risk right below the button
The line of microcopy under a CTA is where the visitor’s main objection gets answered or ignored. For a SaaS trial the objection is usually money (“will I be charged?”) and lock-in (“can I leave?”). For ecommerce it is shipping cost, returns and delivery time. For a demo request it is the fear of a sales call.
This is one of the CTA changes with a plausible chance of a large effect, with one condition: the microcopy must be true on the next screen. Writing “No credit card needed” above a signup step that still asks for a card is the fastest way to turn a click lift into abandonment, and it also breaks trust with the visitors who do sign up. The honest version of this test changes the offer itself, which is exactly what Test 2 below does.
Placement and repetition
A CTA can only convert someone who sees it at the moment they decide. On a short page that moment is near the top; on a long page that answers objections, the decision often happens after the proof section, so repeating the CTA there gives the visitor a way to act without scrolling back. These tests are legitimate, but they tend to shift clicks between buttons more than they create new signups, which is one more reason to judge them on the macro conversion. Count all CTA instances as one goal, or you will read cannibalization as growth.
Contrast, size and accessibility: requirements, not hypotheses
Contrast is where button color has a real job, and it is a pass or fail job. WCAG 2.2 gives the thresholds:
- Success Criterion 1.4.3, Contrast (Minimum): text, including the text on a button, needs a contrast ratio of at least 4.5:1, or 3:1 for large text (at least 18 point, or 14 point bold).
- Success Criterion 1.4.11, Non-text Contrast: the visual information required to identify a user interface component needs at least 3:1 against adjacent colors. The W3C explanation notes that a button without a visual boundary can pass when its text is sufficient to indicate the control.
- Success Criterion 2.5.8, Target Size (Minimum): the target for pointer input should be at least 24 by 24 CSS pixels, with exceptions such as adequate spacing.
If your current button fails any of these, fix it and ship it: a test would only tell you, slowly, that people click what they can see and hit. Once these thresholds are met, the remaining differences between two passing colors are exactly the small effects the next section argues against chasing.
What happens after the click
Every CTA test is secretly a test of the next screen. The click is a step into a flow, and the flow decides whether the promise in the label is kept: how many fields step 2 asks for, whether a card is required, whether the visitor lands on the trial they were promised or on a plan picker. A CTA test with a real chance of moving signups often changes that flow and then tells the truth about it on the button. For the flow itself, see self-serve signup flow optimization.
Why button color is almost never worth a test
The button color test has a famous origin story, and it has a famous rebuttal from people who run experiments at enormous scale. In “Seven Rules of Thumb for Web Site Experimenters” (KDD 2014), Kohavi, Deng, Longbotham and Xu discuss a published result in which “Red Beats Green” for a “Get Started Now” button, and they write: “Since we do not see a lot of sites with red call-to-action buttons, we believe this is not a general result that replicates well.” Their rule is titled “Your Mileage WILL Vary”.
The same paper is honest about the other side. Bing ran font color experiments in 2013 that improved task success and monetization “to the tune of over $10M annually”, and the result was replicated with “a much larger sample of 32 million users”. But the authors also say such breakthroughs are rare, “perhaps one in 500 experiments” at Bing, and their second rule is titled “Changes Rarely have a Big Positive Impact to Key Metrics”: successful experiments there improve key metrics “by 0.1% to 1.0%”.
Put those two facts next to your traffic. Bing can detect a fraction of a percent because it has tens of millions of users per test. A page with 10,000 visitors a week cannot. For our illustrative page, detecting a 5 percent relative lift in signups needs 207,938 visitors per variant, or 292 days. A color test whose honest expected effect is a few percent is not a test you can finish; it is a test you abandon, or worse, one you stop early on a lucky day.
That does not mean color never matters. It means that, after contrast is met, color is a judgment call you ship, not a hypothesis you spend a month of traffic on.
How much traffic a CTA test needs, by metric
The choice of metric changes the sample size more than any other decision. Clicks happen four times as often as signups on our page (12% against 3%), and lower baselines need much larger samples to detect the same relative lift. At 95% confidence, 80% power, a two-sided test, two variants and 10,000 visitors a week:
| relative MDE | CTA click (baseline 12%) per variant | days | signup (baseline 3%) per variant | days |
|---|---|---|---|---|
| 5% | 47,036 | 66 | 207,938 | 292 |
| 10% | 12,004 | 17 | 53,211 | 75 |
| 15% | 5,443 | 8 | 24,193 | 34 |
| 20% | 3,122 | 5 | 13,914 | 20 |
This is the quiet reason click-based CTA tests are so popular: they finish. A click test at a 15 percent MDE is done in 8 days. The same question asked of signups needs 34 days. Teams that switch the metric to clicks are not choosing a better metric; they are choosing a shorter test that answers a different question.
Run the planning step yourself. With current conversion rate 3, minimum detectable effect 15 relative, confidence 95, power 80 and 10,000 visitors per week on a two-sided test, the calculator shows 24,193 visitors per variation, 48,386 in total for two variations and an estimated duration of 34 days.
Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.
Run it backward and you get the budget of a four-week test. With 20,000 visitors per variant, the smallest lift in signups this page can reliably detect is 15.9% relative, or 0.48 percentage points, from 3.00% to 3.48%. Anything you expect to move signups by less than that is not a candidate for a four-week test. The concept and the trade-offs are covered in minimum detectable effect, and the MDE calculator does the backward calculation for any traffic level.
The central trap: deciding by clicks
Here is a common version of that trap, shown with illustrative numbers.
Test 1 compares the generic label “Get started” (A) with the specific “Start my 14-day free trial” (B). Nothing else changes: the next screen still asks for a credit card, as it did before. After four weeks:
| metric | A: “Get started” | B: “Start my 14-day free trial” | relative lift | p-value | 95% CI of the difference |
|---|---|---|---|---|---|
| CTA clicks | 2,400 / 20,000 (12.00%) | 2,760 / 20,000 (13.80%) | +15.0% | below 0.0001 | +1.14 to +2.46 pp |
| signups | 600 / 20,000 (3.00%) | 612 / 20,000 (3.06%) | +2.0% | 0.7263 | -0.28 to +0.40 pp |
| signups per click | 25.0% (600 / 2,400) | 22.2% (612 / 2,760) | lower |
On clicks, B is a textbook winner: +15.0 percent, a z statistic of 5.37 and an interval that sits well away from zero. If you enter 20,000 and 2,400 against 20,000 and 2,760 in the significance calculator further down, it reports a p-value of “< 0.0001”. On signups, the test is inconclusive: +2.0 percent with an interval from -0.28 to +0.40 percentage points, which comfortably includes zero.
The ratio in the last row explains what happened. B’s label promised a free trial, so more people clicked. The next screen still asked for a credit card, so the extra clickers, the ones who clicked because “free” sounded low risk, left at that step. Signups per click fell from 25.0 to 22.2 percent. The label raised expectations that the flow did not meet, which is precisely the failure NN/g describes.
Kohavi and colleagues have a rule for this pattern too: “Reducing Abandonment is Hard, Shifting Clicks is Easy”. They describe experiments in which a module gets good click-through and is declared good for users, and warn that if it only moved clicks around, “it is only useful if those clicks are better”, for example if they “lead to higher success or purchases”. Their summary: “local improvements are easy; global improvements are much harder.” A CTA click is the most local metric on the page.
What to do with Test 1:
- Do not ship B as a win. On the declared macro metric it is inconclusive, and the minimum detectable effect for this test was 15.9 percent; a true effect of 2 percent could exist and this test could never confirm it.
- Read the diagnostic. The click lift plus the per-click drop is a strong hint about where the real problem is: the card step, not the button.
- Write the next hypothesis from it. “Because the free trial label lifted clicks but not signups, and the click to signup rate fell at the card step, we believe removing the card requirement and saying so at the CTA will raise signups per visit.” The format comes from how to write an A/B test hypothesis.
One more caution about click metrics: a fresh, unusual label can attract curiosity clicks in its first days that fade later. That is the novelty effect, and it inflates micro metrics more than macro ones.
Test 2: change the offer, not the paint
Test 2 keeps the hypothesis honest by changing the product, not just the words. Variant B says “Create a free account, no credit card”, and the card field is actually removed from the next step. Variant A is the original page. Same traffic, same four weeks:
| arm | visitors | signups | signup rate |
|---|---|---|---|
| A (original: “Get started”, card required) | 20,000 | 600 | 3.00% |
| B (“Create a free account, no credit card”, card removed) | 20,000 | 702 | 3.51% |
The relative lift is +17.0%, with a p-value of 0.0041 (z of 2.87) and a 95% confidence interval of the difference from +0.16 to +0.86 percentage points. The interval does not include zero, and the observed lift sits just above the 15.9 percent this test was sized to detect.
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Enter 20000 and 600 for Control (A) and 20000 and 702 for Variation (B). The calculator shows a rate of 3.00% for A and 3.51% for B, a relative lift of +17.0%, a p-value of 0.0041, a 95% CI of the difference of +0.2% … +0.9% (pp) and the verdict Significant winner · B wins. The z statistic and the two-decimal interval quoted above come from the same formula; the screen rounds the interval to one decimal and does not display z. Plug in the click data from Test 1 to see the opposite lesson: the screen shows a p-value of “< 0.0001” for a change that did nothing for signups.
Before calling Test 2 final, check its guardrail metrics. Removing the card requirement is a classic trade: more signups, possibly fewer of them converting to paid when the trial ends. Declare trial to paid conversion (or first week activation) as a guardrail before launch and read it once those cohorts mature; trial to paid conversion A/B testing covers how to wait for that number without stalling every test.
What the winner is worth
Turn the lift into signups per month. At 10,000 visits a week the page gets about 43,333 visits a month (10,000 times 52, divided by 12):
| estimate | absolute lift | extra signups per month (about 43,333 visits) |
|---|---|---|
| lower bound of the CI | +0.16 pp | about 70 |
| point estimate | +0.51 pp | about 221 |
The point estimate is the number that goes on a slide, and it is the one to distrust. When you ship a variant because it cleared a significance bar, the observed effect is, on average, larger than the true one, because the tests that happened to draw a lucky high reading are the ones that pass. That is the winner’s curse. Plan on about 70 extra signups a month and treat anything above it as upside.
Testing three or more CTA texts at once
CTA copy is where teams are most tempted to run A/B/n tests: four labels, one control, may the best one win. The trap is arithmetic. Each comparison against the control is another chance for chance itself to cross the line. With three variations against a control at the usual 5 percent threshold, the probability of crowning at least one false winner when none of them works is about 14 percent, not 5.
The standard fix is a multiple comparison correction. The simplest, Bonferroni, divides the threshold by the number of comparisons, so with three variations each one must beat 0.0167 instead of 0.05. It costs sample: every comparison needs more evidence, and every extra arm splits the same weekly traffic further. On a page like ours, where a two-arm test on signups already needs four weeks for a 15.9 percent MDE, a four-arm test on signups is a quarter of a year. The full treatment, including when the extra arms are worth it, is in testing multiple variants without false positives, and the A/B/n significance calculator applies the correction for you.
A practical alternative for low and medium traffic: pick the two most different labels from your list, the ones that make different promises, and test those head to head on signups. Small wording tweaks between similar labels are the copy equivalent of button color.
A CTA A/B testing plan you can copy
Fill in every line before the variant goes live.
- Macro conversion defined: the exact event that counts (signup completed, purchase, demo booked), not the click.
- Click kept as a diagnostic: tracked, reported, never used to decide.
- Hypothesis written with the CTA part being changed (label, risk, placement, next step) and why.
- Promise checked against the next screen: whatever the label and microcopy say is true one click later.
- Accessibility baseline met: button text at 4.5:1, component boundary at 3:1, target at least 24 by 24 CSS pixels.
- MDE checked against traffic: the expected lift on the macro metric is at or above what the planned weeks can detect. Use the sample size calculator.
- Duration fixed in whole weeks and a stopping rule written: no decision before the planned sample. Evan Miller’s classic advice applies: “Decide on a sample size in advance and wait until the experiment is over.”
- Number of arms justified, with a correction declared if there are three or more.
- Guardrails declared: trial to paid conversion or activation for SaaS, refund rate or revenue per visitor for ecommerce.
- Value plan prepared with the lower bound of the interval.
Common mistakes in call to action testing
- Deciding on clicks. The click is the easiest metric to move and the least connected to revenue.
- Writing a promise the next step breaks. “Free” on the button and a card field on step 2 converts curiosity into abandonment.
- Testing color before contrast. If the button fails WCAG contrast, fix it; there is nothing to learn.
- Testing color at all on medium traffic. The expected effect is far below what four weeks can detect.
- Counting repeated CTAs as separate wins. A second button that steals clicks from the first is not growth.
- Stopping on a good day. Peeking inflates false positives far beyond 5 percent; in Evan Miller’s worst case example, checking after every observation gives a 26.1 percent false positive rate. See the peeking problem.
- Running five labels without a correction. The smallest p-value in an uncorrected table is not a winner.
- Forecasting with the point estimate. Plan with the lower bound; the winner’s curse makes the point estimate optimistic.
Make this automatic with Donnu
The hard part of CTA testing is not thinking of a better label. It is resisting the two shortcuts that make a test look finished: deciding on the click because it moves fast, and stopping the moment the dashboard turns green.
Donnu runs A/B tests and Split URL tests on web pages. You build the CTA variant in the visual editor directly on the live page, including the microcopy under the button, and when the real change lives in the next step, such as a signup screen without the card field, Split URL sends each arm to its own page. The conversion goal can be a click, a form submission or a page visit, so you can point it at the signup confirmation page instead of the button and let the test be decided on the macro conversion. Reports are Bayesian, and the product is built PT-BR and LGPD first.
What stays with you is the discipline in this guide: size the test on the metric that pays, write the stopping rule before launch and plan with the lower bound. The sample size calculator and the significance calculator reproduce every number above on your own data, free.
References
- W3C. Understanding Success Criterion 1.4.11: Non-text Contrast (WCAG 2.2). Page read in full. The passages used: “The visual presentation of the following have a contrast ratio of at least 3:1 against adjacent color(s): User Interface Components Visual information required to identify user interface components and states” and the example of a button without a visual boundary that passes because “the button’s text is sufficient to indicate the presence of the control”. Supports the contrast requirement for button boundaries. Checked on October 23, 2026. w3.org.
- W3C. Understanding Success Criterion 1.4.3: Contrast (Minimum) (WCAG 2.2). Page read. The passages used: “The visual presentation of text and images of text has a contrast ratio of at least 4.5:1”, “Large-scale text and images of large-scale text have a contrast ratio of at least 3:1” and the definition of large scale as “at least 18 point or 14 point bold”. Supports the button text contrast threshold. Checked on October 23, 2026. w3.org.
- W3C. Understanding Success Criterion 2.5.8: Target Size (Minimum) (WCAG 2.2). Page read. The passage used: “The size of the target for pointer inputs is at least 24 by 24 CSS pixels”, followed by the listed exceptions such as spacing. Supports the target size line in the accessibility section and the checklist. Checked on October 23, 2026. w3.org.
- Harley, A. and Flaherty, K. “Get Started” Stops Users. Nielsen Norman Group, August 20, 2017. Article read in full. The passages used: “A generic Get Started call-to-action attracts clicks, but also misleads users”, “Tricking people by using a Get Started link may indeed get more clicks on the CTA, but a link is a promise”, “you may reach superficial conversion goals” and the advice to increase information scent “by stating precisely what users should expect”. Supports the section on specific labels and the reading of Test 1. Checked on October 23, 2026. nngroup.com.
- Kohavi, R., Deng, A., Longbotham, R. and Xu, Y. Seven Rules of Thumb for Web Site Experimenters. KDD 2014, author copy. Paper read. The passages used: Rule 3 on “Red Beats Green” for a “Get Started Now” button, “we believe this is not a general result that replicates well”; Rule 1 on Bing font colors, “over $10M annually”, replicated with “32 million users”, and breakthroughs at “perhaps one in 500 experiments”; Rule 2, “Changes Rarely have a Big Positive Impact to Key Metrics”, where successful experiments “improve key metrics by 0.1% to 1.0%”; Rule 5, “Reducing Abandonment is Hard, Shifting Clicks is Easy” and “local improvements are easy; global improvements are much harder.” Supports the button color section and the click trap. Checked on October 23, 2026. exp-platform.com.
- Miller, E. How Not To Run an A/B Test. April 18, 2010. Article read in full. The passages used: the scenario stopped “as soon as there is 5% significance”, with the answer “Try 26.1%”, and the advice “Decide on a sample size in advance and wait until the experiment is over”. Supports the stopping rule and the peeking mistake. Checked on October 23, 2026. evanmiller.org.
- The page, both tests, sample sizes and value estimates in this guide are illustrative, not customer data, and were computed with the same formulas that power the calculators on this blog (two-proportion normal approximation and two-sided z-test).
Read also: Landing page optimization · Conversion rate optimization guide · Overall evaluation criterion · Minimum detectable effect · Testing multiple variants · The winner’s curse · Self-serve signup flow optimization · Leia em português
Frequently asked questions
- What should I test in a CTA besides button color?
- Test the things that change the decision, not the paint: the offer and promise in the label, the perceived risk right below the button (card required or not, cancel anytime), how specific the copy is, where the button sits and whether it repeats after the proof, and above all what happens after the click. Contrast and size are accessibility requirements to meet first, not hypotheses: WCAG 2.2 asks for 4.5:1 for button text and 3:1 for the visual boundary that identifies the control.
- Is button color worth A/B testing?
- Almost never at medium traffic. Once contrast is met, a color change tends to produce a small effect, and small effects need huge samples: on a page with 10,000 visitors a week and a 3 percent signup rate, detecting a 5 percent relative lift needs 207,938 visitors per variant, or 292 days. Kohavi and colleagues note that a published "red beats green" button result is not one they expect to replicate well.
- Should I judge a CTA test by clicks or by signups?
- By signups or purchases, the macro conversion. Clicks are a diagnostic. In the illustrative test in this guide, "Start my 14-day free trial" beat "Get started" on clicks by 15.0 percent with a p-value below 0.0001, but signups moved only 2.0 percent with a p-value of 0.7263, inconclusive. Signups per click fell from 25.0 to 22.2 percent: the new label pulled extra clicks from people who quit at the next step.
- How much traffic does a CTA A/B test need?
- Far more when you measure the metric that matters. At 10,000 visitors a week, 95 percent confidence and 80 percent power, detecting a 15 percent relative lift on a 12 percent click rate takes 5,443 visitors per variant (8 days), while the same lift on a 3 percent signup rate takes 24,193 per variant (34 days). A 4-week test with 20,000 visitors per variant can detect about 15.9 percent relative on signups and nothing smaller.
- Can I test three or four CTA texts at the same time?
- Yes, but correct for multiple comparisons. With three variations against a control at a 5 percent threshold, the chance of at least one false winner when nothing works is about 14 percent. A Bonferroni correction asks each comparison to beat 0.0167 instead of 0.05, and every extra arm also splits your traffic further, so each comparison needs more calendar time.
- How much is a CTA test winner worth?
- Multiply monthly traffic by the absolute lift and plan with the lower bound of the confidence interval. In the illustrative example, a CTA that removed the credit card requirement lifted signups from 3.00 to 3.51 percent. At about 43,333 visits a month that is about 221 extra signups a month at the point estimate, but about 70 at the lower bound, and 70 is the number to plan with because of the winner's curse.