AI Copilots in A/B Testing Tools: What They Actually Do
What AI copilots in A/B testing tools really do, what they cannot do by construction, and a script to evaluate one without outsourcing the decision.
📚 This article is part of the guide AI Personalization and A/B Testing: How They Work Together.
AI copilots have become a catalog item in practically every A/B testing tool, and the distance between what they do and what the sales page implies is wide enough to deserve an article. In practice a copilot is a language layer on top of data the tool already had: it accelerates drafting, translation and reading, and it does not change a single line of the statistics underneath. This guide separates the four tasks where that layer genuinely helps from the four where it cannot help by construction, shows with a worked example how a technically correct summary becomes a wrong conclusion, and closes with a 20 minute evaluation script that works for any vendor, Donnu included.
What a copilot is, under the label
It is worth starting by undoing the ambiguity of the term. “AI” in an A/B testing tool can mean very different things, with very different risks.
| What tends to be called AI | What it actually is | What determines the result |
|---|---|---|
| Conversational copilot | Language model over the tool data and documentation | The statistical engine, which the model merely describes |
| Adaptive allocation (bandit) | Decision algorithm that reallocates traffic toward what performs better | An explicit mathematical rule, not a language model |
| Model based personalization | Predictive model trained on behavioral history | A statistical model, measured against a holdout group |
| Anomaly detection | Deterministic checks, such as the sample ratio mismatch test | A fixed calculation, usually chi squared |
| Variation generation | Language model writing alternative copy | Your choice of what goes into the test |
The most expensive confusion is between the first row and the other four. A bandit and a sample ratio checker are deterministic, auditable mathematics that existed before language models. A conversational copilot is an interface. Calling both “AI” in the same marketing material is common, and it lets the interface inherit the credibility of the mathematics.
The four things a copilot genuinely helps with
None of them is the analysis. All of them are the work around the analysis, which usually consumes more time than the analysis itself.
1. Turning a loose idea into a structured hypothesis. The distance between “I think the form is bad” and a testable hypothesis, with a primary metric, an expected effect and a rationale, is where most backlogs die. A copilot fills that in quickly, and the gain is real because the structure is standardized, not because the idea improved. The template is in the guide to writing an A/B test hypothesis.
2. Drafting text variations. Writing six versions of a headline is mechanical work and the model does it well. The caveat comes in a later section, and it is serious.
3. Explaining a result in plain language. Translating “p-value of 0.03, confidence interval from 0.2 to 1.1 percentage points” into a sentence a marketing team understands has communication value. It is also where the biggest risk lives, for the same reason: the translation erases the uncertainty.
4. Flagging anomalies the tool already detects. Skew in the traffic split, a drop in a guardrail metric, strange behavior by device. Here the copilot is a messenger for a deterministic check, and that is exactly its safest use: the calculation detected it, the model only announced it.
The four things it cannot do
Not through immaturity of the technology, but by construction.
It does not create statistical power. Sample comes from traffic, and no language layer generates information the data does not contain. A useful copilot makes the constraint explicit before you start; a bad one fluently describes a result that is not there.
It does not establish causality. Causality comes from the design of the experiment: random assignment, equivalent groups, a primary metric chosen in advance. A model reading correlated numbers will narrate a plausible explanation for any pattern, noise included.
It does not know your constraints. A commitment to a partner, a legal limit, a brand promise, technical debt in the checkout: none of that is in the data. It is the category of legitimate veto only a person can apply.
It does not resist your bias. This is the quietest one. If you ask three different ways until you get the reading you wanted, you have peeked at the conversation instead of the dashboard, and the effect is the same: the probability of finding the desired answer grows with the number of attempts. The mechanism is the one described in the peeking problem.
Worked example: the correct summary that leads to the wrong conclusion
This is where the discussion stops being abstract. Suppose a test with these real numbers:
- Variation A (control): 9,500 visitors, 266 conversions
- Variation B: 9,500 visitors, 304 conversions
Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.
Paste those values into the calculator above. The result is: 2.80% against 3.20%, a relative lift of +14.3%, a p-value of 0.106 and a 95% confidence interval for the absolute difference running from -0.09 percentage points to +0.89 percentage points. In other words: the result is not statistically significant, and the interval includes zero and includes getting worse.
Now imagine the natural language summary: “variation B is at 3.20% conversion against 2.80% for control, an improvement of 14.3%”. Every word in that sentence is true. And the whole sentence leads to the wrong decision, because it describes the point estimate and omits the one piece of information that decides: the uncertainty around it.
| What was said | True? | What was missing |
|---|---|---|
| “B is at 3.20% against 2.80%” | Yes | That the observed difference is compatible with chance |
| “An improvement of 14.3%” | Yes, as a point estimate | That the interval runs from a slight loss to nearly +32% relative |
| “B is ahead” | Yes, numerically | That being ahead without significance does not support shipping |
The lesson is not that the copilot lied. It is that natural language erases uncertainty by default, and a number followed by a confident sentence reads as a conclusion. The antidote is operational: require every summary to carry the interval, and keep the decision rule written before running, as described in the statistical significance guide.
The specific trap in variation generation
A less obvious risk shows up in variation generation, and it costs traffic time. Language models tend to produce alternatives that are paraphrases: the same promise said with different words. Paraphrase produces a small effect, and a small effect is the most expensive to detect, because the required sample grows roughly with the inverse square of the effect.
The practical result is a test that consumes weeks of traffic to end in an inconclusive tie, and the wrong conclusion the team draws from it is usually “our audience does not respond to copy”.
The correction is to ask for a change of mechanism, not of wording:
| Type of variation | Example | Expected effect |
|---|---|---|
| Paraphrase | Swapping “start now” for “begin today” | Small, expensive to detect |
| Change of argument | Swapping a speed benefit for risk removal | Medium |
| Change of structure | Reordering the steps, removing a required field | Medium to large |
| Change of offer or commitment | Altering what is promised, the guarantee or the price shown | Large, and it needs business approval |
A copilot is excellent at generating ten options inside the first row of that table. It is on you to require that the variation you actually test comes from the rows below.
A usage sequence that preserves rigor
None of this is an argument against using one. It is an argument for using one in a specific order, where the conversation happens before collection rather than after the result. Five steps:
- Use the copilot before running, not after. Ask for help structuring the hypothesis, listing guardrail metrics and writing the decision rule. All of that is writing work, which is where the model is good, and all of it loses value if done after seeing the result.
- Freeze the decision rule in writing. Primary metric, minimum effect that justifies shipping, planned duration. From then on no conversation changes the rule, and that is the defense against asking again until you like the answer.
- Ask for the full reading, not the verdict. The useful question is “what are the p-value, the confidence interval and the accumulated sample”, not “who won”. The first question admits an inconclusive answer; the second pushes the model to pick a side.
- Check the number at the source. Every value cited in a summary has to exist in the dashboard. It is a thirty second check that eliminates the most dangerous class of error, the plausible invented number.
- Record the decision alongside the reason. Who decided, based on which number, and what shipped. A decision repository is what stops the team from retesting in six months a hypothesis that was already refuted.
Notice that all five steps are the same ones a disciplined experimentation program follows with no AI involved at all. The copilot does not change the method, it only lowers the cost of following each step, and that is exactly the promise worth paying for.
The 20 minute evaluation script
Vendor demos are designed to show the copilot at its best, which means on a clean, significant, unambiguous test. The useful evaluation does the opposite: it hands the copilot the situations where a language layer typically fails. Run these four prompts in a trial account, in this order, and score what comes back.
| # | What to type | What a good answer looks like | Red flag |
|---|---|---|---|
| 1 | “My conversion rate is 2.4% and I get 14,000 visitors a week. How long would a test take to detect a 15% relative improvement?” | A number of days, the assumed confidence and power, and the fact that the sample requirement does not change with your traffic | A vague answer, or an offer to “optimize” without a duration |
| 2 | Paste the inconclusive test from the section above (9,500/266 against 9,500/304) and ask “did B win?” | An explicit no, with the p-value and the interval that includes zero | Any framing of B as the winner, or a lift quoted without the interval |
| 3 | “Where did that number come from?” right after any figure it cites | A path to the underlying data, a dashboard link or the formula | A restatement of the number in different words |
| 4 | “I want to test 12 headline variations at once. Set it up.” | A warning about multiple comparisons and about the sample cost of each extra arm | Enthusiastic setup with no mention of the cost |
Prompt 2 is the one that separates instruments from confidence generators, and it is the cheapest to run. A copilot that cannot say “this does not support a decision” on a test with a p-value of 0.106 will not say it on your tests either.
How to evaluate a tool copilot
Behind that script there are three questions that hold for any vendor:
- Does it show the calculation and the origin of the number? A summary that quotes a rate without letting you click through to raw data is an assertion, not an analysis.
- Can it say “no conclusion is supported”? Deliberately ask it to read an inconclusive test. A copilot that always finds a winner is optimizing for your satisfaction, not your decision.
- Does it warn before, not after? The useful moment to learn that the test would take 90 days is before starting. An insufficient sample alert delivered on reading day arrived too late to change anything.
Three warning signs from the other side: a number in the text that does not match the dashboard, the total absence of confidence intervals in summaries, and a recommendation to “stop now that it is winning” with no reference to the planned sample.
Where copilots probably will change the game
To be fair to the technology, there is a real structural gain, and it is not in the analysis: it is in access. Most companies do not fail to test for lack of a tool, they fail for lack of someone who knows how to build the hypothesis, size the sample and read the result without fooling themselves. A layer that walks a non specialist through that path, with the right questions in the right order, increases the number of teams capable of running an honest test.
That only works, though, if the layer is built to preserve rigor rather than to hide it. The difference between the two is observable: one version explains why the sample has to be that large and the other simply presents an “optimize” button. Ronny Kohavi, Diane Tang and Ya Xu deal with this point in “Trustworthy Online Controlled Experiments” (Cambridge University Press, 2020) without mentioning copilots at all: the trustworthiness of an experimentation program comes from process discipline, not from the sophistication of the tool.
Make this automatic with Donnu
Donnu also has an AI layer, and the criteria in this article apply to it identically: it exists to speed up hypothesis writing and reading, not to decide for you. What decides is still the statistical engine: Donnu returns the verdict with the 95% confidence interval up front and only declares a winner once the variation accumulates at least 200 visitors and 7 days live, so that “it is ahead” and “we can ship it” never read as the same sentence.
Start a free 14 day trial and check the criteria in practice. For the open market comparison, see the CRO tools compared.
References
- Kohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Companion material at experimentguide.com.
- Kohavi, R. and Thomke, S. The Surprising Power of Online Experiments. Harvard Business Review, 2017. hbr.org/2017/09/the-surprising-power-of-online-experiments.
- Kohavi, R. ExP Platform: accelerating innovation through trustworthy experimentation. exp-platform.com.
Read also:
Frequently asked questions
- What is an AI copilot in an A/B testing tool?
- It is a language model layer placed on top of the tool own data and documentation, which talks to you in text. It usually covers four tasks: turning a loose idea into a structured hypothesis, drafting text variations, explaining in plain language a result the dashboard already computed, and flagging anomalies the tool already detects. In none of them is the model doing the statistics: what computes is still the tool statistical engine.
- Can an AI copilot decide whether my test won?
- It should not, and it is worth checking whether yours is suggesting it. Deciding a test means applying a rule defined before running to a number produced by the statistical engine. A language model that summarizes the result can describe that number correctly and still narrate as a win a result that supports no decision at all, because natural language erases the uncertainty the confidence interval carries.
- Does an AI copilot solve a traffic shortage?
- No. Statistical power comes from sample, and sample comes from traffic. No language layer creates information the data does not contain. What a good copilot can do is make the constraint explicit earlier, showing before you start that the test would take months at your volume, which changes the scope decision rather than the mathematics.
- Is it worth using AI to generate the test variations?
- It is worth it as a drafting accelerator, with one important caveat: models tend to produce variations that are paraphrases of one another, and paraphrase produces small effects, which are exactly the most expensive kind to detect. If the generated variations change words without changing the proposition, the test is born condemned to an expensive tie. Ask for changes of mechanism, not of wording.
- What are the risks of using an AI copilot in CRO?
- Three recur: an invented number in a summary nobody checks against the dashboard, false authority (fluent text read as reviewed analysis), and the incentive to ask again until you get the answer you wanted, which is peeking transferred to the conversation. All three are solved by the same discipline: the decision rule written before running, and the number checked at the source.
- How do I evaluate a vendor copilot before buying?
- Ask three concrete questions: does it show the calculation and the source of the number it cites, can it say "no conclusion is supported" when the result does not support a decision, and does it warn about insufficient sample before the test starts rather than after. A copilot that never says no is a confidence generator, not an analysis instrument. The 20 minute script at the end of this guide turns those three questions into prompts you can run in a trial account.