Landing Page Video A/B Testing: Help or Hurt?

Putting a video on a landing page is never one change. It is at least four at once: it changes what the page communicates, how long it takes to load, what fits on the first screenful, and how much effort the visitor has to spend to consume the information. That is why video tests come out so contradictory, and why the “80 percent lift” number that has circulated for a decade has no retrievable primary source. This guide covers the four decisions hidden inside “let’s add a video”, what browser policy and accessibility standards require of autoplay, why video counts toward LCP, and a worked three-arm example where the autoplay video moves nothing and the click-to-play video sits 7.8 percent above control without surviving the multiple-comparison correction. This guide is part of our complete guide to conversion rate optimization.

The number that circulates without a source

Search for landing page video and you will find, across dozens of pages, the claim that video lifts conversion by up to 80 percent, sometimes 86 percent, attributed to a company called EyeView. The number shows up in agency articles, in posts from video vendors and in marketing statistics roundups.

What does not show up is the original report. The citations point at one another and at blog posts, not at a consultable document with methodology, sample, period, experimental design and a definition of conversion. Without that, the basics are unknowable: were these randomized A/B tests or before-and-after comparisons? What counted as “conversion”? How many pages? In which industries? What was the spread around the average?

Under the fact-check rule we apply on this blog, a number with no recoverable primary source does not enter as fact. It can enter as what it is: market folklore that survived because it is convenient for anyone selling video production. The honest claim that available evidence supports is far more modest and far more useful: the effect of video varies enormously case by case, and much of that variation is explained by implementation decisions nobody records when quoting the average.

This is the same pattern we documented in urgency and scarcity and in trust badges: a market average quoted secondhand says very little about what will happen on your page.

The four decisions hidden inside “let’s add a video”

decision	options	what it actually changes
Type of video	product demo, customer testimonial, offer explainer, brand film	the only one of the four that changes the page’s information; the other three change the delivery
Placement	above the fold replacing the hero image, below the fold, inside a proof block	above the fold it competes with the copy for the 57 percent of attention time that lives there; below it, it only exists for people who scrolled
Start behavior	muted autoplay, autoplay with sound, click to play	autoplay downloads the video for everyone; click downloads it only for people who asked, which changes the performance cost entirely
Weight and format	self-hosted file, platform embed, poster image with lazy loading	determines whether the video becomes the largest visible element and therefore the page LCP

A test that swaps “page without video” for “page with video” moves all four at once, and the result does not teach which one carried the effect. That is the same confounding trap we cover in long copy vs short copy: a legitimate package test, as long as the report does not pretend to say “video works”.

The cheapest decision to isolate, and often the decisive one, is the third: autoplay or click. It changes nothing about the content and changes almost everything about performance and behavior.

Landing page video counts toward LCP

This is the part that rarely comes up in a copy discussion and that usually explains negative results.

Per the web.dev documentation on Largest Contentful Paint, the video element is considered for the metric: the measurement uses the poster image load time or the first frame presentation time, whichever is earlier. A large video at the top of a page is almost always its biggest visible element, so it becomes the LCP. And the published target is explicit: sites should strive for an LCP of 2.5 seconds or less, measured at the 75th percentile of page loads, segmented across mobile and desktop.

The practical consequence is that a video test is always also a performance test, and without measuring that you cannot interpret a negative result. If the video version lost, was it the video failing to convince or 1.5 extra seconds of waiting? Without LCP for both arms the question stays open. The effect of speed on conversion has its own guide in page speed and conversion.

Autoplay: what the browser allows and what the standard requires

Two layers of constraint, and both are usually discovered after the test is already live.

The browser. The Chrome autoplay policy, in force since version 66 (April 2018, with the Web Audio API included in version 71), establishes that muted autoplay is always allowed, and that autoplay with sound happens only when at least one of these holds: the user has interacted with the domain (click, tap); on desktop, their Media Engagement Index threshold has been crossed, meaning they have previously played video with sound; the user added the site to their home screen on mobile or installed the PWA on desktop; or the top frame delegated autoplay permission to the iframe. Otherwise the promise returned by play() is rejected with NotAllowedError, and it is on the developer to offer manual controls.

Accessibility. WCAG 2.2 Success Criterion 1.4.2, Level A, states: if any audio on a page plays automatically for more than 3 seconds, either a mechanism is available to pause or stop the audio, or a mechanism is available to control audio volume independently from the overall system volume level. Criterion 2.2.2, also Level A, requires that any moving information that starts automatically, lasts more than five seconds and is presented in parallel with other content offer a mechanism to pause, stop or hide it, unless the movement is essential to the activity.

autoplay decision	does the browser allow it?	what the standard requires	practical effect on the test
no autoplay, click to play	always	nothing beyond normal controls	does not affect LCP if the file loads after the click
muted autoplay, short loop	yes	a pause, stop or hide mechanism once it runs past 5 seconds alongside other content (2.2.2)	becomes the page LCP when it sits above the fold
autoplay with sound	almost never, without prior interaction	a pause mechanism or independent volume control past 3 seconds (1.4.2)	in practice the visitor sees a muted video playing itself, the worst of both worlds

The middle row is where nearly everyone ends up: muted, looping, above the fold. It is worth knowing that this choice arrives with a built-in LCP bill and a pause-control obligation, and that it turns the video into a decorative element, because a muted video playing to itself rarely communicates anything the copy would not communicate better.

What to measure, and what is only a diagnostic

signal	role in the test	trap
the page primary conversion	primary metric	none, as long as it is declared up front
LCP in both arms	declared guardrail	without it, a negative result is ambiguous
play rate	diagnostic	measures who pressed, not who bought
percent watched	diagnostic	useful for knowing whether the video is too long, not for calling a winner
time on page	diagnostic	rises automatically with a video playing, with nothing having improved
scroll depth	diagnostic	can fall when the video holds the visitor at the top, without that being bad

Play rate deserves its own paragraph because it is the number that appears most in video reports and decides the least. It answers “how many people were interested in the video”, which is a media question, not a conversion one. A video can have a 40 percent play rate and conversion identical to control, and the right reading is that it entertained without convincing. It can also have an 8 percent play rate and move conversion, because that 8 percent was exactly the set of people carrying the doubt the video answers. Use play rate to decide whether version 2 is worth producing, and conversion to decide what stays live. That counts as a guardrail declared up front, along the lines of guardrail metrics.

Sample size

page conversion	detect +5% relative	detect +10% relative	detect +15% relative
2.0%	315,206 per variant	80,682 per variant	36,693 per variant
5.1%	119,600 per variant	30,588 per variant	13,899 per variant
10.0%	57,763 per variant	14,751 per variant	6,693 per variant

Current conversion rate%Minimum detectable effectrelative (%)absolute (pp)Confidence909599%Power8090%Visitors per week (total)Testtwo-sidedone-sided

-Visitors per variation

-Total (2 variations)

-Estimated duration

Two-proportion normal approximation, 2 variations (50/50). Tweak the inputs and watch it update live.

With three arms and 31,000 visitors per variant, the test runs 55 days at 12,000 visitors a week, or 33 days at 20,000. Since three arms mean two comparisons against control, the significance threshold needs correcting: with Bonferroni at 5 percent, each comparison now needs a p-value below 0.025. See A/B/n testing and correction for multiple variants.

Worked example: image, autoplay and click to play

A software company runs a trial-capture landing page converting at 5.10 percent. The hypothesis is that a 90-second video explaining the offer raises conversion. Instead of testing “with video against without video”, the team builds three arms around the same video, varying only how it enters the page:

• A (control): static image above the fold, copy unchanged.
• B: muted, looping autoplay video in the image’s place.
• C: poster image with a play button in the same place; the video file is only downloaded after the click.

The test runs 55 days at 12,000 visitors a week, with 31,000 visitors per arm (above the 30,588 the calculator asks for at 10 percent relative).

arm	visitors	trials	conversion	play rate	LCP (75th percentile)
A (static image)	31,000	1,581	5.10%	not applicable	1.9s
B (muted autoplay)	31,000	1,550	5.00%	100% by construction	3.4s
C (poster and play)	31,000	1,705	5.50%	11.4%	2.0s

The two comparisons against control:

comparison	relative lift	p-value	95% CI of the difference	corrected threshold (Bonferroni, 2 comparisons)	verdict
A against B	-2.0%	0.5697	-0.44 to 0.24 percentage points	0.025	tie
A against C	+7.8%	0.0262	0.05 to 0.75 percentage points	0.025	misses, narrowly

Control (A)

VisitorsConversions

Variation (B)

VisitorsConversions

Confidence90%95%99%

Control (A) · Rate-

Variation (B) · Rate-

Relative lift-

p-value-

95% CI of the difference-

Two-sided two-proportion z-test. "Not significant" almost always means not enough sample, not that the versions are equal.

Paste 31000 and 1581 into side A and 31000 and 1550 into side B to reproduce the first row: p-value 0.5697, relative lift -2.0 percent, confidence interval from -0.44 to 0.24 percentage points, straddling zero. Swap side B for 31000 and 1705 and the calculator returns a p-value of 0.0262 with an interval from 0.05 to 0.75 percentage points.

The honest reading. Arm C was the best of the three, but the result does not clear the corrected threshold. With three arms, a p-value of 0.0262 sits above the 0.025 the Bonferroni adjustment requires, and declaring victory here is exactly the kind of decision that inflates the false positive rate of an entire program. What the team can legitimately do is one of three things: extend the test until the effect confirms or disappears; run a two-arm confirmation, A against C, at the full 0.05 threshold; or file it as promising and move to the next hypothesis. What they cannot do is write “the video lifted conversion by 7.8 percent” in the report.

And what the test already taught, winner or not. Arm B closes one question: autoplay does not help, and it costs 1.5 seconds of LCP, which pushes the page outside the 2.5 second target. That is actionable regardless of the rest. If the organization does nothing else with this test, it still learned not to put the video on autoplay above the fold.

Notice too the play rate in arm C: 11.4 percent. Put another way, nearly nine of every ten visitors in that arm never touched the video. To know how much of the gain runs through the video itself you would have to compare the conversion rate of those who pressed play against those who did not, which the aggregate number does not answer. If the effect is real, a good part of it may come from the poster and the button, which change the look of the first screenful and signal “there is an explanation here”, rather than from the video’s content. That is the kind of hypothesis that earns the next test, and it is cheap: not a fake play button, which is deceiving the visitor, but an image with an explicit invitation to the explanation.

Platform embeds: what you stop controlling

The fastest way to put video on a page is to embed a video platform’s player. It is also the one that most confounds a test, because you take on third-party decisions inside your own experiment.

Four concrete consequences, all verifiable on the page itself before you launch:

1. Weight and requests you did not choose. The player brings its own code, its own fonts and its own telemetry. In a test where the control arm loads none of that, the performance gap between arms is not “the video”, it is “the video plus the player”.
2. Cookies and consent. Many players write identifiers on first load. Depending on your consent banner, that means the video arm may be blocked until the visitor accepts, and part of your conversion data behaves differently between arms. That is silent confounding, of the kind described in tracking loss from consent and ad blockers.
3. Elements the platform decides. End-screen suggestions for other videos, service branding, share buttons, links away from your page. A video that ends by offering three of somebody else’s videos is an attention leak your control arm does not have.
4. Changes outside your calendar. The platform updates the player whenever it wants, including in the middle of your test. If a behavior changes on day 20 of a 55-day test, the two arms are no longer comparable to what they were on day 1.

Hosting the file yourself costs work and bandwidth but hands the four decisions back. The middle ground that usually resolves it: a light poster image native to your page, with the third-party player loaded only after the click. You keep the first screenful under control, keep LCP low, and still offer the video to whoever wants it. That is exactly arm C in this guide’s example, and it is why arm C is in the test.

Which question does the landing page video answer, and when is it the wrong answer

Video is expensive to produce, expensive to load and expensive to keep current. Before testing, check whether the pain it solves exists on your page.

visitor’s pain	does video help?	cheaper alternative that usually ties
“I did not understand what this does”	yes, a 45 to 90 second demo shows what copy describes badly	a block of three annotated screenshots with captions
“I am not sure it fits my case”	not directly, a generic video does not speak to their case	a named use-case section, in text
“it looks hard to install”	yes, showing the real screen beats promising simplicity	a short clip or a sequence of screenshots
“I do not trust this company”	partly, if it is a testimonial from an identifiable customer	social proof in text, with name, company and a number
“how much does it cost?”	no	show the price
“I want to compare it with a competitor”	no	an honest comparison table

The “how much does it cost” row is there because it is the most common case of video used as a curtain. When the page does not answer the main objection, adding media does not solve it, it postpones it. If a video test ties, the next hypothesis is rarely “let’s produce a better video” and almost always “the objection blocking the decision is a different one”.

Length. There is no universal number, but there is a useful asymmetry: the cost of a video that is too long is high and the cost of one that is too short is low. A 45-second video that fails to convince leaves the visitor with the copy; a 6-minute one that fails has consumed all the attention with nothing left over. If you get one production budget, make the short one. With two, the more informative design is not “short against long” on the same page: it is short above the fold with the long one linked just below for whoever wants it, which is the same layering logic we argue for in long copy vs short copy.

Captions are not an accessory. Because web video starts muted by browser default and by user habit, an uncaptioned video communicates almost nothing in the first seconds, which are exactly the seconds that decide whether anyone continues. Reviewing the video with sound on, on the team’s desktop, hides that problem completely. Always review it muted, on a phone, with the whole page around it.

How to read the result without fooling yourself

1. Was LCP measured in both arms? Without it, a negative result may be performance rather than content.
2. Was the threshold corrected for the number of comparisons? Three arms means two comparisons against control.
3. Is play rate being used as the verdict? If so, redo the reading on conversion.
4. Did the video push content off the first screenful? If it did, the test also changed hierarchy, not just media.
5. Was the video version tested on a real phone? Video is where the desktop and mobile gap shows up most, in screen and in network.
6. Are there captions? Since nearly every web video starts muted, an uncaptioned video says far less than the team assumes when reviewing it with sound.
7. Is the video loaded from a third party? If so, you added an external dependency to the page, with effects on performance and cookie consent; see tracking loss from consent.
8. Was the first week stronger? A new video draws attention for being new; see the novelty effect.

Pre-launch checklist

1. Separate the four decisions and pick which one is the test variable. Testing all four together answers a different question.
2. Declare LCP as a guardrail, with the target value, before you launch.
3. If it autoplays, keep it muted, short and pausable, per WCAG 1.4.2 and 2.2.2.
4. If it is click to play, make sure the file is not downloaded before the click. Light poster, video on demand.
5. Caption the video. Muted is the web’s default state.
6. Track play rate and percent watched as declared diagnostics, never as the verdict.
7. Correct the threshold whenever there are more than two variants, and record the correction in the plan.
8. Check the visitor split with the SRM checker before you look at the result.

Automate this with Donnu

The specific pain of testing video is that the change touches content and performance at the same time, the good design needs three arms, and the most visible metric, play rate, is precisely the one that decides nothing.

In Donnu, an experiment can carry a variant C on the Pro plan, which is what lets you separate autoplay from click-to-play inside one test instead of running two tests back to back. The conversion goal can be a click, a form submission, a page visit or a confirmation from your own server, which keeps the verdict on real conversion. The report is Bayesian, flags when the visitor split drifts from what was configured, and only calls a winner with at least 200 visitors per variant and 7 days of data.

What stays with you: measuring LCP in both arms and deciding whether the video deserves to exist. For the rest, the sample size calculator, the significance calculator and the page speed conversion impact calculator run every number in this guide on your own data, free.

References

• web.dev (Google). Largest Contentful Paint (LCP). Documentation read in full. Source for the list of elements considered by the metric, including the video element via the poster image load time or the first frame presentation time, whichever is earlier, and for the 2.5 second target measured at the 75th percentile of page loads, segmented across mobile and desktop. Checked on September 21, 2026. web.dev.
• Chrome for Developers (Google). Autoplay policy in Chrome. Post read in full. Source for the rules in force since Chrome 66 (April 2018) and Chrome 71 for the Web Audio API: muted autoplay always allowed; with sound only after prior interaction with the domain, a crossed Media Engagement Index threshold on desktop, the site added to the home screen or installed, or autoplay permission delegated by the top frame to an iframe; and for the promise rejection with NotAllowedError. Checked on September 21, 2026. developer.chrome.com.
• W3C Web Accessibility Initiative. Understanding Success Criterion 1.4.2: Audio Control and Understanding Success Criterion 2.2.2: Pause, Stop, Hide, WCAG 2.2, both Level A. Pages read in full. Source for the 3-second rule on automatic audio and the 5-second rule on moving information presented in parallel with other content. Checked on September 21, 2026. w3.org (1.4.2) · w3.org (2.2.2).
• On the 80 to 86 percent figure attributed to EyeView: the claim circulates in agency articles and video vendor posts without a public original report carrying methodology and sample. We do not use it as fact in this guide, and we recommend the same treatment for any conversion statistic whose primary source cannot be retrieved.

Read next: Conversion rate optimization · Page speed and conversion · Long copy vs short copy · Social proof A/B testing · B2B SaaS landing page A/B testing · A/B/n testing · Guardrail metrics · Leia em português