Incrementality Testing
Incrementality testing is a controlled experiment that measures how many conversions marketing actually caused, rather than how many happened to follow an ad. It works by withholding the campaign from a randomly selected control group, then comparing that group’s conversions against the group that saw the ads. The difference between the two is the incremental result, and everything else would have happened anyway.
Why Incrementality Testing Matters
Attribution reports credit, not cause. A customer who was already going to buy still clicks the retargeting ad on the way to checkout, and the platform counts that conversion as advertising-driven. The revenue would have arrived without the ad.
That gap is measurable and often large. In a large-scale field experiment at eBay, researchers Thomas Blake, Chris Nosko, and Steven Tadelis found that returns to branded search advertising were close to zero for existing customers, who reached the site through organic results when the ads were switched off (Econometrica, 2015).
Two channels distort budgets most predictably. Branded search and retargeting both sit near the end of the purchase path, so they collect credit for demand created upstream. Incrementality testing is the only method that separates the two.
Types of Incrementality Tests
Five designs dominate, and they differ in what they randomize.
- User-level holdout (conversion lift). Randomly split the audience, suppress ads for the control group, compare conversion rates. Meta Conversion Lift and Google Ads Conversion Lift run this design natively.
- Geo experiments. Hold out entire regions instead of individuals. Matched control markets go dark while test markets run the campaign. Works on aggregate sales data, so no user-level tracking is required. Meta released GeoLift as an open-source R package for this design.
- Ghost ads. The control group is logged at the moment it would have won an auction, but no ad is served. This removes the cost of showing a placeholder. The method was formalized by Garrett Johnson, Randall Lewis, and Elmar Nubbemeyer (Journal of Marketing Research, 2017).
- PSA tests. The control group sees an unrelated public service announcement instead of the campaign. Older and simpler than ghost ads, but the placeholder impression still costs money.
- On-off tests. Turn a channel off entirely for a defined period and model the counterfactual from historical data. Google’s open-source CausalImpact package handles this time-series case. Cheapest to run, weakest evidence, because seasonality and other campaigns are not controlled.
How to Calculate Incremental Lift
Incremental conversions are the test group result minus what the control group produced at the same scale.
Incremental conversions = Test conversions - (Control conversions x Scaling factor)
Lift % = (Test conversion rate - Control conversion rate) / Control conversion rate
iROAS = Incremental revenue / Ad spend
A worked example. A test group of 100,000 users converts 2,400 times. A control group of 100,000 users converts 2,000 times. Incremental conversions are 400, and lift is 20%. If the campaign cost $8,000, the incremental cost per acquisition is $20, not the $3.33 the platform reports against all 2,400 conversions.
The scaling factor matters when groups are unequal. A 10% holdout compared against a 90% test group needs the control result multiplied by nine before subtraction.
How to Run an Incrementality Test
- Pick one question. A test measures one channel, campaign, or tactic at a time. Testing several changes at once produces a result nobody can attribute.
- Size the holdout. Control groups commonly run between 10% and 20% of the audience. Smaller holdouts protect revenue but need longer runtimes to reach significance.
- Run a power calculation first. Estimate the minimum detectable effect from baseline conversion volume before launching. Tests that cannot detect the expected lift waste the entire runtime.
- Randomize, do not segment. Assignment must be random. Splitting by geography, device, or customer tier introduces differences that look like lift.
- Hold the test period fixed. Two to four weeks is typical for digital conversions. Longer is required where the purchase cycle is long.
- Do not stop early on a good result. Checking daily and stopping at the first significant reading inflates false positives.
- Tag consistently across both groups. Analysis depends on identifying which campaign each session came from, so both arms need identical parameter conventions. Modeled credit from those tagged sessions is covered in the campaign attribution guide.
Incrementality Testing vs Attribution
The two answer different questions and disagree by design.
| Attribution | Incrementality testing | |
|---|---|---|
| Question answered | Which touchpoints preceded the conversion | Which spend caused the conversion |
| Method | Observational modeling | Randomized experiment |
| Output | Credit split across channels | Causal lift and iROAS |
| Runs | Continuously | In discrete test windows |
| Cost | Reporting overhead | Real revenue withheld from the control group |
Attribution is the daily operating report. Incrementality testing is the periodic audit that tells you how far the daily report has drifted from reality.
Frequently Asked Questions
What is incrementality in marketing?
Incrementality is the share of a result that would not have occurred without the marketing. A campaign generating 1,000 conversions where 700 would have happened regardless has an incrementality of 30%, or 300 incremental conversions. The term describes causal contribution, which is why it is measured through experiments rather than reported from platform dashboards.
What is a holdout test?
A holdout test is an incrementality test where a randomly selected portion of the audience is deliberately excluded from a campaign. That excluded group becomes the control. Because assignment is random, the only systematic difference between the groups is ad exposure, so the conversion gap between them is causal rather than correlational.
How long should an incrementality test run?
Two to four weeks suits most digital conversion tests. The real constraint is statistical power, not the calendar: the test needs enough conversions in both arms to detect the expected lift at 90% or 95% confidence. Businesses with long consideration cycles need runtimes that cover the full purchase window, otherwise late conversions land outside the measurement period.
Why do incrementality results differ from platform-reported conversions?
Platforms count conversions that followed an ad view or click within their attribution window, including conversions that would have happened anyway. Incrementality tests subtract that baseline. Retargeting and branded search show the widest gaps, because both reach people who already intended to buy.
Is incrementality testing the same as A/B testing?
Both randomize users into groups, but they test different things. A/B testing compares two versions of an experience, such as two landing pages, and both groups receive marketing. Incrementality testing compares marketing against no marketing, which makes the absence of the ad the variable being measured.
To keep both test arms identifiable in your analytics, tag every campaign link consistently with the free UTM builder at linkutm.