Incrementality tests you can actually run as a small business
“Would these sales have happened anyway?”
Retargeting almost always reports a spectacular return. It is shown to people who were already going to buy. The question incrementality answers is not “how many conversions did this campaign report” but “how many would not have happened without it” — and you do not need a research budget to get a usable answer.
Design 1: the pause test
The bluntest and most honest. Turn a campaign off entirely for a defined period and watch total business revenue, not the campaign's own numbers.
- 1Pick a period of at least two full weeks with no seasonal distortion.
- 2Pause the campaign completely. Do not reduce budget — partial changes give ambiguous results.
- 3Track total orders and total gross profit for the business, not the platform's reported conversions.
- 4Compare against the equivalent prior period, adjusted for any trend the rest of the account shows.
Design 2: the geo holdout
Split by geography instead of by time, which removes the seasonality problem. Exclude one or more regions from a campaign, keep everything else identical, and compare per-capita order rates.
1 − (order rate in holdout ÷ order rate in exposed)
The held-out region converts at 82% of the exposed region's rate. Only 18% of that campaign's reported conversions were caused by it.
This needs enough volume per region to see past noise. As a rough guide, if a region produces fewer than about 50 orders over the test window, the result will not be distinguishable from randomness.
Design 3: the staggered rollout
If you cannot afford to pause and cannot split by geography, introduce a new campaign to half your regions first and the rest two weeks later. The gap between the two is your read, and you never have to switch anything off.
How to avoid fooling yourself
- Decide what result would change your decision before you start. If no result would, do not run the test.
- Write down the expected effect size first. “It'll go down a bit” is not falsifiable.
- Never test during a promotion, a holiday, or a PR moment.
- One test proves little. A repeated result proves a lot.
- Expect an uncomfortable answer from retargeting and brand search. That discomfort is the value.
In short
- ✓Measure total business revenue, never the campaign's own reported conversions.
- ✓Pause tests are blunt and honest; geo holdouts remove seasonality; staggered rollouts avoid switching anything off.
- ✓As a rough guide, below about 50 orders per cell you are mostly reading noise.
- ✓Decide in advance what result would change your mind.
Where this method runs out
Everything above works in a spreadsheet. Keeping it current, and matching every order back to the ad that actually caused it, is the part that does not. That is what Kepra does — and the demo runs on sample data with no signup, so you can judge it before believing any of this.
Open the demo →Read next
Scaling without killing it: why average ROAS can’t answer “spend more?”
Average ROAS describes money you already spent. The decision in front of you is about the next krone. Here is how to measure marginal return and find the point where scaling stops paying.
Why Meta and Google both claim the same sale
Add up the conversions your platforms report and you will find more sales than you made. Here is exactly where the double-counting comes from, and what to do about it without buying anything.