Incrementality testing: geo experiments, conversion lift and holdouts

0:00 Incrementality testing

Every metric we've discussed so far is, in the end, a proxy. There's only one question that really matters: how much more did you sell because of this advertising, compared with what would have happened without it? That's incrementality. In this lecture you'll learn the three main test designs, how to design a geo experiment step by step, why power analysis saves you from wasted tests, and how to turn results into decisions.

0:32 Why it matters

Why does this matter? Because without incrementality, you can't tell the difference between advertising that creates sales and advertising that just stands next to sales. Here's an analogy. A rooster crows every morning, and every morning the sun rises. If you only looked at the correlation, you'd credit the rooster. Plenty of ad spend is a rooster: it shows up right before people buy, but they'd have bought anyway. An incrementality test is simply putting the rooster in another room for a few days and seeing if the sun still rises.

1:12 User-level lift

Design one: user-level lift. The platform randomly holds back ads from a control group and compares conversion rates. It's properly randomized and easy to run. The catch is that the platform runs and grades its own test, you need enough conversions, and it covers one platform at a time. Meta and Google both offer Conversion Lift, and Google has moved to a Bayesian method that allows smaller studies than before. Check current eligibility in your account.

1:45 Geo experiments and holdouts

Design two: geo experiments. You split regions into treatment, where ads run or increase, and control, where they're off or unchanged. Then you compare your own sales by region. The big advantage: it uses your backend data, so it captures offline sales and works across channels. The challenges: you need enough regions with stable data, and people move between regions. Design three: holdouts, where part of your customer list or a region gets no ads for a longer period. Great for retargeting and email.

2:22 Geo test, part one

Let's design a geo test. Start with a crisp question, like: does our YouTube spend increase total orders in the UK? Choose your units, maybe postcode areas in the UK, emirates or city zones in the Gulf, cities in Pakistan. Gather eight to twelve weeks or more of history to confirm treatment and control move together. Then use a tool, not gut feel, to select matched markets. Meta's open-source GeoLift does this.

2:53 Simple example: Lahore grocer holdout (illustrative)

Let's walk a simple worked example. An online grocer in Lahore wonders if its retargeting ads help. They create a holdout: ten percent of recent site visitors, chosen at random, are excluded from retargeting for four weeks. At the end, the retargeted group bought at a rate of eight percent, the holdout at seven point five percent. So only half a percentage point is incremental. If retargeting cost fifty thousand rupees and the extra purchases earned about the same in margin, it's barely breaking even, even though the platform reported a lovely return. These numbers are illustrative, but holdouts like this regularly reveal it.

3:38 Geo test, part two

Part two. Make the treatment meaningful. Tiny spend changes produce effects too small to see. Run long enough to capture the lag between seeing an ad and buying, often three to six weeks, plus a cool-down. Measure backend orders or revenue by region, never platform conversions. Analyze with a synthetic control or difference-in-differences, report the lift with an interval, and calculate incremental cost per acquisition and incremental return on ad spend.

4:09 Power analysis

Now power, the step people skip. Before you spend a single dirham, ask: if the true lift were five percent, would this design detect it? If the answer is no, you're about to buy an inconclusive result. Fix it by making a bigger spend change, adding regions, or running longer. Or consciously accept a directional result. GeoLift includes power analysis, and the lesson has a skeleton in R to get you started.

4:40 Example: UAE electronics (illustrative)

Here's an illustrative example. A UAE electronics retailer asked whether Meta prospecting adds sales beyond Google Search. They matched zones by delivery postcode across Dubai and Abu Dhabi, paused Meta prospecting in control zones for four weeks, and measured backend orders by zone. The incremental cost per acquisition came out about one point four times the platform-reported figure, giving an incrementality factor of roughly zero point seven. Meta stayed, but with a revised target and a Conversion Lift study scheduled to cross-check.

5:16 Four mistakes

Avoid four classic mistakes. Using platform conversions as your geo test outcome. Too few regions or too small a change. Running promotions or price changes in only some regions, which contaminates the test. And stopping early because the result looks good. Build a test calendar instead, with at least one causal test per major channel every six months, and store every result with its interval.

5:44 Mistakes + try this now

Common incrementality mistakes. Using platform-reported conversions as the outcome in a geo test. Choosing test regions by gut feel instead of matching them. Running a promotion in only some regions during the test. Making the spend change so small that no effect could ever be detected. And stopping as soon as the result looks nice. Try this now: pick one channel you're unsure about and write a one-paragraph test design: the question, treatment and control, the outcome from your backend, the duration, and your decision rule.

6:21 Watch me do it: YouTube geo test (illustrative)

Watch me do it. Let's design an illustrative geo test for a UK online bakery on YouTube. Question: does YouTube Demand Gen increase total orders in England? Step one, units: I group delivery postcodes into forty areas with stable weekly orders. Step two, history: twelve weeks of orders by area. Step three, market selection: I run GeoLift's market selection, asking for treatment groups of five areas, a twenty-eight-day test, and our budget. It returns candidate sets with their power: the best set can detect roughly a five percent lift. Step four, treatment: YouTube runs only in the five treatment areas, at a meaningful budget, while everything else stays the same everywhere. Step five, contamination check: the promotions calendar shows a discount planned nationally in week three. That's fine because it applies to all areas equally. Step six, decision rule, written before launch: if incremental cost per order is below our target, scale YouTube nationally; if above, cut back; if inconclusive, repeat with a larger budget. Step seven, after the test, analyze with GeoLift and report the lift with its interval, not just a single number.

7:41 Recap and next step

Recap. Incrementality is the question that matters. Use lift studies, geo tests and holdouts for their strengths. Design geo tests with matched markets, meaningful treatment, backend outcomes and power analysis. Report incremental CPA or iROAS with intervals and turn them into incrementality factors. Your next step: design a geo test for one channel, including the question, regions, treatment, duration, metric and the minimum effect you need to detect.