How should we design a geo holdout test for connected TV?
Our answer
Turn connected TV off in matched market areas with the platform's location exclusions. Run one unchanged prospecting campaign for 4 to 6 weeks plus the 2 to 4 week window MNTN's Haus guidance sets. Then read your own total orders.
- Confidence
- Medium
- Strongest proof
- 3 official rules
- 4 datasets, 3 practitioners, 4 opinions or claims
- Sources
- 10 sources
- 14 statements
M
R
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentMatched markets beat a random split. Measured (incrementality testing firm) builds its connected TV holdouts from media markets matched on past orders, turns ads off in test markets that usually hold 7 to 40% of national sales and reads the gap in orders.
Hold everything else steady. MNTN's guidance for Haus geo tests requires one unchanged prospecting campaign, steady spend elsewhere and no peak season. It sets 4 to 6 weeks of ads, a 2 to 4 week window after and a power score of at least 90%.
Location by IP address often misses, so control markets can leak. Only 23% of residential IP addresses reached their intended geographic target in Adstra and InterMedia's study (PPC Land). Prescient AI warns that geo tests can be contaminated between test and control markets.
Our judgment
Build the test from market areas or states rather than ZIP code lists, because location by IP address often misses its target and would put exposed homes in the control group. Treat a platform's own household holdout as a cross-check beside the geo test, not the verdict, because the seller designs and reports it. That follows Measured's view of platform lift studies as a directional signal and MNTN's description of its 10% holdout.
Our judgment, built from PPC Land, Prescient AI, Measured and MNTN: our reading of these sources, not a finding from any one of them.
What the sources say
14 statements from 10 sourcesTinuiti, an agency that sells TV buying and measurement, says platform attribution can overstate streaming TV results, so judge it on geo holdout, matched-market or lift tests planned before launch.
“Platform attribution alone can't establish whether advertising generated additional business.”
Measured's CTV panel, over $25M of spend, found streaming about 75% more incremental than linear TV and YouTube CTV and video at 1.6x the category benchmark. Small-budget CTV tests often showed no measurable lift.
“the platform is rarely the variable that decides the outcome”
Jetfuel, an agency, says Meta stays more efficient and CTV earns a test once social and search stop scaling. A clean 60 to 90 day test usually needs a five-figure budget, read with a holdout or geo test.
“Meta stays the more efficient direct-response channel. CTV earns a test once social and search stop scaling.”
CTC's Luke Austin weighs a new channel's staff and measurement cost against the 2 to 5% of mix it will likely hold, and says cheaper geo testing now justifies $15 to 20K channel tests.
Measured, which sells geo testing, says TV and CTV holdouts should pick test markets by algorithm, not at random, from about 210 US markets, turn TV off there, and typically run four to eight weeks.
“Random assignment does not work here: with only around 210 media markets in the US, random splits produce structurally imbalanced groups”
Measured says that in one four-week geo holdout of a retailer's streaming TV, roughly half the conversions the platform claimed were incremental. It says TV tests typically run four to eight weeks.
“most streaming platforms support geographic targeting exclusions, so the holdout can be implemented directly in the buying platform.”
PPC Land reports Jamloop's CTV household holdout and Recast geo tests. It notes a 10% holdout can use about 10% of budget, so advertisers often hold out only 5 to 10%. Jamloop's result is one interim read.
“CTV is a natural fit for geographic-based testing”
Pushes back on geo tests: Prescient AI says they suffer contamination between markets and show only one window, and that its own mix model shows Tatari TV campaigns reporting $13 where it credits $28,053.
“None of these methods works best in isolation. The strongest approach uses one to check the other.”
PPC Land reports a study by identity sellers Adstra and InterMedia: only 23% of residential IP addresses reached their intended geographic target, and device IDs were 71% consistent month to month versus 57% for IPs.
“found that just 23 percent of residential IP addresses reached their intended geographic target”
Eightx, a CFO firm, says CTV pays for DTC beauty brands only once Meta and Google saturate and a measurement plan is set. Below $15M revenue it rarely works, and a 90-day test costs $90K to $200K.
“Decide on a measurement approach before the first dollar of media.”
Adwave, which sells self-serve TV ads, puts a first local CTV test at $800 to $1,500 a month for 8 weeks in one market, judged against a pre and post baseline after 60 to 90 days.
“Four weeks of pre-launch data is the minimum. Six weeks is better.”
MNTN's guide for Haus geo tests of CTV: run 4-6 weeks plus a 2-4 week post-treatment window, aim for a Power Score of at least 90%, keep other spend steady, and avoid lift tests on retargeting audiences.
“Avoid running lift tests for retargeting audiences.”
MNTN's incrementality reporting, for prospecting campaigns with its pixel, holds back ads from 10% of the targeted audience and subtracts that group's scaled results from the exposed group's.
“MNTN automatically holds back ads from 10% of your targeted audience to form a control group.”
Roku Ads Manager can include or exclude states, ZIP codes or market areas, one location type per audience, and ZIP targeting is unavailable for credit, housing or employment ads.
“You can only include or exclude locations using 1 geographic type (states, ZIP Codes, or market areas).”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer