How do we measure the added sales from a channel or offer we can't hold out by region or customer, such as national TV or a sitewide sale?
Our answer
Run the buy in a fixed set of markets first and read total sales against a synthetic control. When it reaches every market at once, forecast sales from your own history and treat the gap as a rough read.
- Confidence
- Medium
- Strongest proof
- 1 controlled test
- 3 official rules, 3 datasets, 6 opinions or claims
- Sources
- 9 sources
- 13 statements
G
CTOIAL
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentRunning a buy in fixed markets first gives a clean read. Haus read Skylight's linear TV, run as a local plan in 7 markets, against a synthetic control built from unaffected regions. Site sales rose 5.3% and sales across all channels rose 8.6%.
A read from a sales forecast is rougher. Haus calls its forecast-based Time Tests more directional than geo tests. INCRMNTAL found about 0.6% of a betting client's Super Bowl conversions were incremental, even as deposits about tripled.
Both a market read and a forecast lean on long, steady history. Google's GeoX requires at least 3 times the test length in daily history before a geo test. Measured builds its TV prediction from about two years of sales in untested markets.
Our judgment
- Keep the forecast for buys that truly reach every market at once. Haus calls its Time Tests more directional than geo tests, so a market-level run comes first when you have the choice.
- Apply Google's geo test rule on length of history to forecast reads. Before a forecast read, have at least 3 times the read's length of steady daily history, and a year or more when sales are strongly seasonal.
- Read the result on total sales. Count sales in every outlet rather than attributed conversions.
- Build more reads into the plan by starting and stopping flights. Plan flights with starts and stops so each gives another read.
Our judgment, built from Haus, Measured, Jetfuel Agency, Google, Soonhong Cho (arXiv 2608.20610), INCRMNTAL and CMSWire: our reading of these sources, not a finding from any one of them.
What the sources say
13 statements from 9 sourcesGoogle's Meridian mix model docs recommend geo-level data over national, imputing national-only media by population share if needed, and modeling the top 50 to 100 DMAs for US advertisers.
“Geo-level data offers several advantages and is thus recommended when possible.”
CMSWire describes how Netflix measures TV and other channels it cannot randomize: each launch, takedown and relaunch is a separate market-level read. It gives no lift figures.
“A six-week holdout measures only a six-week effect.”
Google's GeoX setup needs daily pre-test history by geo of at least three times the test length, and a year or more where seasonality is strong.
“you'll provide a minimum of 3*N days worth of response data, such as revenue or conversion counts, where N is equals to the duration of the actual test period of your experiment”
Measured, which sells geo testing, says TV and CTV holdouts should pick test markets by algorithm, not at random, from about 210 US markets, turn TV off there, and typically run four to eight weeks.
“Random assignment does not work here: with only around 210 media markets in the US, random splits produce structurally imbalanced groups”
A UCLA working paper simulates forecasting a counterfactual from a unit's own history when a change hits every unit at once. Estimates improve with longer pre-periods; it studies policy data, not ads.
“Point estimation improves with the pre-treatment record for every method”
Jetfuel, an agency that audits AppLovin accounts, tests AppLovin with a 14 to 21 day geo holdout against a synthetic control, holding spend steady, and says an inconclusive result is not a negative one.
“an inconclusive result is not the same as a negative one”
Google says a mix model with 26 parameters on two years of weekly national data is too little to estimate reliably; fewer channels, more years or geo-level data help. It calls the guidance rough.
“This sample size is too low to estimate the model reliably.”
Talon.One, which sells promotion software, says redemption rates do not show whether a coupon changed behavior. Hold the offer back from 10 to 20% of the audience and track sales 30 to 180 days after it ends.
“A promotion generating 30% sales lift may represent zero net new revenue if customers bought earlier and then went quiet.”
Atticus Li, a CRO consultant, says before/after SEO comparisons are unreliable because of seasonality and algorithm changes; split pages into test and control groups, or use causal impact models for sitewide changes.
“It is also deeply unreliable.”
INCRMNTAL, which sells incrementality measurement, studied 8 brands around Super Bowl LX. A betting client's deposits about tripled, but only about 0.6% of event-window conversions were incremental.
“But the marginal incrementality view told a different story: during the event window, only about 0.6% of those conversions were estimated to be incremental to paid spend.”
Haus, which sells these tests, says Phoenix's Super Bowl spot lifted signups 26% in one week in a Time Test, a forecast with no control group that Haus calls more directional. One brand.
“the Super Bowl spot drove a 26% lift in signups in just one week”
Haus, which sells these tests, describes Time Tests: forecast a KPI without one big national change and read the gap as lift. It calls them more directional than geo tests and gives no example results.
“are best for national, high‑impact campaigns when you cannot split your marketing into test and control groups.”
In a five-week Haus geo test of local linear TV in 7 markets against a synthetic control, Skylight's total orders rose 8.6%: 5.3% on its site and 11.2% on Amazon. Haus sells this test.
“Haus built a synthetic control group from the unaffected regions”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer