How often should a brand our size run incrementality tests?
Our answer
Plan incrementality tests every quarter once a channel can be read. Test where a budget decision is pending or evidence is oldest, and retest after big changes. Below about $40K a month per channel, skip formal tests.
- Confidence
- Medium
- Strongest proof
- 2 datasets
- 5 practitioners, 11 opinions or claims
- Sources
- 15 sources
- 18 statements
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentA quarterly rhythm is the common advice. Meta recommends single-cell lift studies every quarter (PPC Land). Flighted, a paid media agency, calls quarterly reasonable at $100K a month or more, and Sellforte puts long tests in a quarterly planning session.
Big changes call for a retest. Rockerbox and Haus advise testing again when a channel's spend, buying strategy, media mix, competition or brand strength moves.
More tests are not better when they are noisy. Haus's simulation found 15 noisy tests a year did no better than 6. Elevarus, a paid media agency, skips formal tests below about $40K a month per channel.
Our judgment
Plan tests quarterly and let pending budget decisions and big changes pick the channel. That joins the quarterly rhythm from Meta, Flighted and Sellforte with the retest triggers from Rockerbox and Haus. Favor fewer well-powered tests that don't overlap over many small ones, following Haus's simulation and Sellforte's advice to avoid tests that touch the same audience or regions. The 6 to 12 tests a year that Eightx attributes to Recast stays Eightx's reading, not our answer, because it is not Recast's own words.
Our judgment, built from PPC Land, Flighted, Sellforte, Rockerbox, Haus and Eightx: our reading of these sources, not a finding from any one of them.
What the sources say
18 statements from 15 sourcesPPC Land reports that Meta's peak season guide recommends single-cell Conversion Lift studies to calibrate your measurement each quarter.
“to calibrate your measurement model on a quarterly basis.”
Haus says about half of YouTube tests are built to detect lifts under 4%, and brands that missed a benchmark on a first test beat it on the retest 44% of the time, so one test is not a verdict.
“one test is a read, not a verdict.”
Rockerbox, which sells attribution and testing, says to test growing channels and likely low-incrementality tactics such as remarketing and brand search, and to retest when spend or strategy changes a lot.
“The amount you spend on the channel has increased significantly”
Neato, which runs brands' Amazon sales, describes geo, on-off and audience holdout tests for retail media, says branded ads have low incrementality, and advises retesting quarterly. Its ROAS ranges cite no study.
“branded keyword ads have low incrementality because the customer was already searching for your brand.”
Velocity Sellers, an Amazon agency, says branded campaign ROAS misleads, and advises a holdout of at least four weeks read on total branded purchases. Its 10.8x example is hypothetical.
“The ROAS on a branded campaign is the least informative number in your ad console”
Stackmatix, a paid media agency for startups, says to run a test when a decision hinges on it, especially before scaling or cutting a channel, typically two to four tests a quarter.
“Run a test whenever a decision hinges on the answer, not on a fixed calendar.”
Sellforte, which sells testing, says no fixed spend makes a channel too small to test; it is too small when no feasible design can detect an effect that would change the budget call. Then use broader evidence.
“A full pause in selected units produces more signal than a shallow cut everywhere”
Sellforte, which sells testing and mix models, says to pick tests by impact, uncertainty, feasibility and cost: plan slow geo tests quarterly and run faster platform studies monthly.
“Testing the largest channel every quarter can repeat what the team already knows.”
Silverback Strategies says lead gen brands should fund search to saturation first, then spend the next dollar on demand creation proven with lift tests of four to eight weeks, which small sellers often cannot run.
“Once demand capture is saturated, additional budget there produces diminishing returns, and the next dollar is better spent creating demand.”
Haus simulated a year of test-driven budget moves 36 million times: precise tests beat doing nothing 82% of the time vs 62% for noisy ones, and 15 noisy tests a year did no better than 6.
“The precise scenarios beat a frozen-budget baseline 82% of the time. The noisy ones, only 62%.”
Flighted, a Meta agency, says to retest incrementality periodically: quarterly for accounts spending $100K+ a month, twice a year for smaller ones. It gives no test data.
“What was incremental six months ago may not be incremental today.”
Trackingplan, which sells tracking monitoring, says to update incrementality multipliers at least quarterly by channel and audience, since one company-wide multiplier misallocates budget.
“A multiplier calibrated in Q1 will be wrong by Q4 if market conditions have shifted.”
An Operators podcast host says a brand doing about $5M a year that runs only conversion-optimized Meta campaigns needs neither incrementality testing nor multi-touch attribution tools.
“You don't need incrementality and you do not need MTA tools.”
Eightx, relaying vendors, says Recast recommends 6 to 12 tests a year and Haus runs always-on geo testing, and that a standalone test ages out in about 90 days.
“Standalone tests age out in roughly 90 days as creative, seasonality, and competitor spend move.”
Eightx sketches incrementality testing by DTC revenue: rare under $5M, emerging at $5M to $20M, common at $20M to $50M, and always-on above $150M. It calls this directional, not a survey.
“Directional pattern based on Haus, Recast, and Northbeam public commentary plus Eightx founder CFO calls; not a representative survey.”
Elevarus, an agency, says that above about $80K a month per channel you should run 4 to 6 week geo holdouts for incremental ROAS; below about $40K, use Meta's lift tool or skip formal testing.
“Use a geo holdout when you need to test cross-channel effects, when offline conversions are part of the outcome, or when the platform-native option doesn’t cover the campaign type.”
CTC says it runs always-on Meta conversion lift tests comparing settings head to head, such as incremental attribution versus 7-day click, alongside third-party geo holdouts and mix models.
An IAB survey of 430 US buy-side decision-makers found those who test regularly most often run 3 to 5 incrementality tests a year, and expect AI to push that to 11 or more.
“from 3-5 tests per year to 11+ annually”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- What the research saysWhat academic research adds, kept apart from the answer
- How sure we areWhat its confidence rests on, and what would change the answer