How long should a test on a smaller channel run before deciding?
Our answer
Run a test on a smaller channel at least four weeks before deciding. Make no edits during the first two weeks of learning, then read weekly, and go longer when orders are few or a holdout reads it.
- Confidence
- Medium
- Strongest proof
- 5 official rules
- 1 dataset, 11 practitioners
- Sources
- 12 sources
- 17 statements
- P
RBP
T
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentShort tests understate lift. Measured (incrementality testing firm) advises that four to eight weeks is the common range and almost no test should run under four, because ending early misses late conversions.
Platforms need about two weeks to learn. Pinterest says learning takes about two weeks on average and edits can reset it. Reddit says to wait 7 to 14 days to see results.
Fewer orders mean longer tests. Measured's illustrative example needs about six weeks at 5,000 orders a week and past 20 weeks at 1,250. Elevarus (paid media agency) waits for 60 to 90 conversions before judging Reddit against Meta.
Our judgment
- Hold every smaller-channel test at least four weeks before deciding. Measured says almost no test should run under four weeks, and the platforms' learning periods take up about the first two.
- Make no budget, bid or targeting edits for about the first two weeks. Pinterest says changes during learning can reset it, and Black Propeller makes no significant changes in a new campaign's first seven days. Where an edit can't wait, Pinterest suggests no more than two rounds a week, with three to five days after each.
- Use two to four weeks as a delivery check, not the decision date. That is the window Trackbee and Adwave give, while Pinterest and Reddit put learning and first results at up to two weeks and Black Propeller waits at least two weeks before drawing conclusions.
- Plan three months or more only where a flight or a ramp sets the pace, such as connected TV and podcasts. Simple Pin Media's three months is its own minimum for taking a client, not a test length.
Our judgment, built from Measured, Pinterest, Reddit, Enrich Labs, Trackbee, Adwave, Black Propeller, Simple Pin Media, Jetfuel, Podder and DataLatte: our reading of these sources, not a finding from any one of them.
What the sources say
17 statements from 12 sourcesPinterest says changing a campaign in Learning mode may reset it, the Learning indicator takes about two weeks on average to clear, and campaigns are best judged weekly or monthly, not daily.
“If you make changes to your campaign while it's in Learning mode, it might reset and take longer.”
Pinterest says 50 to 200 conversions a week is usually enough for its system to learn, and advises no more than two rounds of changes a week, with three to five days to recalibrate after each.
“Typically we see that 50-200 conversions per week is sufficient for our delivery system to learn who it's best to show your ads to.”
Reddit tells Dynamic Product Ads advertisers to spend at least $100 a day, wait 7 to 14 days for results, use lowest-cost bidding and have at least 6 products in the catalog.
“Plan to spend at least $100 per day. Wait 7-14 days to start to see results.”
Pinterest's campaign optimization score runs 0 to 100 and starts at 90 or more by default; Pinterest says it is no guarantee, and a validly tested setup that beats it need not change.
“is not a guarantee of improved performance.”
Enrich Labs, relaying AdLiftr data on 2,431 Snapchat campaigns, suggests testing at $50 to $100 a day to exit learning within two weeks; under $20 a day rarely gives stable CPA. Best fit: buyers under about 35.
“Sub-$20 days rarely produce stable CPA”
Simple Pin Media, a Pinterest ads agency, says Pinterest ads work best for ecommerce shops with proven products, and it typically requires clients to commit $150 to $200 a day for at least three months.
“Erin typically won't take on a client who can't commit at least $150 to $200 per day, and that budget needs to run for a minimum of three months”
Measured's panel of 269 expansion-channel programs put Snapchat at 1.9x the panel's incrementality average, with conversion buys far ahead of awareness buys. It says to size tests with a power analysis.
“Size the test with a power analysis, not a round number.”
Jetfuel, an agency, says Meta stays more efficient and CTV earns a test once social and search stop scaling. A clean 60 to 90 day test usually needs a five-figure budget, read with a holdout or geo test.
“Meta stays the more efficient direct-response channel. CTV earns a test once social and search stop scaling.”
Podder, a podcast analytics firm, says to buy at least four episodes, since one is too small to judge. It relays Libsyn's $24 to $26 CPM for host reads and puts a single mid-sized show near $2,000.
“a single small show is not a campaign, it is a test with a sample size too small to read”
Measured, which sells geo testing, says incrementality tests commonly run four to eight weeks and almost never under four, since ending early misses lagged sales. Its illustrative Meta example needs about six weeks.
“almost no test should run under four weeks regardless of what the power math allows”
Reddit tells advertisers to budget at least $100 a day when optimizing for purchases with lowest cost bids, moving to Add to Cart or Page Visits on lower budgets, and to judge results after 7 to 14 days.
“Set budget at least to $100/day for Purchase events with Lowest cost bids. For lower budgets, move to Add to Cart or Page Visits.”
Adwave, an ad platform that buys Reddit, suggests small businesses start at $5 to $20 a day on a few relevant communities for a couple of weeks, then scale what works.
“The goal of the initial budget is learning, not immediate scale.”
Black Propeller, which sells paid social management, advises separate Pinterest ad groups for prospecting, retargeting and re-engagement, and 30 to 50 conversions a week before automated bidding.
“Avoid making significant changes to budgets, bids, or targeting within the first seven days of any new campaign.”
Black Propeller, which sells paid social management, says avoid big budget, bid or targeting changes in a new Pinterest campaign's first seven days, and hold budgets two weeks before judging.
“Set budgets you are comfortable maintaining for at least two weeks before drawing conclusions.”
Trackbee, which sells Pinterest conversion tracking, says Pinterest clicks and impressions often cost 30 to 50% less than Meta, and to test two to four weeks at a daily budget 4 to 5 times target CPA.
“works best as a complementary top and mid-funnel channel rather than a Meta replacement”
Elevarus, an agency selling Reddit ad work, says Reddit product ads suit research-driven buys in active subreddit categories, and to plan for 60 to 90 conversions and a 30-day learning window.
“Plan for 60 to 90 conversions before you judge ROAS against Meta.”
DataLatte, a local marketing consultant, says a local business needs at least $5,000 a month for 3+ months to see CTV results, should skip it under 50,000 households, and should pick Facebook first.
“Skip CTV if you're in a rural area with under 50,000 households.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer