Should we test out-of-home or direct mail, and at what size?
Our answer
Mail new customers only after mail to past customers works. Test out-of-home only as a geo holdout in chosen markets over four to eight weeks. A local business can start with a small Every Door Direct Mail drop.
- Confidence
- Low
- Strongest proof
- 1 controlled test
- 1 official rule, 4 datasets, 4 practitioners, 1 opinion or claim
- Sources
- 10 sources
- 11 statements
EMDICUDS
New here? Take the 1-minute tour
Why this is the answer
4 reasons and our judgmentMail to past customers beats mail to strangers. Measured (incrementality testing firm) found mail to past customers pooled 1.6 times its panel benchmark, while cold lookalike prospecting mail pooled below it. Evolve Media (ecommerce marketing agency) moves to prospecting only after retention mail is proven.
Out-of-home has no clicks to track, so a geo holdout is the fair read. Stackmatix (paid media agency) runs it in two or three markets that match its customers, against similar markets with no exposure, and scales only the markets that showed lift.
A test too small to read wastes the budget. Measured found that sub-scale tests of new channels in its panel often failed to register lift, and it sizes each test with a power analysis rather than a round number.
Local mail is cheap to try. USPS's Every Door Direct Mail reaches every address on chosen routes from $0.26 a piece with no permit. Contractor+ (contractor software maker) tells local firms to track postcards with a unique phone number or QR code.
Our judgment
We put mail to new customers after mail to past customers has beaten a holdout, because the measured results favor warm lists and the guides place prospecting last. We run out-of-home only as a geo holdout in chosen markets against similar markets without it, with the budget sized by a power analysis, judged on site visits, searches and sales rather than clicks or impressions. A local business can test mail cheaply with Every Door Direct Mail and a tracked phone number or QR code, holding back a similar area.
Our judgment, built from Measured, Evolve Media, DirectMail.io, Stackmatix, Web Tonic, USPS and Contractor+: our reading of these sources, not a finding from any one of them.
What the sources say
11 statements from 10 sourcesUSPS's Every Door Direct Mail reaches every address on chosen routes for $0.26 a piece in postage at retail, at 200 to 5,000 pieces per day per ZIP Code, with no permit needed.
“Send at least 200 and up to 5,000 pieces per day per ZIP Code”
Drew Sanocki says a holdout test of 2025 Black Friday direct mail across many brands showed 3.89x incremental ROAS. Brand count and method are not given, and he appears tied to a direct-mail business.
“we holdout-tested 2025 bfcm mail across a ton of brands. 3.89x incremental roas.”
Contractor+, which sells postcard mailing, says contractor postcards need a unique phone number or QR link, should be judged in a controlled test on won jobs against full cost, and its examples are not results.
“Run an affordable test, track qualified leads and won jobs, and compare contribution with the full campaign cost.”
Lob, which sells direct mail, surveyed 2,000 US adults and found 87% read mail the day it arrives. PPC Land notes this measures recall and intent, with no mailed versus unmailed comparison.
“What Lob has measured is recall and intention, not response rates.”
Web Tonic, an agency, says to budget mobile search and social alongside billboards, citing Nielsen's figure that 46% of US adults search after seeing an OOH ad, and to track visits and traffic, not impressions.
“Track store visits, app installs and site traffic with device-level measurement rather than billboard impressions alone”
Measured's panel of enterprise brands found CRM-targeted programmatic direct mail at 1.6x the expansion-channel benchmark, while cold lookalike mail and a large catalog program did worse.
“Target warm, and measure with holdouts.”
Measured's panel of 269 expansion-channel programs put Snapchat at 1.9x the panel's incrementality average, with conversion buys far ahead of awareness buys. It says to size tests with a power analysis.
“Size the test with a power analysis, not a round number.”
Stackmatix, a paid media agency, says to prove out-of-home with geo holdouts: two to three test markets, matched holdouts, a four to eight week window, then scale only markets that show lift.
“If you cannot define a measurement window or a holdout, you are not ready for OOH.”
Evolve Media, an agency, measures direct mail with a holdout: it keeps 5 to 15% of the target audience unmailed and compares conversions between the mailed and unmailed groups.
“Hold-out testing is the gold standard but requires sufficient audience size to maintain statistical significance (typically 5,000+ in each group).”
DirectMail.io, a direct mail seller, says proving mail ROI needs a matched or geographic holdout with 200 to 400 conversions per cell, and that vendors' 1,000%+ ROAS figures are mostly last-touch.
“Batch prospecting mail requires matched control or geographic holdout because the list selection process itself introduces selection bias.”
OUTFRONT Media, which sells billboards, says a six-week Kin Insurance campaign in San Francisco and Los Angeles lifted website traffic 67% against a control. It reports no leads, policies or spend.
“generated a +67% lift in website traffic”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer