How should we judge a measurement vendor's claims?
Our answer
Treat a vendor's claims as unproven. Believe them only when it shows, on your data, its error on weeks the model never saw, a range around each channel's return and agreement with a holdout test you run.
- Confidence
- Medium
- Strongest proof
- 1 official rule
- 1 dataset, 3 practitioners, 10 opinions or claims
- Sources
- 14 sources
- 15 statements
SGMC
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentAsk for error on a period the model never saw. Single Grain, Measured and Marketing Case Bootcamp all advise it. Marketing Case Bootcamp, a marketing training site, calls under 10% good and says over 20% means the model has not earned a recommendation.
An unchecked model can drift far from the truth. An off-the-shelf model credited 10 paid channels with about 75% of new customers, where a custom model on the same data found under 1% (Sonia Chung of Boathouse, an agency, in AdExchanger).
Agreement with a controlled experiment is the test that matters most. Measured, an incrementality testing firm, calls it the gold standard. Marketing Case Bootcamp argues that when a model and a clean lift study disagree, the model is wrong.
Our judgment
We make agreement with a controlled test the deciding check, ahead of how well the model fits past data or its error on weeks it never saw. Measured, Single Grain and Marketing Case Bootcamp all rank it highest, and Google's Meridian guidance says prediction error should not be the primary test. We draw the error lines where Marketing Case Bootcamp, Prescient AI, Single Grain and Measured put them: under 10% is good, above 15% is a warning and above 20% is a reason not to act. We give no weight to a results claim that names no sample, method or comparison group. That follows Charlie Cole's rule against letting a vendor grade its own homework, with Northbeam's customer averages as the example.
Our judgment, built from Measured, Single Grain, Marketing Case Bootcamp, Google, Prescient AI, Charlie Cole and Northbeam: our reading of these sources, not a finding from any one of them.
What the sources say
15 statements from 14 sourcesBoathouse's strategy chief says that for one client an off-the-shelf mix model credited paid media with about 75% of new customers, while a custom model on the same data said under 1%. Check models with lift tests.
“A result below 1% deserves as much scrutiny as a result of 75%.”
Northbeam claims its accounts average 3.5% higher ROAS and 4.9% lower CAC 180 days after activation, without disclosing sample, method or comparison group.
“These are the average improvements we see in ad accounts 180 days after activating Northbeam”
Google says Meridian aims at causal estimates, which only experiments can validate. A model with all confounders, even overfit, can be safer, and out-of-sample fit should not be the main test.
“the out-of-sample prediction metrics shouldn't be the primary way model fit is assessed.”
Sellforte, a mix modeling vendor, gives retail buyers RFP questions, such as how the vendor tests prediction on held-out periods. Matching the old model's ROI does not prove a new one correct.
“Exact reproduction of the incumbent's ROI should not be the sole acceptance test: agreement does not establish correctness.”
In a simulation study reported by PPC Land, with a true paid search ROAS of 4.20, a standard mix model estimated 10.61 and changing priors did not fix it. Estimates built from geo tests landed at 4.31 and 4.14.
“Standard MMM 10.61x (90% interval 6.56-14.36)”
Single Grain, an agency that builds mix models, says ask vendors for diagnostics and out-of-sample accuracy, check against a geo holdout, and want 2+ years of weekly data. Run one holdout before a full model.
“We tell every client the same thing: run one geo holdout test before commissioning a full MMM.”
Sellforte's made-up example shows a calibrated mix model (5.3) can fairly differ from a lift test (4.0), but only when the mapping, KPI, time window, prior and uncertainty stay visible.
“The gap is acceptable only when the mapping, KPI, time window, prior, and uncertainty remain visible and defensible”
Analytic Partners, a mix modeling vendor, says backtests, confidence intervals and holdouts are necessary but not sufficient. Ask for cases where the model's plan was carried out and results matched the forecast.
“Ask for examples where the model’s scenario planning was executed and the results matched the forecast.”
Charlie Cole says brands should never let an ad platform or vendor grade its own homework.
“you should never let any advertiser or vendor grade their own homework.”
Brat Vukovich suggests asking an attribution vendor to reconcile 10 recent orders with Shopify and state how much revenue sits unattributed, which he estimates usually runs 15 to 30%. No data.
“Pull 10 orders from yesterday and compare line by line”
Marketing Case Bootcamp, a training site, says ask for confidence intervals, flag channels under 5% of spend, and distrust holdout error over 20%. When the model clashes with a clean lift test, trust the test.
“When the MMM says paid social drives 18% lift and a clean Meta conversion lift study from the same window said 3%, the MMM is wrong, not the experiment.”
Measured, which sells mix models and tests, lists sanity checks: channel contributions should sum to actual sales within 5 to 10%, and a gap over 30% against experiments means the model is unreliable.
“Only this check directly validates causation”
Jones Road Beauty checks its mix model's recommendations by running the next experiment. Cody Plofker says most MMM vendors work on just trust me. No model error figures.
“just trust me”
Prescient AI, a mix model vendor, warns not to assume your vendor validated the model, and lists backtesting, cross validation and holdout tests, with roughly 10 to 15% error as acceptable.
“Most marketing leaders assume their MMM vendor already validated the model. That’s a dangerous assumption.”
Recast's Michael Kaminsky says demand disclosed methods, a test budget to calibrate the mix model, and out-of-sample accuracy from vendors, and treat consistent large misses as a red flag.
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer