All answers

The Prevailing
Answers for
Business model:
Business model, pick up to two

How should we judge a measurement vendor's claims?

The PrevailingUpdated Nov 13, 2026

Our answer

Treat a vendor's claims as unproven. Believe them only when it shows, on your data, its error on weeks the model never saw, a range around each channel's return and agreement with a holdout test you run.

Confidence
Medium
Strongest proof
1 official rule
1 dataset, 3 practitioners, 10 opinions or claims
Sources
14 sources
15 statements

New here? Take the 1-minute tour

Why this is the answer

3 reasons and our judgment

Ask for error on a period the model never saw. Single Grain, Measured and Marketing Case Bootcamp all advise it. Marketing Case Bootcamp, a marketing training site, calls under 10% good and says over 20% means the model has not earned a recommendation.

An unchecked model can drift far from the truth. An off-the-shelf model credited 10 paid channels with about 75% of new customers, where a custom model on the same data found under 1% (Sonia Chung of Boathouse, an agency, in AdExchanger).

Agreement with a controlled experiment is the test that matters most. Measured, an incrementality testing firm, calls it the gold standard. Marketing Case Bootcamp argues that when a model and a clean lift study disagree, the model is wrong.

Our judgment

We make agreement with a controlled test the deciding check, ahead of how well the model fits past data or its error on weeks it never saw. Measured, Single Grain and Marketing Case Bootcamp all rank it highest, and Google's Meridian guidance says prediction error should not be the primary test. We draw the error lines where Marketing Case Bootcamp, Prescient AI, Single Grain and Measured put them: under 10% is good, above 15% is a warning and above 20% is a reason not to act. We give no weight to a results claim that names no sample, method or comparison group. That follows Charlie Cole's rule against letting a vendor grade its own homework, with Northbeam's customer averages as the example.

Our judgment, built from Measured, Single Grain, Marketing Case Bootcamp, Google, Prescient AI, Charlie Cole and Northbeam: our reading of these sources, not a finding from any one of them.

What the sources say

15 statements from 14 sources
  • AdExchangerOct 19, 2026Supports

    Boathouse's strategy chief says that for one client an off-the-shelf mix model credited paid media with about 75% of new customers, while a custom model on the same data said under 1%. Check models with lift tests.

    “A result below 1% deserves as much scrutiny as a result of 75%.”

    Read it on adexchanger.com

  • NorthbeamOct 17, 2026Caveat

    Northbeam claims its accounts average 3.5% higher ROAS and 4.9% lower CAC 180 days after activation, without disclosing sample, method or comparison group.

    “These are the average improvements we see in ad accounts 180 days after activating Northbeam”

    Read it on northbeam.io

  • GoogleSep 28, 2026Disagrees

    Google says Meridian aims at causal estimates, which only experiments can validate. A model with all confounders, even overfit, can be safer, and out-of-sample fit should not be the main test.

    “the out-of-sample prediction metrics shouldn't be the primary way model fit is assessed.”

    Read it on developers.google.com

  • SellforteSep 24, 2026Supports

    Sellforte, a mix modeling vendor, gives retail buyers RFP questions, such as how the vendor tests prediction on held-out periods. Matching the old model's ROI does not prove a new one correct.

    “Exact reproduction of the incumbent's ROI should not be the sole acceptance test: agreement does not establish correctness.”

    Read it on sellforte.com

  • PPC LandAug 31, 2026Caveat

    In a simulation study reported by PPC Land, with a true paid search ROAS of 4.20, a standard mix model estimated 10.61 and changing priors did not fix it. Estimates built from geo tests landed at 4.31 and 4.14.

    “Standard MMM 10.61x (90% interval 6.56-14.36)”

    Read it on ppc.land

  • Single GrainAug 30, 2026Supports

    Single Grain, an agency that builds mix models, says ask vendors for diagnostics and out-of-sample accuracy, check against a geo holdout, and want 2+ years of weekly data. Run one holdout before a full model.

    “We tell every client the same thing: run one geo holdout test before commissioning a full MMM.”

    Read it on singlegrain.com

  • SellforteAug 11, 2026Caveat

    Sellforte's made-up example shows a calibrated mix model (5.3) can fairly differ from a lift test (4.0), but only when the mapping, KPI, time window, prior and uncertainty stay visible.

    “The gap is acceptable only when the mapping, KPI, time window, prior, and uncertainty remain visible and defensible”

    Read it on sellforte.com

  • Analytic PartnersJul 22, 2026Supports

    Analytic Partners, a mix modeling vendor, says backtests, confidence intervals and holdouts are necessary but not sufficient. Ask for cases where the model's plan was carried out and results matched the forecast.

    “Ask for examples where the model’s scenario planning was executed and the results matched the forecast.”

    Read it on analyticpartners.com

  • DTC Newsletter / DTC PodcastMay 25, 2026Supports

    Charlie Cole says brands should never let an ad platform or vendor grade its own homework.

    “you should never let any advertiser or vendor grade their own homework.”

    Read it on podcasters.spotify.com

  • Inceptly (Brat Vukovich)May 18, 2026Caveat

    Brat Vukovich suggests asking an attribution vendor to reconcile 10 recent orders with Shopify and state how much revenue sits unattributed, which he estimates usually runs 15 to 30%. No data.

    “Pull 10 orders from yesterday and compare line by line”

    Read it on inceptly-agency.beehiiv.com

  • Marketing Case BootcampApr 25, 2026Supports

    Marketing Case Bootcamp, a training site, says ask for confidence intervals, flag channels under 5% of spend, and distrust holdout error over 20%. When the model clashes with a clean lift test, trust the test.

    “When the MMM says paid social drives 18% lift and a clean Meta conversion lift study from the same window said 3%, the MMM is wrong, not the experiment.”

    Read it on marketingcasebootcamp.com

  • MeasuredApr 21, 2026Supports

    Measured, which sells mix models and tests, lists sanity checks: channel contributions should sum to actual sales within 5 to 10%, and a gap over 30% against experiments means the model is unreliable.

    “Only this check directly validates causation”

    Read it on measured.com

  • Cody PlofkerApr 1, 2026Supports

    Jones Road Beauty checks its mix model's recommendations by running the next experiment. Cody Plofker says most MMM vendors work on just trust me. No model error figures.

    “just trust me”

    Read it on adexchanger.com

  • Prescient AIJan 12, 2026Supports

    Prescient AI, a mix model vendor, warns not to assume your vendor validated the model, and lists backtesting, cross validation and holdout tests, with roughly 10 to 15% error as acceptable.

    “Most marketing leaders assume their MMM vendor already validated the model. That’s a dangerous assumption.”

    Read it on prescientai.com

  • RecastNov 19, 2025Supports

    Recast's Michael Kaminsky says demand disclosed methods, a test budget to calibrate the mix model, and out-of-sample accuracy from vendors, and treat consistent large misses as a red flag.

    Read it on adexchanger.com

In the full answer

For readers who sign in
  • What to do this week3 steps
  • The numbers behind it3 figures from cited sources
  • Where each source standsA chart placing each source on the question
  • The strongest disagreementThe best case against the answer, from a cited source
  • How sure we areWhat its confidence rests on, and what would change the answer

Read next

More in Tools and vendors