How should we check whether a media mix model is right?
Our answer
Check a mix model in four steps: its built-in diagnostics, a forecast of 30 to 90 days it never saw, stability across reruns, then whether it predicts a holdout test you run. Let that test decide.
- Confidence
- Medium
- Strongest proof
- 5 official rules
- 2 practitioners, 10 opinions or claims
- Sources
- 11 sources
- 17 statements
- G
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentA model must pass basic health checks before you read it. Google's Meridian documentation fails models that haven't converged or whose baseline runs negative. It gives goodness of fit only 7.5% of its health score, because the goal is cause, not prediction.
A backtest shows whether the model can forecast weeks it never saw. Recast refits on data that stops 30, 60 or 90 days back and forecasts the unseen weeks. It warns that branded search, sessions and affiliate spend leak the answer.
Agreement with an experiment is the real test. Measured calls a model within 20% of an experiment good and over 30% unreliable. Michael Kaminsky of Recast checks whether a lift test lines up with what the model said.
Our judgment
Run the cheaper checks first (diagnostics, backtest, stability) and let only a test decide. Google's health checks and Recast's backtests can show a model is broken, but not that its channel returns are right. Google's documentation and Measured say only an experiment shows that, and Single Grain and Michael Kaminsky check it with a holdout or lift test. For a model you will use to plan budgets, backtest on a contiguous stretch of weeks it never saw, as Recast and Michael Kaminsky do. We accept Google's point that Meridian is built to estimate cause rather than to forecast, so a Meridian model that forecasts poorly needs a test before it is dropped. Compare the model with a test that has the power to disagree with it, and read a small test's bound nearest zero rather than its point estimate, going by Recast's 11 times example and Measured's 20% and 30% bands.
Our judgment, built from Google, Recast, Measured, Single Grain and Michael Kaminsky: our reading of these sources, not a finding from any one of them.
What the sources say
17 statements from 11 sourcesGoogle's Meridian docs list checks to run before trusting a mix model, such as convergence and a negative baseline, and say fit is secondary because the goal is causal inference.
“the primary goal of an MMM is accurate causal inference, not predictive accuracy”
Boathouse's strategy chief says that for one client an off-the-shelf mix model credited paid media with about 75% of new customers, while a custom model on the same data said under 1%. Check models with lift tests.
“A result below 1% deserves as much scrutiny as a result of 75%.”
Google says a Meridian holdout sample is for comparing model specifications, should be spread across regions and time, and should not be a final block, since the model is not built to forecast.
“There is no guarantee that the model with the best out-of-sample model fit is the best model for causal inference”
Google says Meridian aims at causal estimates, which only experiments can validate. A model with all confounders, even overfit, can be safer, and out-of-sample fit should not be the main test.
“the out-of-sample prediction metrics shouldn't be the primary way model fit is assessed.”
Sellforte, a mix modeling vendor, gives retail buyers RFP questions, such as how the vendor tests prediction on held-out periods. Matching the old model's ROI does not prove a new one correct.
“Exact reproduction of the incumbent's ROI should not be the sole acceptance test: agreement does not establish correctness.”
Stella, which sells testing, says a budget call should state direction, size and a check date, with breakeven at 1 divided by margin (a 40% margin means 2.5x). Its step-size examples are illustrative.
“Cutting a channel because the point estimate looked soft is how brands kill channels that were working.”
Google's Meridian flags channels whose results look uncertain, biased or implausible (a score below 67.5) as priorities for experiments, putting higher-spend channels first.
“Spend-weighting also prioritizes calibration for higher-spend channels due to their greater business importance.”
Google's Meridian health score is zero if the model has not converged; otherwise it weights the checks, with negative baseline, predictive p-value and calibration at 22.5% each and R-squared at 7.5%.
“emphasizes business guardrails and causal plausibility over raw predictive power”
In a simulation by a Zalando researcher, a standard mix model reported 10.61x ROAS for paid search whose true return was 4.20x; estimating from four geo experiments gave 4.14x.
“Structural estimation across four geo-experiments landed on 4.14x against a true 4.20x”
Single Grain, an agency that builds mix models, says ask vendors for diagnostics and out-of-sample accuracy, check against a geo holdout, and want 2+ years of weekly data. Run one holdout before a full model.
“We tell every client the same thing: run one geo holdout test before commissioning a full MMM.”
LiftLab, which sells mix models, says each experiment should feed back into the model, and the channel with the most uncertainty should be tested next. No data.
“A fast MMM should refresh because new evidence changed the model, not because time passed.”
A working paper says mix models are rarely checked against ground truth, and builds a simulated dataset where spend reacts to promotions and performance, with known true effects.
“are rarely validated against ground truth, because ground truth is unobservable in real data.”
Michael Kaminsky, whose firm Recast sells mix models, says most businesses do not need one and it can mislead them; he cites studies where last-touch was about 85 percent as good as experiments in simpler mixes.
“most businesses do not need an MMM. In fact, an MMM will be net negative for them because it will actively mislead them.”
Analytic Partners, a mix modeling vendor, says backtests, confidence intervals and holdouts are necessary but not sufficient. Ask for cases where the model's plan was carried out and results matched the forecast.
“Ask for examples where the model’s scenario planning was executed and the results matched the forecast.”
Measured, which sells mix models and tests, lists sanity checks: channel contributions should sum to actual sales within 5 to 10%, and a gap over 30% against experiments means the model is unreliable.
“Only this check directly validates causation”
Recast says small, noisy tests that reach significance tend to overstate impact: one GeoLift read 11x ROI where a larger lift study found about 2.3x. It advises using the interval bound nearest zero.
“treat the bound of the confidence interval closest to zero as your conservative estimate”
Recast, which sells mix models, says to backtest by holding out a continuous block of future time at 30, 60 and 90 days, and to drop inputs such as branded search that leak sales.
“Backtests must hold out a contiguous block of future time”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- What the research saysWhat academic research adds, kept apart from the answer
- How sure we areWhat its confidence rests on, and what would change the answer