All answers

The Prevailing
Answers for
Business model:
Business model, pick up to two

How should we handle a vendor's results that conflict with our own tests?

The PrevailingPublished Oct 28, 2026

Our answer

Check that both measure the same thing, then let a sound test win. Match scope, dates, outcome and spend level. If the test holds, move that channel's budget on it and have the vendor recalibrate.

Confidence
Medium
Strongest proof
1 official rule
3 practitioners, 10 opinions or claims
Sources
10 sources
14 statements

New here? Take the 1-minute tour

Why this is the answer

3 reasons and our judgment

Many gaps between a vendor's results and your own tests are legitimate. A test may count platform-tracked conversions while the vendor's model covers total sales, and the two will differ for that reason alone (Sellforte, Google's Meridian guide).

A clean experiment outranks a model's estimate. A model claiming 18% lift is wrong when a clean lift study from the same window found 3% (Marketing Case Bootcamp). Measured calls a gap over 30% fundamentally unreliable.

Acting on the test can pay off. Haus reports OluKai cutting CAC by 20% after trusting its tests over a correlational model. LiftLab (marketing mix modeling vendor) sends the widest gaps to the next experiment.

Our judgment

  • Check that the vendor and the test measured the same scope, dates, outcome and spend level before deciding which is right. Sellforte, Google's Meridian guide, Haus and WorkMagic each name mismatches that produce legitimate gaps.
  • Once the match holds and the test is sound, let the test decide that channel's budget for that period. Marketing Case Bootcamp, Measured, Haus and LiftLab all rank a clean experiment above a model's estimate. Rerun a test whose range includes zero or whose design was weak before it overrides the vendor, since LiftLab says a low-quality test calibrates a model wrongly and Sellforte down-weights an imprecise one.
  • Have the vendor fold the result in with its uncertainty rather than copy it or average it. This follows Sellforte's and Google's calibration guidance and Ittai Shacham's warning against tuning priors toward expected answers. Then check the vendor's next forecast against what happens, since Analytic Partners counts a model accurate only when its forecast comes true once you act on it.
  • Treat a gap that persists as the next channel to test. Judge both numbers against store revenue and contribution margin, following LiftLab's practice of testing the widest gaps first and Common Thread Collective's rule that incremental ROAS stays subordinate to contribution margin.

Our judgment, built from Sellforte, Google, Haus, WorkMagic, Marketing Case Bootcamp, Measured, LiftLab, Ittai Shacham, CTC and Analytic Partners: our reading of these sources, not a finding from any one of them.

What the sources say

14 statements from 10 sources
  • SellforteOct 22, 2026Supports

    Sellforte divides incremental ROI by platform ROAS to get a multiplier: if Meta's reported return is worth half, an incremental target of 4 means setting Meta's target ROAS at 8.

    “Good news is that ROAS metrics can be corrected for incrementality, using calibration multipliers.”

    Read it on sellforte.com

  • AdExchangerOct 19, 2026Supports

    Boathouse's strategy chief says that for one client an off-the-shelf mix model credited paid media with about 75% of new customers, while a custom model on the same data said under 1%. Check models with lift tests.

    “A result below 1% deserves as much scrutiny as a result of 75%.”

    Read it on adexchanger.com

  • GoogleSep 24, 2026Supports

    Google says there is no single formula for turning an experiment into a mix model prior: combine evidence and allow extra uncertainty, since lift tests and the model measure different things.

    “There is no single formula to translate an experiment result into a prior.”

    Read it on developers.google.com

  • HausSep 23, 2026Supports

    A Haus scientist says a typical in-house mix model takes weeks to stand up, but clashes with tests lead teams to tune priors, and around month twelve they must fund a team or pay an outside provider.

    “And prior tuning is essentially cherry-picking.”

    Read it on haus.io

  • SellforteAug 18, 2026Supports

    Sellforte, which sells mix models and testing, says to weight multiple lift tests by relevance, recency, confidence and spend when calibrating a mix model, leaning on the best test rather than averaging.

    “The calibration should instead draw most heavily on the highest-quality test.”

    Read it on sellforte.com

  • SellforteAug 11, 2026Supports

    Sellforte says a calibrated mix model can legitimately show a different incremental ROAS than a lift study, and a large gap should prompt a calibration audit. Its numbers are hypothetical.

    “Calibration should make the two sources coherent. It should not force a broad model output to copy one experimental point estimate in every period.”

    Read it on sellforte.com

  • HausAug 4, 2026Supports

    Haus, which sells tests and a mix model, says to trust the experiment when the two disagree; it says client OluKai cut CAC 20% by moving spend to channels its tests validated.

    “What happens when your MMM and your incrementality tests disagree? Trust the experiment.”

    Read it on haus.io

  • Analytic PartnersJul 22, 2026Supports

    Analytic Partners, a mix modeling vendor, says backtests, confidence intervals and holdouts are necessary but not sufficient. Ask for cases where the model's plan was carried out and results matched the forecast.

    “Ask for examples where the model’s scenario planning was executed and the results matched the forecast.”

    Read it on analyticpartners.com

  • Common Thread CollectiveJul 2, 2026Supports

    CTC says even a well-calibrated incremental ROAS should answer to revenue and contribution margin; if iROAS improves while margin is flat or falling, recheck the measurement.

    “Even a well-calibrated incremental return on ad spend should always be subordinate to the realities of revenue and contribution margin”

    Read it on commonthreadco.com

  • LiftLabMay 14, 2026Supports

    LiftLab, which sells mix models and lift tests, says geo holdouts answer keep-or-cut questions, while budget planning needs the return on the next dollar from spend-level tests. No data.

    “An incrementality test without a pre-committed decision is measurement theater.”

    Read it on liftlab.com

  • WorkMagicApr 25, 2026Supports

    WorkMagic says Meta's Incremental Attribution counts only conversions Meta can see, so sales on Amazon or in stores go uncredited; omnichannel brands need a geo test or a calibrated mix model.

    “It only measures what Meta can see.”

    Read it on workmagic.io

  • Marketing Case BootcampApr 25, 2026Supports

    Marketing Case Bootcamp, a training site, says ask for confidence intervals, flag channels under 5% of spend, and distrust holdout error over 20%. When the model clashes with a clean lift test, trust the test.

    “When the MMM says paid social drives 18% lift and a clean Meta conversion lift study from the same window said 3%, the MMM is wrong, not the experiment.”

    Read it on marketingcasebootcamp.com

  • MeasuredApr 21, 2026Supports

    Measured, which sells mix models and tests, lists sanity checks: channel contributions should sum to actual sales within 5 to 10%, and a gap over 30% against experiments means the model is unreliable.

    “Only this check directly validates causation”

    Read it on measured.com

  • LiftLabApr 17, 2026Supports

    LiftLab, which sells mix models and geo tests, says in one example a mix model read Paid Social at $1.20 per dollar while a geo holdout found $0.85; a gap should point to where to test next.

    “Low-quality experiments do not produce weak calibration. They produce incorrect calibration, degrading the MMM accuracy more than no calibration.”

    Read it on liftlab.com

In the full answer

For readers who sign in
  • What to do this week3 steps
  • The numbers behind it3 figures from cited sources
  • Where each source standsA chart placing each source on the question
  • The strongest disagreementThe best case against the answer, from a cited source
  • How sure we areWhat its confidence rests on, and what would change the answer

Read next

More in Tools and vendors