Should we pre-test creative with AI attention-prediction tools before spending on it?
Our answer
Don't let attention-prediction scores decide spend. At most use them to cut the clearly weakest drafts before production. Then launch the rest as new ads and judge them on Meta spend and cost per purchase.
- Confidence
- Low
- Strongest proof
- 3 controlled tests
- 5 datasets
- Sources
- 8 sources
- 8 statements
LLSERA
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentAttention-prediction models screen out weak versions better than they pick the best. In an email test by Meta's own researchers, the model's top pick finished 43rd of 50, yet its top 30 held 5 of the 6 versions that beat the best human one.
Prediction and attention scores track something other than sales. Raw LLM predictions recovered only 39% of real headline-test effects (Spotify Engineering), and measured attention did not map one to one onto brand lift in four live campaigns (Advertising Research Foundation).
The best sales evidence ranks ads rather than testing them on Meta. Vidmob's Kroger model sorted unseen ads with 81% accuracy, and Kantar found the top third of 8,500 ads by score drove 37% more incremental sales.
Our judgment
Screening drafts before production is the one use with support. The sales links from Vidmob and Kantar come from rankings of finished ads at very large advertisers. Cut only the clear bottom of a batch, not everything but the top few. In the email test, keeping the model's top 10 kept none of the versions that beat the best human one, and Spotify's data scientists back predictions only for filtering weak ideas before a real test. Both tests scored email or headline text, not Meta ads. A score should not veto a deliberately different concept, because Kantar says that where AI and human reads differ, the truth likely lies between the two.
Our judgment, built from Vidmob, Kantar, Lee, Letham, Bakshy et al. (Meta researchers, arXiv 2026) and Spotify Engineering: our reading of these sources, not a finding from any one of them.
What the sources say
8 statements from 8 sourcesIn four preregistered field studies for a major music label, AI-made video ads beat human-made ones on TikTok by 26% in click-through, with no significant difference on Meta, at 98% lower cost.
“did not detect statistically significant performance differences”
Vidmob's study of 1,934 Kroger ads reports its model classified high and low converters with 81% accuracy, and guideline-following creative showed up to 4x conversion rate. No randomized or holdout test yet.
Vidmob's Alex Collmer cites a vendor study of 1,900+ assets on Meta and DV360 where guideline-following ads converted about 4x better. He says not spending behind assets that will not work is a big deal.
“It's more important to have a signal up front.”
Kantar says its LINK AI matches human survey ratings for more than 7 in 10 digital video and TV ads. For one FMCG client, the top third of 8,500 ads by score delivered 37% more incremental sales than the bottom third.
“This is invariably a mistake”
Spotify engineers found raw LLM predictions recovered only 39% of the real effect in thousands of headline A/B tests. They say LLMs can screen out weak ideas but cannot stand in for human tests.
“can be used to filter out weak ideas before they consume an experiment slot”
Meta researchers found a model trained on past A/B tests is a poor winner picker (its top pick placed 43rd of 50 in a 5.1 million email test) but useful for building a slate to test.
“The offline model's single highest-ranked creative finishes 43rd of 50 in realized open rate, below the average human arm”
The ARF's study of four national campaigns found attention does not map one to one onto brand lift, and attention vendors' methods often disagreed by platform and placement. It measured brand lift, not sales.
Motion's 2026 data on 578,750 Meta creatives from 6,015 accounts found about 5% become winners, from 3.8% under $10K a month to 8.2% at $1M or more, and about half get little or no spend.
“About half of all ads receive little or no spend”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer