Should we use AI tools to analyze marketing data?
Our answer
Use AI tools to pull, summarize and explore your marketing data with read-only access. Give them your metric definitions, check their totals against store numbers and treat their explanations as hypotheses, not decisions.
- Confidence
- Medium
- Strongest proof
- 1 controlled test
- 3 official rules, 1 dataset, 7 practitioners, 2 opinions or claims
- Sources
- 14 sources
- 14 statements
PWAGPA
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentAI gets the numbers right more often than the reasons. In e-dialog's test of Claude on a Meta account, the numbers matched its reports and the trend reading was strong. But the AI blamed ad fatigue for a campaign that had simply ended.
AI tools slip most on metric definitions. Precisian (ecommerce data modeling firm) finds the arithmetic right but the definitions guessed. Windsor.ai's logs show about 1 in 11 AI data queries failing, with Claude and ChatGPT counting GA4 users differently.
The makers of AI tools themselves say to check the output. Google says its analytics assistant may make mistakes, and Shopify says AI outputs can contain errors. In Spotify Engineering's test, raw AI predictions recovered only 39% of real headline test effects.
Our judgment
- Treat any cause or recommendation as a hypothesis. Use AI tools to pull, summarize and explore data, and check what they say about why, because tests found the numbers usually right and the explanations sometimes wrong.
- Give the assistant written metric definitions. Use one assistant for each recurring report, because assistants pick their own definitions and two of them count GA4 users differently.
- Connect analysis tools with read-only access.
- Don't let AI output stand in for a test. Test before moving budget.
Our judgment, built from e-dialog, PageFly, Common Thread Collective, Precisian, Windsor.ai, Meta, Shopify and Spotify Engineering: our reading of these sources, not a finding from any one of them.
What the sources say
14 statements from 14 sourcesOn Marketing Operators, HexClad's Connor Rolain says his team turned a monthly report that took three weeks into a live AI dashboard. No performance figures given.
Shopify says its Sidekick assistant drafts store copy and theme blocks, many AI features cost nothing extra, availability varies by plan, and AI output can contain errors that need review.
“AI outputs can contain errors, so review anything a tool creates or changes.”
Precisian, which sells a data layer, says ChatGPT gets the arithmetic right but invents definitions like revenue or ROAS when they are not in the file. Its main check: tie totals back to Shopify or the books.
“It will make up the definition, then compute the number correctly on top of it.”
Google's beta Ask Advisor is a Gemini assistant in Google Analytics meant to diagnose drops and spikes, using only data from that one property. Google warns it may make mistakes.
“Generated responses related to Google Analytics data rely only on information from within that specific Google Analytics property.”
Meta says its ads MCP rules can give an AI agent read access to performance data while blocking budget, campaign or creative changes, and advises starting read-only or approval-required and widening access over time.
“Start an account in a read-only or approval-required posture and widen permissions programmatically as confidence grows”
Appbrew, which sells a rival app builder, says Shopify Sidekick is strong at drafting copy and building Flow automations but unreliable for bulk edits, custom code and precise numbers, and rarely admits gaps.
“It's unreliable for bulk edits, custom code, metafields, and precise numbers.”
Windsor.ai, which sells the connector it logged, says AI assistants made about 11% of marketing data queries in one week across 35,000+ teams, and about 1 in 11 AI queries failed to run, whatever the vendor.
“Pick one assistant per workflow and standardize.”
Spotify engineers found raw LLM predictions recovered only 39% of the real effect in thousands of headline A/B tests. They say LLMs can screen out weak ideas but cannot stand in for human tests.
“can be used to filter out weak ideas before they consume an experiment slot”
Matt Bertulli built AI tools and dashboards that pull live ad account data, but says he lost days debugging and shipped tools that broke the next day.
“the prototype is not the product.”
CTC runs Claude with written playbook skills over an MCP connection to analyze client Meta accounts, and its media buyers act as editors who check the output and add client context.
“they're an editor, they're a filter, they get to gut check.”
e-dialog tested Claude on a Meta ad account: figures matched its reports and trend analysis was strong, but causal explanations were often wrong, and Meta's own assistant leaned toward bigger budgets.
“Meta’s proposals follow the platform’s logic, which tends to favor increased budgets and greater automation.”
PageFly, which sells a page builder, tested Shopify Sidekick on one small demo store: its analytics summary matched the dashboard, but it cannot redesign pages. Treat output as a first draft.
“Treat its output as a fast first draft, not a final answer.”
True Classic's CEO says it runs 100% of its Meta spend through Triple Whale's autonomous media buyer, Moby 2, on an episode with Triple Whale's COO. No results or price are disclosed.
“True Classic now runs 100 percent of its Meta spend through an autonomous media buyer, Moby 2”
1ClickReport, which sells the tool used, asked Claude and ChatGPT 30 analytics questions on live accounts, scored by its founder: Claude did better on attribution, ChatGPT on ad platform specifics.
“without live data access, both models guess plausibly but cannot verify.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it2 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer