Should our site publish an llms.txt file or other AI-specific files?
Our answer
Don't build or buy an llms.txt file for search visibility. Google ignores it, and 97% of the files Ahrefs checked got no requests. Keep any file your platform makes accurate, and set robots.txt to admit AI search crawlers.
- Confidence
- High
- Strongest proof
- 13 official rules
- 2 datasets, 2 practitioners, 2 opinions or claims
- Sources
- 13 sources
- 19 statements
- G
APTHT
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentGoogle Search doesn't use llms.txt, AI text files or Markdown copies, says Google's guide to its AI features. Creating them will neither harm nor help your visibility in Google.
AI search bots barely read llms.txt files. Almost none of the llms.txt files on 137,210 sites got requests in May 2026 (Ahrefs, which sells SEO software). Its top readers were GPTBot, a training crawler, and the coding agent Claude-Code, ahead of every AI search bot.
Robots.txt is the file that decides AI search visibility. OpenAI, Perplexity and Anthropic say their search crawlers, controlled through robots.txt, decide whether your pages can appear in their answers. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers.
Our judgment
We'd leave a platform-made llms.txt in place and keep it accurate rather than remove it, because Google says such files neither harm nor help Google Search, Shopify serves one for every store by default, and Semrush, Tobira and The HOTH call the file cheap and low risk. We'd spend the time on robots.txt instead, since that is where the AI search crawlers are let in or kept out and Google says Googlebot's robots.txt rules govern its AI features.
Our judgment, built from Google, Shopify, Semrush, Tobira, The HOTH, OpenAI, Perplexity and Anthropic: our reading of these sources, not a finding from any one of them.
What the sources say
19 statements from 13 sourcesShopify says every store automatically serves /agents.md, /llms.txt and /llms-full.txt for AI agents, with store name, policies and sitemap, so no app is needed. Robots.txt rules are advisory only.
“You don't need a third-party app to generate these files.”
Google plans to honor the IETF AIPREF AI-training opt-out in robots.txt in early 2027, treating it like disallowing Google-Extended. Nate Hake notes the IETF work is unfinished and disputes remain.
“Google intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027.”
Search Engine Roundtable reports Bing's new controls let sites block AI training or AI answers from using their content, which can cut AI visibility but leaves normal Bing rankings unaffected. Rollout runs into 2027.
“You can block AI answers from using your content but doing so can reduce your visibility in AI experiences. The traditional Bing rankings are unaffected.”
OpenAI says sites opted out of OAI-SearchBot won't appear in ChatGPT search answers, so allow it in robots.txt. GPTBot is a separate setting for training, and changes take about 24 hours.
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
Perplexity says to allow PerplexityBot in robots.txt and your firewall to appear in its search results; it is not used to train AI models, and its user-requested fetcher generally ignores robots.txt.
“Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
Anthropic says blocking Claude-SearchBot or Claude-User may reduce a site's visibility in Claude's search results; ClaudeBot is the separate training crawler.
“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”
Google says AI Overviews and AI Mode have no extra requirements: a page only needs to be indexed and eligible for a snippet. Their traffic is counted in Search Console's overall search traffic.
“There are no additional requirements to appear in AI Overviews or AI Mode”
The HOTH, an SEO agency, says llms.txt blocks nothing, Google Search ignores it and most AI crawlers never request it, so it goes last on the list; use robots.txt to control AI bots.
“If you want to control which AI bots can crawl your site, robots.txt is still the tool.”
Google says blocking Google-Extended controls whether your content trains or grounds Gemini, and does not affect inclusion or ranking in Google Search.
“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”
Prosopo, a bot protection firm, found in its own verified logs that AI crawlers fetched its robots.txt 120 times and llms.txt zero times over four days; no crawler operator documents llms.txt support.
“None of them documents llms.txt support, and our logs match that.”
Semrush, which sells SEO tools, says there is no proof llms.txt boosts AI visibility and no major AI platform has confirmed reading it, so treat it as a cheap experiment and check server logs.
“As of now, there's no proof that it does.”
StoreMend, which sells Shopify audits, says in 1,091 stores it audited in Q2 2026 the most common SEO gap was missing meta descriptions on product and collection pages, then long or duplicated title tags.
“the most common single SEO gap on a Shopify store is missing meta descriptions on product and collection pages.”
Google's John Mueller says an llms.txt file can't help AI systems tell websites apart, points to normal HTML pages and internal links for discovery, and expects agent standards to take six months to a year or more.
“So in LLM system, it basically, by design, can't trust what is here as a way of differentiating between different websites.”
Ahrefs, which sells SEO software, found 97% of llms.txt files across 137,210 sites got zero requests in May 2026, and calls the file largely decoration for AI search visibility.
“97% of those files received zero traffic in May 2026. Nothing fetched them at all.”
Tobira says llms.txt is cheap and harmless but the evidence does not show it lifts AI citations, so add it as basic hygiene and don't pay an agency to maintain it.
“Treat it as table-stakes hygiene the way you treat robots.txt.”
Shopify stores now serve a default agents.md file, with /llms.txt and /llms-full.txt pointing to it, and theme templates can customize what each path shows.
“Your store includes a default agents.md file accessible at /agents.md. The paths /llms.txt and /llms-full.txt also point to this content by default.”
Google says no special AI text files, Markdown or schema markup are needed to appear in its generative AI search features, though structured data still helps with rich results.
“Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add.”
Google says llms.txt and similar AI text files neither help nor harm visibility in Google Search, which ignores them, and that chunking content or chasing inauthentic mentions is unnecessary.
“For Google Search, you can ignore tactics like "chunking" content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions.”
Google's Chrome team added an experimental Lighthouse check for llms.txt, a summary file for AI agents, but treats the file as optional; it does not say Google Search or AI assistants read it.
“the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer