Should we block or rate-limit bots at our CDN or firewall, and how do we keep Google's crawlers and AI search crawlers reading the site so we don't lose Google Shopping listings, ads or AI citations?
Our answer
Yes, block or rate-limit unwanted bots, but exempt the crawlers that bring shoppers: Google's verified crawlers and AI assistants' search crawlers and fetchers. If you refuse AI training, do it by name, because a Cloudflare Training block also blocks Googlebot.
- Confidence
- High
- Strongest proof
- 18 Google rules
- 2 datasets, 5 practitioners, 2 opinions or claims
- Sources
- 14 sources
- 27 statements
- GCOAP
New here? Take the 1-minute tour
Why this is the answer
4 reasons and our judgmentBlanket bot protection locks Google out. Google's crawlers fail challenge pages, and pages and shopping listings then drop out of Search (Google). One client lost all its Merchant Center listings while Google Ads kept charging (Jonathan Bird).
An AI training block can catch Googlebot. Cloudflare blocks Googlebot, Applebot and Bingbot for any site that blocks Training, from September 15, 2026, while its Disallow option leaves them out (Cloudflare). One agency client's long-standing rule started returning 403s to Googlebot (StubGroup).
Shopping depends on more than one Google crawler. Storebot-Google crawls for all Google Shopping surfaces, and Merchant Center requires robots.txt to allow Googlebot and Googlebot-Image (Google). StoreBot can use other IP addresses, so Google advises reverse DNS checks over fixed IP allowlists.
Blocking AI search crawlers costs citations. Sites closed to OAI-SearchBot don't show in ChatGPT search answers (OpenAI), and blocking Claude-SearchBot may reduce visibility in Claude (Anthropic). One site's firewall refused 11.6% of AI crawler fetches that its robots.txt allowed (CrunchJunkie).
Our judgment
- If you refuse AI training in Cloudflare, use Disallow or name the crawlers, not Block. Block also stops Googlebot, so check that your own robots.txt, which Cloudflare serves after its managed lines, still allows Google’s crawlers, since they obey it and Merchant Center needs it.
- Treat any CDN or firewall bot change as a Google Shopping risk. Check Merchant Center and your logs after each one.
- Give the AI search crawlers and fetchers you want the same exemption as Google’s. Match it on user agent and published IP ranges and place it above any block or challenge rule, because a CDN block keeps pages out of their answers whatever robots.txt says.
- Check whether Cloudflare’s managed robots.txt names Google-Extended before setting Training to Disallow. Blocking that token also keeps your pages out of grounding in Gemini Apps.
- If you advertise in ChatGPT, leave OAI-AdsBot unblocked too. OpenAI uses it to check ad landing pages.
Our judgment, built from Cloudflare, StubGroup, Jonathan Bird, Relevant Audience, Google, OpenAI, Anthropic, Perplexity, Ian Nuttall and CrunchJunkie: our reading of these sources, not a finding from any one of them.
What the sources say
27 statements from 14 sourcesGoogle says firewalls, CDNs like Cloudflare and bot detection tools can block StoreBot. Verify Google by reverse DNS, not static IP allowlists. Fixed local inventory products typically return in 12 to 48 hours.
“If you use a firewall or a CDN like Cloudflare, ensure that Google's crawlers are whitelisted and your server is not returning 403 or 404 errors only to automated bots.”
Google says Merchant Center cannot check products if robots.txt blocks Googlebot or Googlebot-image. Products keep showing for up to 30 days on previously approved images, and fixes take 24 to 72 hours to show.
“ensure that your robots.txt file allows both user-agents 'Googlebot' (used for landing pages) and 'Googlebot-image' (used for images) to crawl your full site.”
OpenAI says its crawler settings are independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot for training. ChatGPT-User fetches pages for users, and robots.txt may not apply.
“we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below”
Anthropic says ClaudeBot collects content for training, while blocking Claude-User or Claude-SearchBot may reduce your visibility in Claude's search results. It warns IP blocking may not reliably opt you out.
“Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”
Perplexity says sites behind a firewall may need to explicitly allow its bots, matching each user agent to its own IP ranges, and notes Cloudflare's Bot Fight Mode can still challenge them.
“Cloudflare’s Bot Fight Mode can’t be skipped with WAF custom rules.”
Google says its user-triggered fetchers, including Google-Agent, act on a user's request and so generally ignore robots.txt rules.
“Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”
Google says its Google-Agent browses and acts on users' requests, and asks sites to skip bot challenges for verified requests, with that allow rule placed above broad data center, bot and AI agent blocks.
“For verified requests, skip bot challenges.”
Ian Nuttall explains that a firewall or CDN can block AI crawlers that robots.txt allows. Of 499 top domains with a readable robots.txt, 105 block GPTBot and 55 block OAI-SearchBot on their home page.
“Blocking GPTBot or ClaudeBot doesn’t remove you from their search answers.”
Google says blanket CDN or firewall bot protection that challenges automated requests blocks its crawlers, so pages and shopping listings drop out of Search. It advises allowing verified bots or brief 429 or 503 replies.
“If you need to block bots, allow verified bots in your CDN and firewall settings.”
Cloudflare, a bot management vendor, lets sites block or allow AI search, agent and training crawlers. All three default to allow on new domains, and it notes crawlers can ignore a robots.txt disallow.
“robots.txt directives are voluntary. Crawlers can ignore them.”
Cloudflare, a bot management vendor, says its managed robots.txt blocks AI training crawlers such as GPTBot and Google-Extended and allows other agents, with Googlebot not named. It notes compliance is voluntary.
“Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response.”
Cloudflare's docs say its Agent policy covers bots acting in real time for a person, and Block on pages with ads blocks them where it detects ads. Personal Agents inherit it, and Cloudflare recommends allowing them.
“Cloudflare recommends Allow (do not block) for Personal Agents.”
OpenAI says sites opted out of OAI-SearchBot won't appear in ChatGPT search answers, so allow it in robots.txt. GPTBot is a separate setting for training, and changes take about 24 hours.
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
Anthropic says blocking Claude-SearchBot or Claude-User may reduce a site's visibility in Claude's search results; ClaudeBot is the separate training crawler.
“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”
Perplexity says to allow PerplexityBot in robots.txt and your firewall to appear in its search results; it is not used to train AI models, and its user-requested fetcher generally ignores robots.txt.
“Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
StubGroup says a client's Cloudflare rule blocking AI training bots began returning 403 errors to Googlebot in late August 2026, de-indexing pages. Product pages were re-indexed after the fix.
“When Googlebot attempts to crawl an actively managed Cloudflare website set to block it, it returns a 403 error.”
Google says robots.txt rules for Googlebot affect Google Search, rules for Storebot-Google affect all Google Shopping surfaces, and user agent strings can be spoofed.
“Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping”
Google says blocking Google-Extended controls whether your content trains or grounds Gemini, and does not affect inclusion or ranking in Google Search.
“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”
TechnologyChecker, citing Cloudflare Radar, says the 403 share of AI bot requests rose from 5.98% in Q3 2025 to 13.47% in Q3 2026, and advises blocking training bots while allowing search bots.
“block ClaudeBot for training, allow Claude-SearchBot for search.”
CrunchJunkie, a log analysis vendor, found 11.6% of 3,695 AI crawler fetches to one of its own sites got a 403 or 406 though robots.txt allowed them, and 29 of the blocks hit live-answer crawlers.
“Blocking training bots is a policy decision. Blocking live-answer crawlers is almost always an accident.”
Search Engine Roundtable reports two sites that blocked bots at Cloudflare or the firewall. One lost Google crawling, Merchant Center listings and organic traffic while Google Ads kept spending. The other lost its SEO.
“This completely blocked Google from crawling the site.”
On one client site, Brett Ridenour found Cloudflare's per-bot AI blocks were limited to pages with ads, so on a site with no ads they blocked nothing.
“This site had no ads. So the toggles were doing nothing.”
Relevant Audience, an agency selling technical reviews, says to check CDN logs for 403s to Googlebot and Bingbot and Search Console's sitemap report, but not to change Cloudflare settings on one unverified report.
“Read your logs first and act on what they actually show.”
Cloudflare's AI Crawl Control shows each AI crawler's allowed and unsuccessful requests, where failures can come from any rule or error. Blocking a crawler there creates a WAF custom rule.
“Unsuccessful requests may come from any rule or response error, not just the block action in AI Crawl Control.”
Depict, which sells Shopify search apps, says compare sessions and orders by country to size bot traffic. Shopify's bot filter is a floor; if bots are 30% or more, filter traffic before Shopify, as with Cloudflare.
“Treat its number as your floor, not your total.”
Help Net Security reports that from September 15, 2026, Cloudflare would block Training and Agent crawlers on pages with ads by default for new domains, while allowing Search crawlers.
“Training and Agent crawlers will be blocked by default on pages that display ads, while Search crawlers will remain allowed by default.”
Cloudflare said that from September 15, 2026, customers who chose to block AI training crawlers would also block multi-purpose crawlers such as Googlebot, Applebot and BingBot.
“multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer