All answers

The Prevailing
Answers for
Business model:
Business model, pick up to two

Should we let Meta's crawlers read our product and landing pages under the revised Commercial Terms, or limit them with robots.txt?

The PrevailingUpdated Nov 9, 2026

Our answer

Allow Meta-ExternalAds on your product and landing pages if you advertise. Set Meta-ExternalAgent, the AI-training crawler, separately in robots.txt. The paused revised terms limit crawling to ads and business products, and Meta says robots.txt still applies.

Confidence
Medium
Strongest proof
10 Meta rules
2 datasets, 3 practitioners, 7 opinions or claims
Sources
13 sources
22 statements

New here? Take the 1-minute tour

Why this is the answer

3 reasons and our judgment

Ad crawling has its own crawler and a narrow purpose. Meta's revised terms, in both US and UK English, limit crawling to advertising and business products plus integrity and security. Its crawler page gives ad crawls their own token, apart from Meta-ExternalAgent for AI training.

Blocking the ads crawler can cost you. Blocking landing or catalogue pages may affect the product information Meta pulls, Relevant Audience warns. Temso, an AI search tracking software maker, argues that blocking ad bots can break ad approval.

Robots.txt is a stated preference, not a contract term. Meta's Daniel Roberts says robots.txt preferences still apply (Social Media Today), but the terms never mention robots.txt. Legal experts quoted by Search Engine Journal say Meta could argue the terms supersede blocking.

Our judgment

We weighed the narrow ad purpose in the terms, the risk to product information and ad approval, and the ad terms already in force, which let Meta crawl advertisers' domains (Meta, Search Engine Journal), and came down on letting the ads crawler in. Temso's ad-approval point is a general statement with no Meta source behind it. Meta-ExternalAgent is a separate choice about AI training, since Meta gives it its own token and purpose and a rule for one crawler does not cover the other. We would not rely on robots.txt as a legal limit on what you grant by continued use, because the terms are silent on it and only Meta's spokesperson says it still applies.

Our judgment, built from Meta, Relevant Audience, Temso, Search Engine Journal, Promptwatch and Dark Visitors: our reading of these sources, not a finding from any one of them.

What the sources say

22 statements from 13 sources
  • Common Thread CollectiveNov 20, 2026Supports

    Common Thread Collective, a Meta ads agency, reports Meta paused new ad terms due November 20, 2026. The old terms already let Meta crawl advertisers' sites, and Advantage+ features can be turned off one by one.

    “Meta confirmed advertisers can opt out of individual Advantage+ features at any time from within Ads Manager.”

    Read it on commonthreadco.com

  • MetaNov 18, 2026Caveat

    Meta says it put proposed changes to its commercial terms on hold after publishers raised concerns about how they apply to its web crawlers. It plans to consult publishers and regulators in early 2027 before any rollout.

    “the proposed updates to our commercial terms of service have been put on hold”

    Read it on about.fb.com

  • PPC LandNov 8, 2026Caveat

    Meta paused its revised ad and commercial terms that were due on November 20, 2026, with no new date, and says its existing terms stay in effect. It has not tied the pause to recent court or regulator moves.

    “In the meantime, our existing terms remain in effect”

    Read it on ppc.land

  • ReutersOct 30, 2026Caveat

    Reuters reports the UK competition regulator has opened an investigation into Meta's revised terms, which it says are on hold, to decide whether they abuse a dominant position.

    “could amount to an unlawful imposition of unfair trading conditions.”

    Read it on yahoo.com

  • Press GazetteOct 30, 2026Caveat

    Press Gazette reports Meta has delayed the terms and pledged not to impose them before a full consultation with publishers, and a Norwegian court has barred them in Norway until it reviews them.

    “appear to give it sweeping rights to access publisher content”

    Read it on pressgazette.co.uk

  • The UK's CMA opened an investigation into whether Meta's commercial terms abuse a dominant position. Meta has paused the new terms pending consultation, and no finding has been made.

    “The CMA will examine whether Meta is abusing a dominant position by imposing exploitative terms on publishers and advertisers.”

    Read it on gov.uk

  • ReutersOct 28, 2026Caveat

    Meta told Reuters on October 28 that it will not change its terms without a full consultation with industry groups, which could run past November 20; until then its existing terms stay in place.

    “Until then our existing terms remain in place.”

    Read it on wsau.com

  • Social Media TodayOct 25, 2026Supports

    Social Media Today, relaying Reuters and others, says the terms moved from October 30 to November 20, and Meta says they consolidate existing terms and businesses can still block crawlers through robots.txt.

    “largely a misunderstanding”

    Read it on socialmediatoday.com

  • PPC LandOct 23, 2026Caveat

    Meta told Reuters it was moving the revised terms from October 30 to November 20 to explain what is changing; PPC Land notes Meta did not promise any edits, only that robots.txt preferences still apply.

    “Meta did not say it would amend anything”

    Read it on ppc.land

  • PPC LandOct 23, 2026Supports

    PPC Land reports a Meta spokesperson confirmed the November 20 date to Reuters. The training sentence was gone that evening, while the crawling clause, test credentials and lack of a robots.txt commitment stayed.

    “explain to our partners and stakeholders what's changing and what's not”

    Read it on ppc.land

  • Common Thread CollectiveOct 23, 2026Disagrees

    Pushes back on relying on robots.txt: CTC, an agency, says the delay changes only the deadline and that robots.txt is no guaranteed opt-out, and advises checking and updating robots.txt before November 20.

    “Meta has stated that businesses can signal preferences via robots.txt, but this is not a guaranteed opt-out.”

    Read it on commonthreadco.com

  • Dark VisitorsOct 20, 2026Caveat

    Pushes back on simply allowing the ads crawler: Dark Visitors finds 6% of the top 1000 websites block meta-externalads, and shopping sites are the category it visits most.

    “Weigh any benefit against sharing business or page-level data with its operator or clients.”

    Read it on darkvisitors.com

  • Dark VisitorsOct 20, 2026Supports

    Dark Visitors finds 19% of the top 1000 websites block meta-externalagent, and says allowing it may raise the chance your brand appears in AI answers; search rankings are unaffected either way.

    “Block meta-externalagent if you want more control over whether your work is used for AI training.”

    Read it on darkvisitors.com

  • MetaOct 19, 2026Supports

    Meta's revised terms, in US and UK English, let its crawlers and test credentials read pages tied to your products for advertising and business products and for integrity and security. They never mention robots.txt.

    “in order to provide, develop, improve, research and maintain advertising and business products”

    Read it on facebook.com

  • Social Media TodayOct 17, 2026Supports

    Meta spokesperson Daniel Roberts says the crawling terms apply only to ads and business products, that content owners can still use robots.txt to signal preferences, and that the terms do not override them.

    “these updates don't give us new rights that supersede those preferences.”

    Read it on socialmediatoday.com

  • Relevant AudienceOct 17, 2026Supports

    Relevant Audience says to set a rule for each Meta crawler: blocking facebookexternalhit can break link previews on posts and ads, and blocking landing or catalogue pages may affect product information Meta pulls.

    “A blanket decision is rarely right; many sites will want different rules for product pages and for editorial or gated content.”

    Read it on relevantaudience.com

  • Nieman LabOct 17, 2026Supports

    Meta says it revised its commercial terms to make clear robots.txt preferences are respected and the update gives it no new rights over them, though the terms text never mentions robots.txt.

    “These updates don't give us new rights that supersede those preferences”

    Read it on niemanlab.org

  • Search Engine JournalOct 16, 2026Supports

    Search Engine Journal says to review how robots.txt treats each Meta crawler before the terms take effect, and relays legal experts who say the wording lets Meta argue the terms supersede blocking.

    “Neither the updated Commercial Terms nor the updated Self-Serve Ad Terms mention robots.txt.”

    Read it on searchenginejournal.com

  • PromptwatchSep 6, 2026Supports

    Promptwatch, a tool vendor, says allowing meta-externalads gives Meta access for advertising and business purposes only, and blocking it sets no rule for Meta-ExternalAgent.

    “Meta does not publish a detailed breakdown of what each visit is meant to evaluate.”

    Read it on promptwatch.com

  • TemsoJul 29, 2026Supports

    Temso, a tool vendor, gives the robots.txt rule to block meta-externalads but warns that advertisers usually need to let a platform's ad bots in.

    “If you run ads on a platform, its ad bots usually need access; blocking them can break ad approval and quality checks for your own campaigns.”

    Read it on temso.ai

  • MetaMay 21, 2026Supports

    Meta's crawler page gives each crawler its own robots.txt token: Meta-ExternalAds for advertising and business products, Meta-ExternalAgent for uses such as AI training. Changes can take up to 24 hours.

    “might bypass robots.txt when performing security or integrity checks”

    Read it on developers.facebook.com

  • MetaAug 22, 2025Caveat

    Says the access is not new for advertisers: Meta's ad terms in force since August 22, 2025 already let Meta crawl your domains when you place an order, including for research and training.

    “By placing an Order, you grant us the right to access, index, cache, analyze or crawl the content available through your domains, webpages or URLs”

    Read it on facebook.com

In the full answer

For readers who sign in
  • What to do this week3 steps
  • The numbers behind it3 figures from cited sources
  • Where each source standsA chart placing each source on the question
  • The strongest disagreementThe best case against the answer, from a cited source
  • How sure we areWhat its confidence rests on, and what would change the answer

Read next

More in Team, tools and account risk