What you actually block when you block AI crawlers

Someone on your team has probably already added a rule to robots.txt to keep AI out of your content. It is a reasonable instinct and it is quietly expensive, because on 2 of the 3 platforms that document a training crawler at all, the agent that trains on your pages is the same agent that decides whether you appear in the answer.

What does blocking AI crawlers actually control?

On 8 September 2026 we read the published crawler documentation at Google, OpenAI, Perplexity and Meta, and recorded every named fetch path with the sentence stating what it is for. That produced 29 named paths, sorted into buckets fixed before we started reading.

• Model training. The platform says the fetch may be used to train or improve its models
• Live answer retrieval. The platform says the fetch feeds a search index or grounds an answer shown to a user
• User-triggered fetch. The platform says the fetch happens because a person pasted or requested that specific URL
• Operational. Verification, link previews, ad safety checks and internal research

Of the 29 named paths, 3 of 29 serve model training and 9 of 29 serve live answers or a search index. Not one path left its purpose unstated. If you want the citation side of this, we covered how Perplexity picks what it cites separately.

Which named agents decide whether you appear?

The bucket that matters commercially is the second one. These are the agents doing the reading when an AI answers a question about you, and each is controlled by its own line in Google's list of common crawlers and its equivalents.

Why does blocking Google cost more than blocking OpenAI?

Because OpenAI splits the decision and Google does not. OpenAI's crawler overview states the two tokens are independent, and that a site can allow one while refusing the other.

"Each setting is independent of the others, for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training"

Google bundles them. Its documentation describes Google-Extended as covering both Gemini training and grounding, the step that supplies content to the model at prompt time, so one token carries two consequences. Meta does the same with Meta-ExternalAgent. Perplexity's crawler page goes the other way and documents no training crawler at all.

"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search"

Does robots.txt stop an AI from reading your page?

Not when a person asks for the page by name. All 4 of 4 platforms document at least one user-triggered agent that may ignore robots.txt, and this is the most misunderstood line in the whole set. It also explains how a page can be missing from automated answers while staying readable on request, which is one of the two failures behind absence from AI answers.

The Meta wording comes from Meta's web crawler documentation, and the other three say the equivalent in their own pages. A robots file governs automatic crawling, not a human request.

What does a block actually cost you?

This is the table to take into a meeting. It is our reading of what each platform's own documentation says you give up, not a claim any platform frames this way.

• User-triggered paths are the largest bucket in the set, at 12 of 29
• A disallow aimed at everything hits agents you needed and misses ones you meant to stop
• The absence of a training crawler at Perplexity is a documented absence, not an oversight we inferred

What should go in your robots.txt?

Decide the two questions separately, then write the lines. What follows is our recommendation, not a rule any platform publishes.

• 1

  Decide whether you object to model training, which is a commercial and legal question rather than an SEO one

• 2

  Decide whether you want to be present in AI answers, which for most businesses is a yes

• 3

  Write a separate group per agent, never a blanket rule aimed at everything

• 4

  Price the blanket disallow before shipping it, using the cost table above

That pair refuses training on your content while keeping you eligible for ChatGPT search answers, and OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. If you want the same reasoning applied to your own file, our AI visibility audit covers exactly this.

What this does not tell you

This measures documentation, not behaviour. A platform saying a crawler does something is evidence about stated policy, and we did not observe a single crawler to confirm it. Three further limits are worth printing plainly.

• Anthropic is out of frame, because its crawler guidance for site owners sits on a host outside our research list. That is a limit of our frame, not a finding about Anthropic
• Google's total is a floor. We read its common crawlers and user-triggered fetchers pages, not its special-case page
• Four platforms is a small sample, and we state it as four rather than dressing it up as an industry
• Our 13 August post counted 18 named paths across these same four platforms and this one counts 29. The difference is mainly scope, since that study did not read Google's user-triggered fetchers page, which alone carries 9 of the paths counted here. We did not diff the pages against their August versions, so we are not claiming the platforms added the rest

Permission is also a separate question from rendering, which we handled in whether AI crawlers execute JavaScript.