GPTBot and AI crawlers: what to allow and why

Two jobs, one robots.txt

AI crawlers are not monolithic. Some exist to retrieve and surface your content so an engine can cite you in an answer; others exist to gather content used to train or improve models. These are different value exchanges. The first directly affects your visibility in AI answers; the second affects whether your content contributes to a model's general knowledge, with no direct citation benefit.

You manage access to all of them in robots.txt by user-agent. The key insight is to decide per-bot based on its job, rather than reflexively blocking everything unfamiliar - a broad disallow can quietly cut you out of the very AI answers you want to appear in.

The major AI crawlers and what they do

Here is how the main agents map to outcomes. The names and behaviors evolve, so confirm current documentation, but the categories are stable: citation-oriented crawlers versus training-oriented crawlers.

• OAI-SearchBot - surfaces content for ChatGPT Search results and citations (citation-oriented).
• GPTBot - OpenAI crawler associated with model training (training-oriented).
• PerplexityBot - Perplexity's crawler for retrieval and citation in its answers.
• ClaudeBot - Anthropic's crawler for accessing web content.
• Google-Extended - controls whether your content can ground/train Google's generative features.
• Googlebot / Bingbot - classic search indexing that also underpins AI Overviews and Copilot grounding.

How to decide what to allow

Start from your goal. If you want to be cited in AI answers - which is the point of GEO - you should allow the citation and grounding crawlers for the engines you care about, and keep your classic search bots allowed since they underpin AI Overviews and Copilot. Blocking these is self-defeating for visibility.

Training crawlers like GPTBot are a genuine judgment call. Some publishers allow them to contribute to model knowledge; others restrict them over content-rights concerns. Crucially, blocking a training crawler does not, by itself, remove you from that engine's live search citations, because those are governed by the separate search crawler. Decide the two questions independently.

Implement and verify

Set rules per user-agent in robots.txt, then verify reality against intent. robots.txt is a directive that well-behaved crawlers respect, so check your server logs to confirm the bots you allowed are actually fetching pages and the ones you blocked are not. Re-check periodically, because crawler names and behaviors change. If a bot you want is not appearing in logs, that is your first GEO problem to fix - eligibility precedes everything.