Citation Crawlers vs Training Crawlers

Citation crawlers and training crawlers are two kinds of AI bots that visit websites for different reasons. Training crawlers collect content that may be used to train future AI models. Citation crawlers fetch content so that an AI search product can find, quote and link to a page in its answers. Most AI companies run separate bots for each job with separate robots.txt tokens, so a site can allow one and block the other.

Citation crawlers come in two forms. Search index crawlers, such as OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and PerplexityBot, crawl the web in advance to build the index an AI search feature retrieves from. User-triggered fetchers, such as ChatGPT-User, Claude-User and Perplexity-User, request a specific page at the moment a user's question needs it. Training crawlers include GPTBot, ClaudeBot and Common Crawl's CCBot. The AI crawlers and OpenAI crawlers entries list more.

The providers document the split explicitly. OpenAI's crawler documentation says each setting is independent: a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep its content out of training. Sites that block OAI-SearchBot are left out of ChatGPT search answers, apart from plain navigational links. OpenAI also notes that ChatGPT-User acts on behalf of a user, so robots.txt rules may not apply to it, and that it does not decide search inclusion. Anthropic's help center describes the same three roles for ClaudeBot, Claude-SearchBot and Claude-User, and says all three follow robots.txt.

Google and Apple use a different model: one crawler plus a control token. Google-Extended is a robots.txt token, not a bot. Disallowing it stops content from being used to train Gemini models and to ground answers in the Gemini app and Vertex AI, but Google's crawler documentation says it does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Search, so they follow Googlebot rules. Apple's Applebot-Extended works the same way for Apple's model training.

Mixing these up leads to the most common GEO mistake: copying a 'block AI bots' robots.txt snippet that disallows every AI user agent, including the search and user-triggered bots. That keeps content out of training but also out of ChatGPT, Claude and Perplexity answers. The opposite mistake happens too: a site allows everything in robots.txt while its CDN's bot protection returns 403 errors to OAI-SearchBot or Claude-User.

A workable policy decides the two questions separately. Citation crawlers should reach any page you want cited. Training crawlers are a business decision about licensing, competitive exposure and long-term brand presence in model knowledge. Then verify the result in your AI crawler logs: check status codes per bot, confirm requests come from the providers' published IP ranges (AI crawler verification), and watch whether citation-crawler activity rises on the pages you care about. Promptwatch's crawler data shows how quickly this traffic can change: Claude's citation crawler grew more than 100x as a share of tracked AI citation-crawler traffic between December 2025 and June 2026.