LLM Crawler User Agents: The Complete 2026 List of AI Bots, Tokens and IP Ranges

TL;DR

• LLM crawler user agents fall into four groups: training crawlers, search-indexing crawlers, user-triggered fetchers, and agents, plus two robots.txt tokens (Google-Extended, Applebot-Extended) that never appear in logs.
• Match bots on their token (GPTBot, ClaudeBot, PerplexityBot), since vendors change version numbers and browser prefixes without notice.
• Training and search crawlers from the major vendors follow robots.txt, while most user-triggered fetchers say they may ignore it because a person asked for the page.
• A user-agent string can be faked by anyone, so the published IP range is the check that proves a hit is real.
• Promptwatch's Agent Analytics logs every AI crawler hit from your CDN, verifies it against OpenAI, Anthropic, Google, Perplexity and Mistral IP ranges, and shows which pages each bot fetched and what status code it got.

This page lists the LLM crawler user agents that matter in October 2026, grouped by the job each bot does, with the exact string the vendor publishes, the robots.txt token to use, how the vendor says it treats robots.txt, and where to find its IP ranges. Every entry was checked against the vendor's own documentation on October 19, 2026. Where a vendor publishes nothing, the table says so.

If you need the background first (what an AI crawler is, why training and retrieval bots behave differently, and what 30 days of real crawl traffic looks like), start with what is an AI crawler. For bots outside the AI majors, Promptwatch's bot directory profiles more than 400 crawlers, monitors and agents.

How are LLM crawler user agents grouped?

LLM crawler user agents are grouped by job, because the job decides what blocking the bot changes. Most large vendors now run a separate bot for each job, each with its own robots.txt token.

• Training crawlers collect pages to train future models. Blocking one keeps your content out of that vendor's training data.
• Search-indexing crawlers build the index an AI search product cites from. Blocking one removes your pages from that product's cited answers.
• User-triggered fetchers load a specific page because a person asked an assistant about it. Most vendors say robots.txt may not apply to them.
• Agents browse and act on pages for a user. Several identify themselves with cryptographic signatures.
• Opt-out tokens are robots.txt names with no crawler behind them. They control how data already collected by another bot gets used.

A few practical rules apply across the tables below. Vendors update version numbers (GPTBot is at GPTBot/1.4 in OpenAI's current example) and browser prefixes (Chrome/W.X.Y.Z), so detection should match on the token. Anthropic publishes tokens only, with no full strings. And a token in a log line proves nothing on its own until the IP checks out.

Training crawler user agents

Training crawlers are the bots most sites decide on first. They follow robots.txt at every vendor that documents one.

Two notes on this group. CCBot builds Common Crawl's open dataset, which many AI labs train on, so blocking it limits training use by labs you will never see in your logs. Amazonbot also feeds Amazon products beyond training, which is why Amazon added the noarchive option for sites that want to stay crawled but excluded from model training.

Search-indexing crawler user agents

Search-indexing crawlers decide whether your pages can be cited in AI search answers. Blocking one of these is the change that removes a site from that product's answers.

Googlebot and Bingbot appear here because both feed AI answers. Google AI Overviews and AI Mode draw on Googlebot's index, and Microsoft uses Bingbot for Copilot. Microsoft controls Copilot use through page-level meta tags: NOCACHE limits Copilot to your URL, title and snippet, and NOARCHIVE keeps the page out of Copilot answers and training. Google-CloudVertexBot only crawls sites whose owners asked for it to build Vertex AI agents.

Promptwatch's verified traffic data shows how concentrated this group is. In the week of June 8 to 14, 2026, OpenAI accounted for 94.8% of IP-verified AI crawler requests across Promptwatch's connected logs. By August 31 to September 6 that share had fallen to 79.8%, as other providers picked up crawl volume (AI crawler traffic report, September 9, 2026).

User-triggered fetcher user agents

User-triggered fetchers load one page because a person asked about it, mid-conversation. This is the group where vendor robots.txt policies diverge the most.

Anthropic, Mistral and DuckDuckGo are the exceptions that apply robots.txt to their user fetchers. Disallowing Claude-User stops Claude from retrieving your pages when a user asks about them. Google-GeminiNotebook replaced Google-NotebookLM, which Google says it supported until August 2026, so older log filters may miss it.

AI agent user agents

AI agents browse, click and fill forms for a user. Two of the largest identify themselves through HTTP message signatures (Web Bot Auth, RFC 9421).

Because ChatGPT agent runs in a standard browser, a plain user-agent filter will count it as human traffic. Signature checks are the only documented way to separate it, and CDNs label it differently today (Cloudflare uses chatgpt-agent, Vercel uses chatgpt-operator).

Which AI user agents are robots.txt tokens only?

Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind them, so they will never appear in a server log.

• Google-Extended controls whether content Google has already crawled can be used for Gemini training and for grounding in Gemini Apps and Vertex AI. Google states it has no effect on inclusion or ranking in Google Search, and AI Overviews run on Googlebot's index.
• Applebot-Extended controls whether pages crawled by Applebot can be used for Apple's AI models. Apple's documentation says "Applebot-Extended does not crawl webpages." Applebot itself (...(Applebot/0.1; +http://www.apple.com/go/applebot)) follows robots.txt, ignores Crawl-delay, and verifies through *.applebot.apple.com reverse DNS.

Any log line or guide that shows a full user-agent string for either token is wrong.

Which AI bots have no official documentation?

Several AI user agents show up in real logs without any vendor page behind them. Promptwatch tracks them in its directory because they appear in customer logs, but none can be verified against a published IP range.

• xAI (Grok): Promptwatch's directory profiles GrokBot, Grok Search and Grok DeepSearch as observed in logs. xAI publishes no crawler documentation.
• DeepSeek: DeepSeek's EU public summary of training content says its crawler is "designed to respect robots.txt instructions," but publishes no user-agent name or IP range.
• ByteDance (Bytespider): widely blocked by name, with no official ByteDance documentation.
• Cohere: Cohere's documentation states it does not use bots or user agents to crawl or scrape web content. Requests labeled cohere-ai cannot be tied to Cohere through any official source.
• Retired names: anthropic-ai and claude-web no longer appear on Anthropic's crawler page, and FacebookBot no longer appears on Meta's. Neither company has published a deprecation notice, so keeping them in robots.txt does no harm.

Block undocumented bots at the CDN or firewall if you want them gone. A robots.txt line only works on bots that choose to read it.

How do you verify a request is really from an AI crawler?

You verify an AI crawler by checking the request's source IP against the vendor's published ranges, because the user-agent header is a line of text any script can copy. The scale of impersonation is large: since Promptwatch started checking crawler visits against published IP ranges on August 19, 2026, verified traffic has come out roughly 18% lower than the unverified count across accounts. For Google's desktop crawling agent, 87.7% of claimed volume failed verification in one representative week, and 99.98% of that failed volume came from Google Cloud's customer-assignable IP ranges (AI crawler verification, August 2026).

Use these checks in order:

1. Match the token in the user-agent string (for example GPTBot, PerplexityBot).
2. Check the IP against the vendor's JSON file. OpenAI, Anthropic, Google, Perplexity, Mistral, DuckDuckGo, Amazon, Apple and Common Crawl publish ranges (see the tables above).
3. Run reverse DNS with forward confirmation where the vendor documents a hostname: crawl-*.googlebot.com for Google's common crawlers, *.applebot.apple.com for Applebot, *.crawl.commoncrawl.org for CCBot.
4. Check Web Bot Auth signatures for agents that sign requests (ChatGPT agent, some Google-Agent traffic).

Meta, xAI, ByteDance and Mistral's training crawler publish no ranges, so their traffic can only be identified by user agent. Treat those counts as unverified. Google also moved its IP files to developers.google.com/static/crawling/ipranges/ in 2026, so scripts that still point at googlebot.json should be updated.

robots.txt templates for AI crawlers

These three templates cover the decisions most sites make. Each one uses the tokens from the tables above.

Block training, stay citable in AI search. This keeps search-indexing bots and user fetchers in while opting out of model training:

# Training crawlers and opt-out tokens
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: CCBot
Disallow: /

# Everything else, including OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, meta-webindexer, ChatGPT-User and Claude-User
User-agent: *
Allow: /

Allow every AI bot explicitly. Useful when a CMS or security plugin added blanket AI blocks you want to override:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: DuckAssistBot
User-agent: Amzn-SearchBot
Allow: /

Block all documented AI bots. Use the same token list with Disallow: /. Expect ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User and Google's user-triggered fetchers to keep loading pages a person asks about, since their vendors say robots.txt may not apply. Enforcing a full block takes a CDN or firewall rule.

Three details trip people up. Google-Extended and Applebot-Extended block nothing from being crawled, so pair them with the crawler rules you actually want. Amazonbot sits outside the training template because it also powers Alexa and shopping features, and Amazon's noarchive meta tag is the narrower training opt-out. And robots.txt changes take time to land: up to 24 hours for PerplexityBot and 72 hours for DuckAssistBot, per their docs. Promptwatch's AI crawler robots rules generator builds these files for you.

How do you find AI crawler user agents in your server logs?

You find AI crawlers in raw logs by searching the user-agent field for each bot's token, then counting hits per bot. A single command covers the documented AI user agents:

grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|meta-webindexer|Google-Agent|Google-CloudVertexBot|Google-GeminiNotebook|GoogleOther|Amazonbot|Amzn-SearchBot|Amzn-User|Applebot|MistralAI-[A-Za-z]+|DuckAssistBot|CCBot|Bytespider" access.log | sort | uniq -c | sort -rn

That gives you claimed volume per bot. IP verification, the link to cited pages, and new bot names all have to be added by hand, and vendors add and rename bots regularly (Google-NotebookLM became Google-GeminiNotebook in 2026). For a step-by-step walkthrough of manual and tool-based methods, see how to see which AI bots are crawling your website.

Promptwatch automates this step. AI crawler logs in Agent Analytics stream from Akamai, AWS CloudFront, Cloudflare, Fastly, Google Cloud CDN, Netlify, Vercel or WordPress, or from any source through Manual Setup. Each hit from a known AI crawler or search bot is stored as one 10-field record (timestamp, status code, method, path, query string, content type, client IP, hostname, user agent and referrer), and human traffic is discarded before anything is saved. Verified crawlers carry a shield badge, and dashboards count verified traffic by default. Crawler logs are included from the Professional plan ($245/mo, 25M log lines) and on every Agency tier.

FAQ

What is the user agent string for GPTBot?

OpenAI's current example is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. The version number changes over time, so match on the GPTBot token and confirm the IP against openai.com/gptbot.json.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended only controls Gemini training and grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode use Googlebot's index, so the only way to leave them is to restrict Googlebot or use snippet controls, which also affects regular Search.

Which AI user agents ignore robots.txt?

User-triggered fetchers are the main group. OpenAI says robots.txt "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores" it, Google says the same of Google-Agent and its other user-triggered fetchers, Meta says meta-externalfetcher "may bypass" it, and Amazon says Amzn-User "may not follow all" directives. Anthropic's Claude-User and Mistral's MistralAI-User do follow robots.txt.

Why does Anthropic only publish tokens for ClaudeBot, Claude-User and Claude-SearchBot?

Anthropic's crawler documentation lists the three bot names and one shared IP file, claude.com/crawling/bots.json, without full user-agent strings. Detect them by token and verify the IP against that file. ClaudeBot also honors the non-standard Crawl-delay directive.

How often do AI crawler user agents change?

Often enough that static lists go stale within months. In 2026, Google renamed Google-NotebookLM to Google-GeminiNotebook and moved its IP range files to a new path, while OpenAI now documents four bots (including OAI-AdsBot) and Mistral three (MistralAI-User, MistralAI-Index and MistralAI-Training).

Can a bot fake an AI crawler user agent?

Yes. Any script can send a GPTBot or Googlebot user-agent header. Only the source IP, reverse DNS where documented, or a Web Bot Auth signature proves the request came from the vendor.