TL;DR
- An AI crawler bot is an automated program that fetches web pages for an AI company, and each one does one of three jobs: train future models, build an AI search index, or fetch a page live when someone asks an AI a question.
- Training crawlers like ClaudeBot sweep a site broadly, while live-retrieval bots like ChatGPT-User return to a smaller set of pages that people are asking about.
- Because training and retrieval run through separate user-agents, a site can opt out of AI training and stay eligible to be cited in AI answers.
- Promptwatch's Agent Analytics streams AI crawler logs from your CDN, verifies each bot against its published IP ranges, and shows which pages every crawler fetched and which status code it received.
What is an AI crawler bot?
An AI crawler bot is an automated program that fetches web pages on behalf of an AI company. It works like a traditional search engine crawler, and the pages it collects feed a large language model, an AI search index, or both. Each one identifies itself with its own user-agent string, the text a server sees in the request header, so site owners can tell them apart in their logs.
AI crawlers do different jobs, and grouping them all as "AI bots" hides the decision that matters: whether to let a given crawler in, and why.
The three jobs an AI crawler bot can do
Every major AI crawler bot falls into one of three categories, and the category determines what blocking or allowing it actually changes.
- Training crawlers collect content to improve a future version of a model. GPTBot (OpenAI), ClaudeBot (Anthropic), and meta-externalagent (Meta) are the best-known examples. Blocking one of these stops your content from shaping that company's next model; it has no effect on whether your brand shows up in that company's live AI search results today.
- Search-indexing crawlers build the index an AI search product draws on when answering a question, closer to how Googlebot works for Google Search. OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), and meta-webindexer (Meta) fall here. OpenAI's own publisher documentation says a site has to allow OAI-SearchBot to be included in ChatGPT's search summaries and snippets; a disallowed page can still surface as a bare link and title if OpenAI encounters its URL elsewhere, unless the page also carries a noindex tag.
- Live-retrieval bots fetch a specific page the instant a real user's question requires it, mid-conversation. ChatGPT-User (OpenAI), Claude-User (Anthropic), and Perplexity-User all work this way. A real person's question triggers each fetch, which is why their behavior in your logs looks so different from a training bot's, as the Homestra data below shows.
The practical upshot: a site can opt out of AI training while staying fully eligible to be cited in AI answers, because the training bot and the retrieval/search bot for the same company are usually two separate user-agents with two separate robots.txt directives.
The AI crawler bots that actually show up in server logs
Promptwatch maintains a directory of every AI crawler, search bot, and automated agent currently active on the web, more than 400 at last count, since new ones appear faster than any single guide can track. In practice, a handful of user-agents account for almost all AI crawler traffic on most sites:
| User-agent | Company | Job |
|---|---|---|
| GPTBot | OpenAI | Training |
| ChatGPT-User | OpenAI | Live retrieval |
| OAI-SearchBot | OpenAI | Search indexing |
| ClaudeBot | Anthropic | Training |
| Claude-User | Anthropic | Live retrieval |
| Claude-SearchBot | Anthropic | Search indexing |
| meta-externalagent | Meta | Training |
| meta-webindexer | Meta | Search indexing (Meta AI) |
| PerplexityBot | Perplexity | Search indexing |
| Perplexity-User | Perplexity | Live retrieval |
| MistralAI-User | Mistral | Live retrieval |
| Google-Extended | Training/grounding opt-out token (never appears in logs) | |
| GrokBot | xAI | Training (no vendor documentation) |
| Grok Search | xAI | Live retrieval (no vendor documentation) |
| DeepSeek Bot | DeepSeek | Training (no vendor documentation) |
| Cohere AI | Cohere | Training (no vendor documentation) |
The last four rows are user-agents Promptwatch sees in customer logs, but their companies publish no crawler documentation, so they can't be verified against an official IP range. For the exact user-agent string, robots.txt token and IP list behind each bot, see the full list of LLM user agents.
The more useful question is what these bots do once they land on a real site, and answering it takes log data.
What 30 days of real AI crawler traffic actually looks like
To show what this looks like on a live site, Promptwatch pulled its crawler log data for Homestra, a European real estate marketplace, covering 30 days from August 3 to September 1, 2026. The site logged 71,060 AI crawler requests in that window, an average of about 2,369 a day.
That average hides a lot of movement. Daily volume ranged from roughly 1,000 requests on the quietest day to over 5,400 on the busiest, a more than fivefold swing. AI crawler activity arrives in bursts as different bots decide it's time to revisit the site.
Which bots showed up, and how often
Two crawlers dominated the traffic, together accounting for 59% of everything AI crawlers did on the site:
- ClaudeBot: 30.3%
- ChatGPT-User: 28.5%
- meta-externalagent: 12.7%
- GPTBot: 11.9%
- PerplexityBot: 7.3%
- OAI-SearchBot: 6.1%
- Everything else (meta-webindexer, Claude-User, GrokBot, DeepSeek Bot, MistralAI-User, Cohere AI): about 3.3% combined
On this site, three vendors (Anthropic, OpenAI and Meta) generated almost 90% of AI crawler requests, and the long tail of other bots added up to about 3%.
Training bots and retrieval bots leave different fingerprints
The log itself shows the training-versus-retrieval distinction clearly. Looking only at requests to individual property pages: ClaudeBot, a training crawler, made 15,182 requests but touched 10,414 distinct URLs, about 1.5 requests per page, consistent with a broad, even sweep of the catalog.
ChatGPT-User, a live-retrieval bot, told a different story: 12,049 requests to property pages landed on only 4,791 distinct URLs, about 2.5 requests per page, roughly 70% higher than ClaudeBot's rate.
That pattern fits a bot answering real user questions about the smaller set of listings people are actually asking ChatGPT about, while the training crawler works through the whole catalog.
"In Promptwatch we are able to see how active the ChatGPT bot is on our client websites and exactly how often our content is being used in responses of LLMs."
Status codes and where the requests went
Most requests succeeded, by status code:
- 200 (success): 88.9%
- 308 (redirect): 6.8%
- 404 (not found): 3.9%
By path:
- Property pages: 57.7%
- List/search pages: 34.4%
- Everything else (blog content, Open Graph image requests, miscellaneous paths): 7.9%
Two things stood out at the crawler level.
First, meta-webindexer, the bot that feeds Meta AI's search results, returned a 404 on 14.8% of its requests, nearly four times the site's 3.9% overall error rate. That's worth checking in any site's own logs (Promptwatch's meta-webindexer profile has more detail on how this crawler behaves).
Second, PerplexityBot hit a 429 (rate-limited) response on 1.8% of its requests, roughly seven times the site-wide rate-limit rate, a signal that Perplexity's crawl pace occasionally outran what the server's edge was configured to accept.
Both findings only surface in AI crawler logs joined to page-level and citation data. A robots.txt file or a list of bot names shows the rules a site set; the logs show what each bot actually received.
Training vs. retrieval: why the distinction should drive your robots.txt
Because training and retrieval/search run through different user-agents for the same company, the decision to allow or block is made per job, one user-agent at a time.
A brand uncomfortable with its content training future models can block GPTBot, ClaudeBot, and meta-externalagent while still allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, and Perplexity-User. That keeps the site eligible to be cited when someone asks an AI a question, without contributing to model training the brand doesn't want to opt into. The user-agent list linked above includes a copy-ready robots.txt template for exactly this setup.
The opposite mistake, and a more common one, is blocking everything with "bot" or "AI" in the name as a blanket policy.
That removes a brand from AI-generated answers entirely, since the search-indexing and live-retrieval bots get caught in the same net as the training crawlers. Robots.txt is also a request: a well-behaved crawler honors it, and enforcement against a crawler that ignores it happens at the server or CDN.
"Since implementing Promptwatch, we've seen a measurable increase in our LLM visibility. Its clear insights into share of voice, content gaps, and AI crawler activity have made it a core part of our marketing reporting stack."
How to see which AI bots are actually crawling your site
Three sources show what's happening on your own domain. For a step-by-step version, see the guide on how to see which AI bots are crawling your website.
- Server or CDN logs. Akamai, AWS CloudFront, Cloudflare, Fastly, Google Cloud CDN, Netlify, Vercel, and WordPress all expose request logs that record every crawler hit, including the user-agent, path, and status code, the same fields used in the Homestra breakdown above. Promptwatch has native integrations for all of these, so the logs stream straight in without being exported and parsed by hand.
- A crawler-log integration that joins that raw activity to citation and page-level data, so a spike in ClaudeBot traffic to a specific page can be connected to whether that page is getting cited.
- IP verification, since user-agent strings can be spoofed. OpenAI, Anthropic, Google, Perplexity and Mistral publish IP ranges for their crawlers, and since August 19, 2026, Promptwatch checks every crawler visit against them and leaves impostors out of counted traffic. When the check launched, 87.7% of the volume claiming to be Google's desktop crawling agent failed it.
Connecting a CDN takes minutes and requires no code changes; any internal IT or security sign-off, where a company's policy requires it, runs on that team's own timeline.
Frequently asked questions
What's the difference between an AI crawler bot and a regular search engine bot?
Mechanically, they're similar: both fetch pages and follow links. The difference is purpose. Googlebot builds a search index for Google Search. An AI crawler bot either trains a model, builds an index for an AI product's answers, or fetches a page live in response to a specific AI conversation.
Do AI crawler bots respect robots.txt?
Training and search-indexing crawlers from OpenAI, Anthropic, Google, Perplexity and Meta follow robots.txt. Live-retrieval bots are mixed: OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it, while Anthropic's Claude-User follows it. Robots.txt is a convention each bot chooses to follow, so server or CDN rules are the reliable way to block anything that ignores it.
If I block GPTBot, does that stop my brand from appearing in ChatGPT answers?
No. GPTBot is OpenAI's training crawler. Whether ChatGPT includes your content in search-style summaries and snippets depends on OAI-SearchBot, a separate, independently-controlled user-agent; OpenAI's own publisher guidance confirms blocking one has no effect on the other.
Why does one AI crawler bot request the same page multiple times?
It depends which kind of bot it is. A live-retrieval bot like ChatGPT-User revisits a page each time a real person's question makes that page relevant, which is why popular pages get hit repeatedly. A training crawler like ClaudeBot does a broader, more even sweep, so its repeat rate per page tends to be lower.
