What Are AI Crawler Bots? A Complete Guide With 30 Days of Real Crawl Data

TL;DR

• AI crawler bots do one of three jobs: train future models, build a search index, or fetch a page live the moment someone asks an AI a question.
• On Homestra, a real estate marketplace using Promptwatch, ClaudeBot and ChatGPT-User together accounted for 59% of all AI crawler requests over 30 days, dwarfing every other bot combined.
• Training crawlers like ClaudeBot spread their requests across far more unique pages than live-retrieval bots like ChatGPT-User, which return to the same handful of URLs again and again.

What is an AI crawler bot?

An AI crawler bot is an automated program that fetches web pages on behalf of an AI company, the same basic mechanism as a traditional search engine crawler, but built to feed a large language model instead of (or in addition to) a search index. Each one identifies itself with its own user-agent string, the text a server sees in the request header, so site owners can tell them apart in their logs.

Not every AI crawler does the same job. Lumping them together as "AI bots" hides the decision that actually matters: whether to let a given crawler in, and why.

The three jobs an AI crawler bot can do

Every major AI crawler bot falls into one of three categories, and the category determines what blocking or allowing it actually changes.

1. Training crawlers collect content to improve a future version of a model. GPTBot (OpenAI), ClaudeBot (Anthropic), and meta-externalagent (Meta) are the best-known examples. Blocking one of these stops your content from shaping that company's next model; it has no effect on whether your brand shows up in that company's live AI search results today.
   
2. Search-indexing crawlers build the index an AI search product draws on when answering a question, closer to how Googlebot works for Google Search. OAI-SearchBot (OpenAI), PerplexityBot (Perplexity), and meta-web-indexer (Meta) fall here. OpenAI's own publisher documentation says a site has to allow OAI-SearchBot to be included in ChatGPT's search summaries and snippets; a disallowed page can still surface as a bare link and title if OpenAI encounters its URL elsewhere, unless the page also carries a noindex tag.
   
3. Live-retrieval bots fetch a specific page the instant a real user's question requires it, mid-conversation. ChatGPT-User (OpenAI), Claude-User (Anthropic), and Perplexity-User all work this way. These are triggered by an actual person asking an actual question, not a scheduled crawl, which is why their behavior in your logs looks so different from a training bot's, as the Homestra data below shows.

The practical upshot: a site can opt out of AI training while staying fully eligible to be cited in AI answers, because the training bot and the retrieval/search bot for the same company are usually two separate user-agents with two separate robots.txt directives.

The AI crawler bots that actually show up in server logs

Promptwatch maintains a directory of every AI crawler, search bot, and automated agent currently active on the web, more than 400 at last count, since new ones appear faster than any single guide can track. In practice, a handful of user-agents account for almost all AI crawler traffic on most sites:

Knowing the name is only half the job. What matters more is what these bots actually do once they land on a real site, which is where most published crawler guides stop at theory.

What 30 days of real AI crawler traffic actually looks like

To ground this in reality rather than another list of user-agents, we pulled Promptwatch's crawler log data for Homestra, a European real estate marketplace, covering 30 days from August 3 to September 1, 2026. The site logged 71,060 AI crawler requests in that window, an average of about 2,369 a day.

That average hides a lot of movement. Daily volume ranged from roughly 1,000 requests on the quietest day to over 5,400 on the busiest, a more than fivefold swing. AI crawler activity doesn't arrive at a steady drip; it comes in bursts as different bots decide it's time to revisit the site.

Which bots showed up, and how often

Two crawlers dominated the traffic, together accounting for 59% of everything AI crawlers did on the site:

• ClaudeBot: 30.3%
• ChatGPT-User: 28.5%
• meta-externalagent: 12.7%
• GPTBot: 11.9%
• PerplexityBot: 7.3%
• OAI-SearchBot: 6.1%
• Everything else (meta-web-indexer, Claude-User, Google-Extended, GrokBot, DeepSeek Bot, MistralAI-User, Cohere AI): about 3.3% combined

That distribution is a reminder that "AI crawlers" as a category is really dominated by two or three vendors on most sites, not a long tail of equally active bots.

Training bots and retrieval bots leave different fingerprints

The clearest evidence for the training-versus-retrieval distinction wasn't in a vendor's documentation, it was in the log itself. Looking only at requests to individual property pages: ClaudeBot, a training crawler, made 15,182 requests but touched 10,414 distinct URLs, about 1.5 requests per page, consistent with a broad, even sweep of the catalog.

ChatGPT-User, a live-retrieval bot, told a different story: 12,049 requests to property pages landed on only 4,791 distinct URLs, about 2.5 requests per page, roughly 70% higher than ClaudeBot's rate.

That pattern fits a bot answering real user questions about a smaller set of listings people are actually asking ChatGPT about, rather than sweeping the whole catalog evenly.

"In Promptwatch we are able to see how active the ChatGPT bot is on our client websites and exactly how often our content is being used in responses of LLMs." Marijn ten Bulte, Head of Organic Channels at Advise

Status codes and where the requests went

Most requests succeeded, by status code:

• 200 (success): 88.9%
• 308 (redirect): 6.8%
• 404 (not found): 3.9%

By path:

• Property pages: 57.7%
• List/search pages: 34.4%
• Everything else (blog content, Open Graph image requests, miscellaneous paths): 7.9%

Two things stood out at the crawler level.

First, meta-web-indexer, the bot that feeds Meta AI's search results, returned a 404 on 14.8% of its requests, nearly four times the site's 3.9% overall error rate.

That's worth checking in any site's own logs (Promptwatch's meta-webindexer profile has more detail on how this crawler behaves).

Second, PerplexityBot hit a 429 (rate-limited) response on 1.8% of its requests, roughly seven times the site-wide rate-limit rate, a signal that Perplexity's crawl pace occasionally outran what the server's edge was configured to accept.

Neither finding is visible from a robots.txt file or a generic "which bots exist" list. Both only show up once you're looking at AI crawler logs joined to citation and page-level data, not just raw hit counts.

Training vs. retrieval: why the distinction should drive your robots.txt

Because training and retrieval/search run through different user-agents for the same company, the decision to allow or block isn't one choice per vendor, it's one choice per job.

A brand uncomfortable with its content training future models can block GPTBot, ClaudeBot, and meta-externalagent while still allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, and Perplexity-User. That keeps the site eligible to be cited when someone asks an AI a question, without contributing to model training the brand doesn't want to opt into.

The opposite mistake, and a more common one, is blocking everything with "bot" or "AI" in the name as a blanket policy.

That removes a brand from AI-generated answers entirely, since the search-indexing and live-retrieval bots get caught in the same net as the training crawlers. Robots.txt is also a request, not a lock: a well-behaved crawler honors it, but real enforcement against a crawler that ignores it happens at the server or CDN, not in the text file itself.

"Its clear insights into share of voice, content gaps, and AI crawler activity have made it a core part of our marketing reporting stack." Sam Franklin, VP Marketing at Landytech

How to see which AI bots are actually crawling your site

Reading a list of user-agents doesn't tell you what's happening on your own domain. Three things do:

• Server or CDN logs. Akamai, Cloudflare, Fastly, AWS CloudFront, Google Cloud CDN, Netlify, and Vercel all expose raw request logs that record every crawler hit, including the user-agent, path, and status code, the same fields used in the Homestra breakdown above. Promptwatch has native integrations for all of these, so the logs stream straight in rather than needing to be exported and parsed by hand.
• A crawler-log integration that joins that raw activity to citation and page-level data, so a spike in ClaudeBot traffic to a specific page can be connected to whether that page is actually getting cited, not just crawled.
• IP verification, since user-agent strings can be spoofed. OpenAI, Anthropic, and Google all publish verified IP ranges for their crawlers, which is the more reliable check when a hit looks suspicious.

Connecting a CDN takes minutes and requires no code changes; any internal IT or security sign-off, where a company's policy requires it, tends to run on that team's own timeline rather than the technical setup itself.

Connecting a CDN takes minutes and requires no code changes; any internal IT or security sign-off, where a company's policy requires it, tends to run on that team's own timeline rather than the technical setup itself.

Frequently asked questions

What's the difference between an AI crawler bot and a regular search engine bot?

Mechanically, they're similar: both fetch pages and follow links. The difference is purpose. Googlebot builds a search index for Google Search. An AI crawler bot either trains a model, builds an index for an AI product's answers, or fetches a page live in response to a specific AI conversation.

Do AI crawler bots respect robots.txt?

The major, named crawlers from OpenAI, Anthropic, Google, and Meta document their user-agents and generally honor robots.txt directives. Robots.txt is a convention a well-behaved bot chooses to follow, not a technical barrier, so pairing it with server or CDN-level blocking is more reliable against anything that ignores it.

If I block GPTBot, does that stop my brand from appearing in ChatGPT answers?

No. GPTBot is OpenAI's training crawler. Whether ChatGPT includes your content in search-style summaries and snippets depends on OAI-SearchBot, a separate, independently-controlled user-agent; OpenAI's own publisher guidance confirms blocking one has no effect on the other.

Why does one AI crawler bot request the same page multiple times?

It depends which kind of bot it is. A live-retrieval bot like ChatGPT-User revisits a page each time a real person's question makes that page relevant, which is why popular pages get hit repeatedly. A training crawler like ClaudeBot is doing a broader, more even sweep, so its repeat rate per page tends to be lower.