Definition
AI crawlers are automated clients run by AI companies to fetch web content. Some collect pages for training future models, some build the search indexes that AI answers draw on, and some fetch a single page live because a user's question needs it. Each purpose usually has its own user agent, so a site can allow one and block another (see citation crawlers vs training crawlers).
The main operators document these bots as of October 2026:
- OpenAI runs GPTBot for training, OAI-SearchBot for the ChatGPT search index, ChatGPT-User for fetches users trigger in ChatGPT (robots.txt rules may not apply) and OAI-AdsBot to check landing pages submitted as ChatGPT ads.
- Anthropic runs ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for fetches made while answering a user, and says all three honor robots.txt.
- Google crawls with Googlebot for Search, including AI Overviews and AI Mode. Google-Extended is a robots.txt token rather than a crawler; it controls use of content in Gemini training and grounding without affecting Search. Google-Agent fetches pages for agents running on Google infrastructure.
- Perplexity runs PerplexityBot for its search index, which it says is not used to train foundation models, and Perplexity-User for user requests, which generally ignores robots.txt.
- Microsoft runs Bingbot, whose index grounds Copilot and Bing's AI answers.
- Apple runs Applebot for Siri, Spotlight and Safari, with Applebot-Extended as a robots.txt token for opting out of foundation model training.
- Meta documents Meta-ExternalAgent for training and indexing, Meta-WebIndexer for Meta AI search and Meta-ExternalFetcher for user requests.
Automatic crawling by these bots generally follows robots.txt. User-triggered fetchers are the exception: OpenAI's crawler documentation, Google and Perplexity all say robots.txt may not apply to fetches a person initiated. In practice, many blocks happen elsewhere. CDN bot protection and firewall rules return 403s that override whatever robots.txt says. Most AI crawlers also read the HTML they receive without running JavaScript the way a browser does, so content rendered only on the client can be invisible to them (JavaScript rendering); Applebot is one that may render pages.
Access is a business decision made per purpose. Blocking training crawlers keeps new content out of future models but leaves AI search visibility intact, since OpenAI, Anthropic, Perplexity, Google and Apple all separate the controls. Blocking search crawlers or user fetchers removes a site from those platforms' answers, which for most brands defeats the point. Publishers that want payment for training use can look at pay-per-crawl schemes and machine-readable usage preferences such as content signals. The imbalance is measurable: Cloudflare reported in 2025 that major AI crawlers fetched anywhere from about 118 to nearly 50,000 pages for every visitor their platforms referred back.
Track what actually happens with AI crawler logs, and verify identities before trusting the counts (AI crawler verification). Our guide to AI crawler bots lists the user-agent strings to match, and the Promptwatch AI crawler traffic data shows how the mix of crawlers changes week to week.
Examples of AI Web Crawlers
- A news publisher disallows GPTBot, ClaudeBot and Google-Extended to keep its archive out of model training, but leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot open so its stories still appear in AI search answers.
- A SaaS company copies a 'block AI bots' robots.txt template that also disallows OAI-SearchBot, and its ChatGPT search citations disappear within days.
- An ecommerce site finds Perplexity-User fetching product pages that show prices only after client-side JavaScript runs, and moves price and stock into the server-rendered HTML.
- A B2B company sees Meta-WebIndexer appear in its logs in July 2026 and adds an explicit robots.txt group for it instead of leaving it to the wildcard rule.
