Anthropic Crawlers (ClaudeBot, Claude-SearchBot, Claude-User)

Anthropic crawlers are the three web robots Anthropic runs for Claude: ClaudeBot collects public pages that may be used to train its models, Claude-SearchBot crawls to improve Claude's search results, and Claude-User fetches a page when a person's question to Claude needs it. Each bot has its own robots.txt token, so a site can block one without blocking the others.

Anthropic's crawler help article, last updated April 7, 2026, describes what blocking each bot does:

• ClaudeBot: restricting it signals that the site's future materials should be excluded from Anthropic's AI training datasets.
• Claude-SearchBot: disabling it stops Anthropic indexing the site for search, which 'may reduce your site's visibility and accuracy in user search results'.
• Claude-User: disabling it prevents Claude from retrieving the site's content in response to a user query, which may reduce visibility in user-directed web search.

Anthropic says all three bots honor robots.txt directives, support the non-standard Crawl-delay extension, and do not try to bypass CAPTCHAs. Rules must be set in the robots.txt file of every subdomain a site wants to opt out. The split mirrors the training, search and user-fetch separation OpenAI uses for its own bots (see OpenAI crawlers).

The names have changed over time. Anthropic previously crawled as anthropic-ai and Claude-Web. In July 2024 the company told 404 Media that both user agents were no longer in use and that ClaudeBot had been configured to respect rules written for them. Anthropic's current page lists only the three bots above, so robots.txt files that still name only the old tokens should be updated to name the current ones explicitly.

User agent strings are easy to fake, which is why AI crawler verification matters. Anthropic's help page links a machine-readable list of its crawler IP addresses at claude.com/crawling/bots.json. The version dated October 7, 2026 contained 38 IPv4 prefixes and does not say which bot uses which range, so a match proves a request came from Anthropic but not from which crawler. Anthropic also warns that blocking its IP addresses may not work as an opt-out, because it can stop the bots reading robots.txt at all.

For AI visibility, the important point is that training and citation are separate decisions, as explained in citation crawlers vs training crawlers. A site that wants Claude to cite it but does not want its pages used for training can disallow ClaudeBot while allowing Claude-SearchBot and Claude-User. Blocking all three to stop training also removes the site from Claude's search answers.

Anthropic's crawling has historically been heavy relative to the traffic Claude sends back. When Cloudflare launched its crawl-to-refer ratio on Radar in July 2025, Anthropic had the highest ratio in its sample week of June 19 to 26, 2025, at about 70,900 HTML page requests per referral. Cloudflare noted that Claude's native apps send no Referer header, so the figure overstates the imbalance by an unknown amount.

In practice, teams watch Claude-User and Claude-SearchBot requests in their AI crawler logs to see which pages Claude reads, and check for 403 or 404 responses that would keep content out of answers. Promptwatch verifies Anthropic requests against the published IP list and has tracked how often Claude's citation crawler visits sites in its Claude citation crawler study.