What is ClaudeBot?
ClaudeBot is Anthropic's crawler for gathering public web content that could contribute to the development of its generative AI models. Anthropic says this material may be used to improve model utility and safety. A request from ClaudeBot is a model-development crawl, not a person asking Claude to open that page.
Anthropic separates this work from its other web access. Claude-SearchBot supports the search index used to improve search results, while Claude-User retrieves pages at a user's direction. Blocking ClaudeBot therefore addresses potential training use without automatically blocking either of those workflows.
Allowing a page to be crawled does not mean Anthropic will include it in a training dataset, and it does not promise a citation or appearance in a Claude answer. The operator describes the content as something that could contribute to training. Conversely, Anthropic says a ClaudeBot restriction signals that future material from the site should be excluded from its AI model training datasets.
According to Anthropic's crawler guidance, its bots honor standard robots.txt directives and the non-standard Crawl-delay directive. Anthropic also says it aims to avoid disruptive request rates and will not try to bypass CAPTCHAs.
The recorded user agent is ClaudeBot/1.0 (+https://anthropic.com/claudebot), with ClaudeBot as the stable token. A user-agent header can be copied by another client, and this bot does not include a Web Bot Auth signature directory. Anthropic's guidance links to its current crawler IP list for network-level checks.
Site owners can make a separate choice for training crawls, search indexing, and user-requested retrieval because Anthropic gives each one its own token. The AI crawler and robots.txt guides explain how those controls fit together.
