Promptwatch Logo

Claude-SearchBot

Claude-SearchBot is Anthropic's crawler for improving search results shown to Claude users.
AnthropicClaude-SearchBot
AI Assistant

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's crawler for improving search results shown to Claude users. Anthropic says it navigates the web and analyzes pages to improve the relevance and accuracy of search responses. Cloudflare classifies the same bot in its AI search category and links to Anthropic's crawler guidance.

Anthropic separates this work from its other two named bots. Claude-User retrieves content at a person's direction during a Claude interaction, while ClaudeBot collects public web material that may contribute to model training. Claude-SearchBot is the middle case: it indexes content for search optimization rather than answering one specific fetch request or building a training dataset.

Requests can revisit online content so Anthropic's search system has material to evaluate for later results. Blocking the bot prevents that indexing on the affected host. Anthropic warns that this may reduce the site's visibility and the accuracy with which it appears in user search results.

Claude-SearchBot can therefore affect AI search visibility inside Claude. Access still does not guarantee a citation or a particular placement because the search system chooses sources for each query. It does not control general web ranking, and Anthropic does not describe this bot as a model training collector.

The recorded User-Agent and robots token are both exactly Claude-SearchBot. Cloudflare's bot directory lists no Web Bot Auth directory, but Anthropic publishes crawler IP information and this entry is marked IP verified. Promptwatch also tracks the bot, allowing classified requests to be reviewed by path, status, and time without treating every self-reported header as genuine.

Anthropic states that its bots honor standard robots.txt directives and supports the non-standard Crawl-delay field. The metadata likewise records true. Cloudflare's entry conflicts with that statement by marking followsRobotsTxt as false, so a careful implementation publishes the Anthropic-documented rule and checks actual traffic after the change.

Relevant for AI searchClaude

Is Claude-SearchBot relevant for AI search?

Yes. Claude-SearchBot feeds Claude, so the pages it can reach shape what those AI products say about you.

Claude-SearchBot crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

Track Claude-SearchBot with Promptwatch

Promptwatch classifies Claude-SearchBot (Claude-SearchBot) in real time from your server and CDN logs. See exactly when it visits, which pages it requests, the status codes it gets, and how those crawls map to AI citations. Requests are validated against the operator's published IP ranges to filter out spoofed user agents.

How to handle Claude-SearchBot

Allow Claude-SearchBot if you want Anthropic to index public pages for Claude search. Block it when that search use is unwanted. Make the choice separately from ClaudeBot training access and Claude-User retrieval because Anthropic gives each bot its own token.

To block search indexing across one host, place this in that host's top-level robots.txt file:

User-agent: Claude-SearchBot
Disallow: /

Anthropic says its bots honor this control and asks site owners to configure every subdomain they want to exclude. It also supports Crawl-delay when the goal is lower request frequency rather than a complete block. Cloudflare's false compliance flag remains a conflicting data point, so verify the resulting crawl behavior.

Anthropic advises against relying on IP blocks as the normal opt-out because blocking the crawler before it reads robots.txt can prevent it from seeing the preference. Use the named robots group first. If you need a hard security boundary, protect the content at the application or edge regardless of crawler identity.

Examples

  • A research publisher allows Claude-SearchBot on finished articles so Anthropic can consider them for relevant Claude search results.
  • A large documentation host adds a bot-specific crawl delay, then checks whether request spacing improves without removing the site from Claude search.
  • A company blocks Claude-SearchBot on each private customer subdomain while leaving its public knowledge base open.

Frequently asked questions about Claude-SearchBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Anthropic operates Claude-SearchBot. Both the bot metadata and Cloudflare's operator record identify Anthropic.

Related bots

Claude-User

Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.

AI Assistant

ClaudeBot

ClaudeBot is Anthropic's crawler for gathering public web content that could contribute to the development of its generative AI models.

AI Crawler

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant

Amazon Q

Amazon Q Business is a generative artificial intelligence (generative AI)-powered assistant that you can tailor to your business needs.

AI Assistant

ChatGPT-Operator

Handles user-initiated requests from ChatGPT operator accessing external content; not used for automated crawling or AI training.

AI Assistant

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard