Does Claude Web respect robots.txt?

The bot record says yes. Anthropic's current robots documentation covers its replacement bots rather than the legacy `claude-web` token.

Is Claude Web still a current Anthropic bot?

It is a legacy bot. Its behavior has largely moved to ClaudeBot and Claude-User, and Anthropic's current guidance also identifies Claude-SearchBot for search indexing.

Did Claude Web collect model-training data?

The supplied record only documents recent-content fetching for the assistant. Anthropic assigns possible training collection to ClaudeBot in its current bot policy.

Do I still need a Claude Web robots rule?

Keep it if the token appears in your logs or if you want the legacy policy to remain explicit. Add separate rules for current Anthropic bots because the old token does not govern them.

Claude Web: AI Crawler Guide & robots.txt

What is Claude Web?

Claude Web is a legacy Anthropic crawler identified by the token claude-web. It fetched recent web content for the Claude assistant. The role has largely moved to newer Anthropic bots, so this token is most useful when interpreting older logs or maintaining a rule that predates the current bot names.

In its original workflow, Claude Web brought current page content into an assistant interaction. That could affect whether Claude had fresh information from a site at response time. It was not the same thing as a standing, general web index, and a historical fetch did not guarantee that Claude cited the page.

Anthropic's current crawler guidance no longer lists Claude Web. It separates the work among ClaudeBot for material that may contribute to model training, Claude-User for user-directed retrieval, and Claude-SearchBot for search result quality. Site owners should configure those current tokens independently.

Nothing in the supplied Claude Web record says that its recent-content fetches trained Claude models. Do not relabel old claude-web traffic as training collection. For present-day training preferences, Anthropic directs webmasters to ClaudeBot rather than this legacy token.

The record marks Claude Web as respecting robots.txt. Keeping a claude-web rule can still express a policy for late or replayed traffic, but it does not control ClaudeBot, Claude-User, or Claude-SearchBot. Each subdomain also needs its own robots.txt file under Anthropic's current guidance.

Claude Web is both tracked by Promptwatch and marked as IP verified in the bot metadata. Those attributes help classify observed traffic more confidently than a token match alone. Since the agent is legacy, compare any new request with current Anthropic IP information and behavior instead of assuming an old address remains valid.

Relevant for AI searchClaude

Is Claude Web relevant for AI search?

Yes. Claude Web feeds Claude, so the pages it can reach shape what those AI products say about you.

Claude Web crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

Track Claude Web with Promptwatch

Promptwatch classifies Claude Web (claude-web) in real time from your server and CDN logs. See exactly when it visits, which pages it requests, the status codes it gets, and how those crawls map to AI citations. Requests are validated against the operator's published IP ranges to filter out spoofed user agents.

How to handle Claude Web

Retain a Claude Web rule if you still see the legacy token or want historical policy coverage. Do not use it as the only Anthropic rule. Decide separately whether to allow ClaudeBot, Claude-User, and Claude-SearchBot because they now have different purposes.

To block the legacy crawler, use:

User-agent: claude-web
Disallow: /

The record says Claude Web respects robots.txt. If it appears unexpectedly, verify the source against current Anthropic IP information and inspect the requested URL before treating it as genuine legacy traffic.

Examples

An analyst labels `claude-web` requests from an older log archive as legacy assistant fetches instead of grouping them with ClaudeBot training crawls.
A publisher keeps its old Claude Web block but adds separate decisions for Claude-User and Claude-SearchBot.
A security engineer investigates a new `claude-web` request because current Anthropic documentation no longer names that crawler.

Frequently asked questions about Claude Web

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Anthropic operated Claude Web as a crawler for bringing recent web content into the Claude assistant.

Related bots

ClaudeBot

ClaudeBot is Anthropic's crawler for gathering public web content that could contribute to the development of its generative AI models.

AI Crawler

Claude-User

Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.

AI Assistant

Claude-SearchBot

Claude-SearchBot is Anthropic's crawler for improving search results shown to Claude users.