Robots.txt

Robots.txt is a plain-text file at the root of a host (for example example.com/robots.txt) that tells automated crawlers which URLs they may and may not fetch. It implements the Robots Exclusion Protocol, first proposed in 1994 and standardized by the IETF as RFC 9309 in 2022.

A file is made of groups. Each group starts with one or more User-agent lines naming a crawler's product token, followed by Allow and Disallow rules for URL paths. A crawler obeys the group that matches its token and falls back to the * group only if none does. When an Allow and a Disallow rule both match, the most specific (longest) rule wins. Sitemap lines point to XML sitemaps. Crawl-delay is not part of RFC 9309; Google ignores it, while Anthropic says its bots honor it. RFC 9309 also says a crawler should treat an unreachable robots.txt (a server error) as a full disallow, so a file that times out under load can block every compliant bot.

The biggest change in recent years is that AI companies now split their crawlers by purpose, and robots.txt is how you choose between them:

• Search and retrieval crawlers fetch pages so they can be cited in answers: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot for the classic engines.
• Training crawlers collect data for future models: GPTBot, ClaudeBot, Meta-ExternalAgent and others.
• User-triggered fetchers load a page because someone asked about it: ChatGPT-User, Claude-User, Perplexity-User. OpenAI says robots.txt rules may not apply to ChatGPT-User, while Anthropic says Claude-User respects them.
• Control tokens that are not separate crawlers: Google-Extended governs whether content Google crawls may be used for Gemini training and grounding, and Google says it has no effect on Search inclusion or ranking.

The settings are independent. OpenAI's crawler docs state that a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot, and that sites blocking OAI-SearchBot will not be shown in ChatGPT search answers apart from navigational links. Changes take about 24 hours to apply on OpenAI's side. Blocking a training bot therefore costs nothing in AI search visibility, while blocking a search bot removes you from that engine's answers. See OpenAI crawlers and citation crawlers vs. training crawlers.

Google is a special case. AI Overviews and AI Mode are built on Googlebot's index, so robots.txt alone cannot keep a site out of them without also leaving Google Search. Since August 31, 2026, the Search generative AI control in Search Console lets any site opt out of AI Overviews, AI Mode and generative features in Discover without affecting regular rankings.

Robots.txt has firm limits. It controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it, so use noindex for that. Compliance is voluntary, so verify real traffic with AI crawler logs and AI crawler verification rather than assuming. It is also a single on/off switch per path. Newer layers sit on top of it: Cloudflare's content signals add search, ai-input and ai-train preferences inside robots.txt, and the IETF AI Preferences working group is drafting a standard vocabulary. Our robots.txt generator for AI drafts purpose-specific rules, and this guide shows how to confirm which bots actually visit.