Promptwatch Logo

OpenAI Crawlers

OpenAI crawlers are the user agents OpenAI uses on the web: OAI-SearchBot for ChatGPT search, GPTBot for training, ChatGPT-User and OAI-AdsBot.
Updated November 15, 2026
AI

Definition

OpenAI crawlers are the automated user agents OpenAI uses to fetch web pages for its products, each with its own robots.txt token and purpose. As of October 2026, OpenAI's crawler documentation lists four: OAI-SearchBot for ChatGPT search, GPTBot for model training, ChatGPT-User for user-triggered visits, and OAI-AdsBot for checking ad landing pages.

The split matters because each agent controls a different outcome. Blocking one does not block the others, and OpenAI states that the settings are independent.

  • OAI-SearchBot crawls pages so they can appear in ChatGPT search answers. OpenAI says sites that opt out will not be shown in search answers, although they can still appear as navigational links. Robots.txt changes take about 24 hours to reach its systems.
  • GPTBot collects content that may be used to train OpenAI's foundation models. Disallowing it signals that your content should not be used for training. If both GPTBot and OAI-SearchBot are allowed, OpenAI may reuse one crawl for both purposes.
  • ChatGPT-User is a user-triggered fetcher: it fetches a page when a person's request in ChatGPT, a custom GPT or a GPT Action needs it. OpenAI says robots.txt rules may not apply to these visits, and the agent is not used to decide search inclusion.
  • OAI-AdsBot visits only landing pages submitted as ChatGPT ads, to check policy compliance and relevance. Its data is not used for training.

A shared training control is on the way as well. On October 30, 2026 OpenAI said it will honor a common robots.txt signal, a group for the policy token __usage_ai-training that several AI providers will read, and aims to implement it 'in the coming months' (OpenAI). OpenAI says its existing crawler controls will continue to apply, and its announcement does not say how a conflict between that group and a GPTBot rule will be resolved, so keep GPTBot rules explicit. The signal covers training only and does not affect OAI-SearchBot.

OpenAI publishes IP ranges for each agent (for example searchbot.json and gptbot.json), and current user-agent strings carry versions such as OAI-SearchBot/1.4 and GPTBot/1.4. Match on the token and the published ranges rather than the full string, because the version changes. ChatGPT's browsing agent, which OpenAI's help center now calls ChatGPT Work's cloud browser, takes a different approach: it signs requests with HTTP Message Signatures and a Signature-Agent: "https://chatgpt.com" header, which is the Web Bot Auth pattern, so CDNs can verify it cryptographically instead of by IP.

For GEO, OAI-SearchBot is the one to protect. OpenAI said in October 2026 that more than 1.2 billion people use ChatGPT each week, and its help article on ChatGPT search tells site owners to allow OAI-SearchBot and OpenAI's searchbot IP addresses at the host and CDN to be eligible for search results. A common mistake is a firewall or bot-management rule, added to stop scrapers, that silently challenges OAI-SearchBot. Robots.txt then looks correct while ChatGPT cannot fetch the page.

Discovery for OAI-SearchBot is changing too. Cloudflare and OpenAI said on October 15, 2026 that a research pilot, running since July on about 1% of Cloudflare's network, uses Cloudflare's network signals to help OpenAI's search crawler find pages, and has brought hundreds of millions of previously unknown pages into OpenAI's search index (Cloudflare). OpenAI still evaluates and ranks those pages on its own.

A practical policy starts with intent. Decide whether your content may be used for training (GPTBot), whether it should be citable in ChatGPT answers (OAI-SearchBot), and whether user-triggered fetches should succeed (ChatGPT-User). Most brands that want AI search visibility allow OAI-SearchBot and ChatGPT-User and make a separate business decision about GPTBot. Then confirm the result in AI crawler logs, for example with Promptwatch's agent analytics: verified OAI-SearchBot hits returning 200 on your key pages are the evidence that matters, and a run of 403s or challenge pages is the warning sign. The broader distinction is covered under citation crawlers vs. training crawlers, and our profiles of GPTBot and OAI-SearchBot list the current tokens and verification details.

Examples of OpenAI Crawlers

  • A news publisher disallows GPTBot in robots.txt but keeps OAI-SearchBot allowed, so its articles remain citable in ChatGPT search while signaling that they should not be used for model training.
  • After enabling a CDN's 'block AI scrapers' preset, a SaaS company sees OAI-SearchBot requests to its pricing page switch from 200 to 403 in its crawler logs and adds an allow rule matched on the token plus OpenAI's searchbot.json IP ranges.
  • A support team notices ChatGPT-User requests spiking on its troubleshooting docs after a product outage, a sign that users are pasting the URLs into ChatGPT or that answers are fetching them live.
  • An advertiser running ChatGPT ads finds OAI-AdsBot visits only on the three landing pages it submitted, confirming the bot is reviewing ads and not crawling the wider site.
  • A bank's security team allowlists ChatGPT's cloud browser by verifying the Signature-Agent header against the public keys at chatgpt.com/.well-known/http-message-signatures-directory instead of maintaining IP lists.

Terms related to OpenAI Crawlers

AI Web Crawlers

AI crawlers are bots run by AI companies that fetch web pages for model training, AI search indexes or live answers, like GPTBot, ClaudeBot and PerplexityBot.

AI

Citation Crawlers vs Training Crawlers

Training crawlers collect pages to train AI models; citation crawlers index or fetch pages so AI answers can cite them. Each has its own robots.txt token.

SEO

Robots.txt

Robots.txt is a root-level file that tells crawlers which URLs they may fetch. It now decides whether AI search and training bots can reach your content.

SEO

ChatGPT Search

ChatGPT Search is the web search built into ChatGPT: it rewrites prompts into search queries, reads results and answers with inline citations to sources.

GEO

AI Crawler Logs

AI crawler logs are server or CDN records of requests from AI bots like GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot: what they fetch and what they get.

Analytics

AI Crawler Verification

AI crawler verification confirms a request claiming to be GPTBot, ClaudeBot or another AI bot really comes from that provider, via IP lists, DNS or signatures.

SEO

Web Bot Auth

Web Bot Auth is an IETF draft protocol in which bots sign each HTTP request with a private key, so websites can verify which operator actually sent it.

AI

OpenAI

OpenAI is the AI company behind ChatGPT, Codex and the GPT-6 models. It said in September 2026 that ChatGPT had 1.2 billion weekly users.

AI

ChatGPT Atlas

ChatGPT Atlas was OpenAI's macOS web browser with ChatGPT built in. Launched in October 2025, it was retired on August 9, 2026 in favor of the ChatGPT app.

AI

User-Triggered Fetchers

User-triggered fetchers are bots like ChatGPT-User and Perplexity-User that fetch a page when a person asks, and may ignore robots.txt rules in AI search.

AI

Anthropic Crawlers (ClaudeBot, Claude-SearchBot, Claude-User)

Anthropic crawlers are the three bots behind Claude: ClaudeBot for training, Claude-SearchBot for search and Claude-User for fetches a user asks for.

SEO

AI Search Index

An AI search index is the store of crawled web pages an AI assistant retrieves from when it answers. Google, OpenAI, Perplexity and Meta each run their own.

AI

Frequently Asked Questions about OpenAI Crawlers

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard