OpenAI Crawlers

OpenAI crawlers are the automated user agents OpenAI uses to fetch web pages for its products, each with its own robots.txt token and purpose. As of October 2026, OpenAI's crawler documentation lists four: OAI-SearchBot for ChatGPT search, GPTBot for model training, ChatGPT-User for user-triggered visits, and OAI-AdsBot for checking ad landing pages.

The split matters because each agent controls a different outcome. Blocking one does not block the others, and OpenAI states that the settings are independent.

• OAI-SearchBot crawls pages so they can appear in ChatGPT search answers. OpenAI says sites that opt out will not be shown in search answers, although they can still appear as navigational links. Robots.txt changes take about 24 hours to reach its systems.
• GPTBot collects content that may be used to train OpenAI's foundation models. Disallowing it signals that your content should not be used for training. If both GPTBot and OAI-SearchBot are allowed, OpenAI may reuse one crawl for both purposes.
• ChatGPT-User fetches a page when a person's request in ChatGPT, a custom GPT or a GPT Action needs it. OpenAI says robots.txt rules may not apply to these visits, and the agent is not used to decide search inclusion.
• OAI-AdsBot visits only landing pages submitted as ChatGPT ads, to check policy compliance and relevance. Its data is not used for training.

OpenAI publishes IP ranges for each agent (for example searchbot.json and gptbot.json), and current user-agent strings carry versions such as OAI-SearchBot/1.4 and GPTBot/1.4. Match on the token and the published ranges rather than the full string, because the version changes. ChatGPT's browsing agent takes a different approach: its cloud browser signs requests with HTTP Message Signatures and a Signature-Agent: "https://chatgpt.com" header, which is the Web Bot Auth pattern, so CDNs can verify it cryptographically instead of by IP.

For GEO, OAI-SearchBot is the one to protect. OpenAI said in October 2026 that more than 1.2 billion people use ChatGPT each week, and its help article on ChatGPT search tells site owners to allow OAI-SearchBot and OpenAI's searchbot IP addresses at the host and CDN to be eligible for search results. A common mistake is a firewall or bot-management rule, added to stop scrapers, that silently challenges OAI-SearchBot. Robots.txt then looks correct while ChatGPT cannot fetch the page.

A practical policy starts with intent. Decide whether your content may be used for training (GPTBot), whether it should be citable in ChatGPT answers (OAI-SearchBot), and whether user-triggered fetches should succeed (ChatGPT-User). Most brands that want AI search visibility allow OAI-SearchBot and ChatGPT-User and make a separate business decision about GPTBot. Then confirm the result in AI crawler logs: verified OAI-SearchBot hits returning 200 on your key pages are the evidence that matters, and a run of 403s or challenge pages is the warning sign. The broader distinction is covered under citation crawlers vs. training crawlers, and our profiles of GPTBot and OAI-SearchBot list the current tokens and verification details.