OAI-SearchBot

OAI-SearchBot is OpenAI's crawler for finding pages that can be linked and surfaced in ChatGPT search results. It runs automatically for search discovery and indexing. That makes it different from ChatGPT-User, which fetches a page in response to a particular user's action.

Its requests can cover pages that no one has just asked ChatGPT to open. A successful crawl gives OpenAI's search system access to the returned page, while redirects, authentication barriers, or denied responses limit what the crawler can retrieve. The bot's presence in a log is evidence of a search crawl, not proof that the page was cited.

For publishers, OAI-SearchBot is the relevant OpenAI control for search inclusion. Allowing it keeps a page eligible to surface in ChatGPT Search. Disallowing it tells OpenAI not to crawl that content for Search, though neither choice guarantees how any particular answer will be assembled.

This bot is not OpenAI's training crawler. OpenAI assigns that purpose to GPTBot and lets site owners set the two agents independently. A site can therefore permit OAI-SearchBot for search while disallowing GPTBot for training.

OAI-SearchBot follows robots.txt. OpenAI notes that a change can take about 24 hours to reach its systems, so a newly added rule may not alter requests immediately. Path-specific rules are useful when public documentation should remain searchable but private or low-value areas should not be crawled.

Cloudflare's bot directory lists the user agent as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot and points operators to OpenAI's crawler documentation. OpenAI publishes current source ranges at https://openai.com/searchbot.json. Validate the token and source address together when identity matters.