What is AI Search?
Cloudflare AI Search is a managed search service that can use a website as one of its data sources. When a customer connects a domain in the customer's Cloudflare account, the website crawler identifies itself as Cloudflare-AI-Search. The crawl belongs to that customer's AI Search instance rather than a general Cloudflare web index.
According to the Cloudflare AI Search documentation, the service fetches each selected page, converts it to Markdown, splits the content into chunks, and adds those chunks to an index. An application can retrieve matching chunks or use them as context for generated answers. Public endpoints are optional and remain under the customer's control.
Page discovery depends on the configured parse type. Sitemap mode reads XML sitemaps declared in robots.txt or supplied directly by the customer. Discover mode can begin at a source URL and follow links, with settings for depth, page count, cache age, subdomains, and external links. Cloudflare periodically syncs website sources to pick up changed or deleted pages.
The internal and external identities describe where a request lands. On domains in the customer's Cloudflare account the token is Cloudflare-AI-Search. If discovery reaches a domain outside that account, Cloudflare uses Cloudflare-AI-Search-External instead. Site owners should not combine the two in logs or assume that one robots group controls both.
Being indexed can make a page retrievable in the customer's search or chat workflow. It does not guarantee that the page is returned for a query, quoted in an answer, or exposed to the public. Cloudflare documents this crawler for customer search indexing and does not state that it gathers foundation-model training data.
Cloudflare AI Search respects robots.txt. The service reports specific errors when a crawl or path is blocked, and it can also reject content based on Cloudflare Content Signals. The recorded full user agent is Cloudflare-AI-Search (https://developers.cloudflare.com/ai-search; [email protected]), which gives site operators a stable token and contact address.
Requests are signed through Web Bot Auth, with public keys at https://search.ai.cloudflare.com/.well-known/http-message-signatures-directory. Signature verification provides stronger origin evidence than the user-agent string and can support a narrow firewall exception. It proves Cloudflare sent the request, not that the requested page will appear in any particular answer.
