Who operates AI Search External?

AI Search External is operated by Cloudflare. It functions as Cloudflare's AI search crawler.

Does AI Search External respect robots.txt?

Cloudflare has not published a robots.txt commitment for AI Search External. Treat its behavior as unconfirmed and watch your access logs to see whether your rules are honored.

Should I block AI Search External?

Usually no. Blocking AI Search External can remove your pages from the AI answers and search results it powers, which reduces visibility. Only block it to deliberately opt out.

AI Search External - What It Is & How to Handle It

Q: How can I verify AI Search External is genuine?

AI Search External is a Cloudflare signed agent. It signs requests using Web Bot Auth HTTP message signatures and publishes its public keys at https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory, so you can validate a request at your edge instead of trusting the user-agent string, which anyone can spoof.

What is AI Search External?

AI Search External is Cloudflare's AI search crawler. Cloudflare AI Search is a managed service that lets you connect your data and easily build AI-powered search.

AI Search External decides which of your pages can appear in the AI search experience it powers. It crawls and indexes content so that experience can retrieve, quote, and link to it, so anything AI Search External cannot reach is effectively invisible there. That makes it worth allowing unless you have a specific reason to opt out of the surface.

Tracking which AI crawlers and agents reach your site, and what they do once there, is the foundation of generative engine optimization. See our guides to AI crawlers and robots.txt to control automated access and protect your AI search visibility.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle AI Search External

Most sites keep AI Search External allowed, because blocking it can remove your pages from the AI answers and search results it powers. Only disallow it if you specifically want to opt out of that surface.

To control AI Search External, add a rule for its user agent to your robots.txt:

User-agent: Cloudflare-AI-Search-External
Disallow: /

Cloudflare does not publish a robots.txt commitment for AI Search External, so confirm how it behaves by watching your access logs.

AI Search External is a Cloudflare signed agent: it signs requests with Web Bot Auth and publishes its keys at https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory, so you can tell genuine traffic from anything spoofing its user agent.

Examples

After publishing a new comparison page, a marketer checks whether AI Search External has fetched it so the content can appear in AI answers.
An SEO lead keeps AI Search External allowed in robots.txt to preserve visibility in AI-powered search.

Related bots

AI Search

Cloudflare AI Search is a managed service that lets you connect your data and easily build AI-powered search.

AI Search

Cloudflare Browser Run

Renders web pages in headless browsers for Cloudflare customers. Used for browser automation (screenshots, PDF generation, content extraction, etc.) and for AI agents to interact with the web.

AI Assistant

Cloudflare Crawler

The Cloudflare Crawler is a well-behaved crawler that retrieves web content. By default, it self-identifies as a bot, honors robots.txt directives, and cannot bypass CAPTCHAs or bot protection.

AI Crawler