Visible Amsterdam: Join us Sep 3
Promptwatch Logo

Cloudflare Crawler

The Cloudflare Crawler is a well-behaved crawler that retrieves web content. By default, it self-identifies as a bot, honors robots.txt directives, and cannot bypass CAPTCHAs or bot protection.
CloudflareCloudflareBrowserRenderingCrawler
AI CrawlerAgent

What is Cloudflare Crawler?

Cloudflare Crawler is Cloudflare's AI agent. The Cloudflare Crawler is a well-behaved crawler that retrieves web content. By default, it self-identifies as a bot, honors robots.txt directives, and cannot bypass CAPTCHAs or bot protection.

Cloudflare Crawler acts on behalf of a person rather than indexing the web in bulk. It arrives when someone asks it to complete a task on your site, so the pages and flows it can reach decide whether that task succeeds. Blocking it will not remove you from AI answers, but it can stop real people from finishing real actions.

Tracking which AI crawlers and agents reach your site, and what they do once there, is the foundation of generative engine optimization. See our guides to AI crawlers and robots.txt to control automated access and protect your AI search visibility.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle Cloudflare Crawler

Most sites keep Cloudflare Crawler allowed, because it shows up when a real person asks it to do something on their behalf. Block it only if you do not want automated agents acting on your site.

To control Cloudflare Crawler, add a rule for its user agent to your robots.txt:

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

Cloudflare Crawler generally honors robots.txt directives.

Cloudflare Crawler is a Cloudflare signed agent: it signs requests with Web Bot Auth and publishes its keys at https://web-bot-auth-crawl.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory, so you can tell genuine traffic from anything spoofing its user agent.

Examples

  • A customer sends Cloudflare Crawler to complete a task on a site, and the operations team confirms the agent reached the pages it needed.
  • An engineer verifies Cloudflare Crawler against its published signature keys before allowing it through the edge.

Related bots

AI Search External

Cloudflare AI Search is a managed service that lets you connect your data and easily build AI-powered search.

AI Search

AI Search

Cloudflare AI Search is a managed service that lets you connect your data and easily build AI-powered search.

AI Search

Cloudflare Browser Run

Renders web pages in headless browsers for Cloudflare customers. Used for browser automation (screenshots, PDF generation, content extraction, etc.) and for AI agents to interact with the web.

AI Assistant

AI2Bot

AI2Bot is operated by the Allen Institute for Artificial Intelligence (Ai2) to crawl the web for content to train open-source AI models.

AI Crawler

aiHitBot

aiHitBot collects and maintains historical information about companies.

AI Crawler

Frequently asked questions about Cloudflare Crawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Cloudflare Crawler is operated by Cloudflare. It functions as Cloudflare's AI agent.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard