Does Cloudflare Crawler respect robots.txt?

Yes. Cloudflare says /crawl follows path directives and crawl-delay. A refused URL is listed as disallowed in the crawl result.

Can Cloudflare Crawler bypass bot protection?

No. Cloudflare says Browser Rendering requests remain identified as bots and do not bypass CAPTCHAs or bot protection by default.

Does a Cloudflare Crawler request mean my page is training an AI model?

Not necessarily. The Cloudflare customer declares the crawl purpose, and the endpoint can be used for search, AI input, AI training, or a permitted combination. Cloudflare Crawler itself is retrieval infrastructure.

How is Cloudflare Crawler different from Browser Run?

CloudflareBrowserRenderingCrawler is reserved for the /crawl endpoint. Other Browser Rendering actions use the separate Browser Run identity and can support rendering or browser automation workflows.

How do I verify a genuine Cloudflare Crawler request?

Validate its Web Bot Auth HTTP message signature with the current keys in Cloudflare's published signature directory. The user-agent string alone is not proof.

Cloudflare Crawler AI Crawler Guide

What is Cloudflare Crawler?

Cloudflare Crawler is the bot identity used by the /crawl endpoint in Cloudflare Browser Rendering. A Cloudflare customer sends that endpoint a target URL, and Cloudflare retrieves web content for the customer's crawl job. The caller chooses the site; Cloudflare Crawler is not an independent public search index.

The /crawl endpoint always uses CloudflareBrowserRenderingCrawler/1.0, and Cloudflare does not let the customer replace that user agent. Requests identify as bots. Cloudflare also says Browser Rendering cannot bypass a site's CAPTCHA or bot protection, although a site owner using Cloudflare can configure its own security rules for traffic it intends to allow.

Cloudflare documents robots.txt support for this endpoint, including crawl-delay. If a site does not specify a delay, the crawler waits 0.5 seconds between requests to the same domain. URLs refused by robots.txt are returned to the Browser Rendering customer with a disallowed status rather than fetched.

The endpoint also supports Cloudflare Content Signals. A crawl declares one or more purposes from search, ai-input, and ai-train; the default request declares all of them. If the target site's robots.txt sets one of the declared purposes to no, Cloudflare says the crawl request is rejected. This gives a publisher a more specific control than treating every automated use alike.

A Cloudflare Crawler visit does not mean Cloudflare is adding the page to its own model-training set or a general AI search engine. The downstream purpose belongs to the Cloudflare customer that started the job. The declared Content Signals are the clearest request-level indication of whether that customer says the content is for search, AI input, or training.

This identity is specific to /crawl. Other Browser Rendering REST actions and Workers bindings can render pages for screenshots, PDF creation, extraction, or browser automation, but Cloudflare's bot directory records those under the separate Browser Run identity rather than CloudflareBrowserRenderingCrawler.

Cloudflare Crawler publishes a Web Bot Auth signature directory at https://web-bot-auth-crawl.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory. A verifier can validate the HTTP message signature against those keys instead of trusting a user-agent string that anyone could copy.

Relevant for AI search

Is Cloudflare Crawler relevant for AI search?

Yes. Cloudflare Crawler collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Cloudflare Crawler fetches, indexes, or retrieves pages for an AI product, so the pages it can reach shape how that system describes your brand and products.

How to handle Cloudflare Crawler

Allow the crawler when your site permits Cloudflare customers to retrieve those pages through Browser Rendering. Use path rules for sections that may be crawled and reserve application authentication for material that should never be public. Content Signals can express whether allowed content may be used for a declared AI or search purpose.

To block the /crawl identity everywhere:

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

Cloudflare says the endpoint honors this rule and reports blocked URLs as disallowed. A Crawl-delay value can reduce the request rate without refusing the job entirely.

At the edge, validate Web Bot Auth before granting any special treatment. Retrieve current keys from https://web-bot-auth-crawl.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory and verify the request's HTTP message signature. Do not create an allow rule based only on CloudflareBrowserRenderingCrawler/1.0.

Examples

A Cloudflare customer starts a Browser Rendering crawl of a documentation site. The job retrieves public guides and reports the `/admin/` URLs refused by robots.txt as disallowed.
A publisher permits crawling for search but denies a declared `ai-train` purpose with Content Signals, while its edge validates Cloudflare's message signature before applying crawler-specific rules.

Frequently asked questions about Cloudflare Crawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

A Cloudflare customer submits a target URL to the Browser Rendering /crawl endpoint. Cloudflare then performs that crawl using its fixed crawler identity.