Definition
AI Crawl Control is Cloudflare's product for managing how AI crawlers access a site. It lists the AI crawlers that request your pages, shows how often each one is allowed or fails, and lets you set an action per crawler: allow, block, or, for sites in the private beta, charge. Because the rule runs at the CDN edge, it applies before a request reaches your server.
This is different from robots.txt, which only asks crawlers to behave. Cloudflare's documentation says the Crawlers tab shows a count of robots.txt violations per crawler, and that you can still choose to enforce robots.txt on crawlers you allow. On the free plan, crawlers are identified by their user agent strings, which catches well-known, self-identifying bots. Paid plans use Cloudflare's bot detection ID for more thorough detection. Spoofed user agents are the reason AI crawler verification matters alongside any allow or block list.
Blocking goes through a WAF custom rule. When you block crawlers in AI Crawl Control, Cloudflare creates or updates one WAF rule named AI Crawl Control, and the rule is added after your existing custom rules. Skip, redirect or transform rules that run earlier can let a blocked bot through, and an upstream rule can block a crawler you set to Allow. Cloudflare's guidance is to move the AI Crawl Control rule to the top if blocks are not holding, and to review upstream rules if allowed bots are still blocked. Pay per crawl runs after WAF, so a WAF block wins over a payment header.
On paid plans you can also choose the block response: 403 Forbidden to signal that access is not wanted, or 402 Payment Required to signal that the crawler must pay. The 402 option ties into pay per crawl, which Cloudflare lists as a closed beta. Usage preferences can be declared separately with Content Signals.
For GEO, the risk is accidental blocking. Cloudflare suggests allowing crawlers that return value through citations or referrals. Blocking a retrieval bot can remove your pages from live AI answers, while blocking a training bot affects something else, so the two should be set separately. Our CDN study on how AI agents crawl and cite content looks at how CDN setup relates to AI visibility, and you can draft the file side with the robots.txt generator for AI.
Examples of AI Crawl Control
- A publisher sets training crawlers to Block and the crawlers behind live AI answers to Allow in AI Crawl Control, so its articles can still be cited.
- A SaaS company sets a crawler to Allow but sees it still get blocked, then finds an older WAF rule that blocks all bots and runs before the AI Crawl Control rule.
- A media site switches its block response from 403 to 402 and adds a message telling crawler operators how to license its content.
- An ecommerce team checks the Crawlers tab, sees an AI crawler with a high robots.txt violation count, and moves from asking it to stop to blocking it at the edge.
