Pay Per Crawl

Pay per crawl is a model in which a website charges an AI crawler a set fee each time it successfully fetches a page. A crawler that agrees to the price gets the content with an HTTP 200 response; one that does not gets HTTP 402 Payment Required along with the price. The term is most closely tied to Cloudflare, which introduced a pay per crawl feature in private beta on July 1, 2025.

Cloudflare's implementation, now part of its AI Crawl Control product, is the reference design most people mean. As of its documentation updated July 28, 2026, Cloudflare pay per crawl is still in closed beta, with access by signup or through an Enterprise account team. The mechanics:

• The site owner sets a default price per successful fetch for a domain (the minimum is $0.001) and chooses, per crawler, whether to allow it free, charge it, or block it.
• Crawlers must identify themselves with Web Bot Auth, signing each request with HTTP Message Signatures so the charge cannot be spoofed onto someone else.
• A crawler either retries after a 402 with a crawler-exact-price header, or sends a crawler-max-price header up front. A served page returns a crawler-charged header.
• With dynamic pricing enabled, the origin can return its own crawler-price header to price individual pages differently.
• Cloudflare acts as merchant of record, aggregates the charges and pays publishers monthly through a dedicated Stripe Connect account.

Two details trip people up. WAF and bot-management blocks run first, so a crawler blocked there never reaches the 'charge' step. And charging a crawler that has no billing relationship with Cloudflare works like a block that also advertises a price.

Pay per crawl sits beside two other building blocks. Really Simple Licensing (RSL) declares, in machine-readable form, what an AI system may do with content and on what terms; its 1.0 specification includes a crawl payment type and explicitly allows pay per crawl services to act as the enforcement layer. x402 is a general protocol for settling HTTP 402 payments, which some agent and API providers use for per-request billing. RSL states the terms, a 402 flow collects the fee.

For AI search visibility the trade-off is direct. A page an AI system cannot fetch cannot be retrieved, quoted or cited, and most AI crawlers are not paying customers of any pay per crawl service. Charging OAI-SearchBot, PerplexityBot or Claude's search crawler on product, pricing and documentation pages usually means disappearing from answers about your own brand. Teams that want both revenue and visibility tend to separate content by role: keep pages whose job is to win customers freely crawlable, and meter archives, datasets or premium research where the content is the product. Training crawlers are the more natural candidates for charging, since blocking them does not remove you from live answers (see citation crawlers vs. training crawlers).

Before switching anything on, measure. AI crawler logs show which bots fetch which paths and how often, which is the only realistic basis for setting a price or estimating what blocking would cost in lost citations. After enabling, watch for crawlers that simply stop visiting rather than pay, and check whether citations of the affected pages drop in the weeks that follow. Our robots.txt generator for AI crawlers helps keep the free-access rules consistent with whatever you meter.