Pay Per Crawl

Pay per crawl is an access model in which a publisher charges AI crawlers and agents a fee for each request, rather than choosing between free access and an outright block in robots.txt. The charge is signaled with HTTP status code 402 Payment Required, a code reserved in the original HTTP specification but effectively unused until AI crawling created a reason for it.

Cloudflare launched the best-known implementation in July 2025, moving the paywall to the CDN edge. Site owners set a per-request price, connect a payment account, and let the network act as merchant of record. A crawler advertises what it is willing to spend using a crawler-max-price header; if the price is within budget the request is served, and if not the edge returns 402. Cloudflare paired this with a default permission model in which unrecognized AI bots are blocked unless the site opts them in.

Pay per crawl answers the settlement question, not the permissions question. It moves money but does not describe what a buyer may do with the content afterward, which is why it is usually paired with a licensing layer such as Really Simple Licensing and an identity layer such as Web Bot Auth. Payment protocols like x402 provide the machine-to-machine rails.

For GEO teams the trade-off is direct: charging for crawls can create a revenue line from AI crawlers, but every blocked or unpaid fetch is content that cannot be retrieved, cited, or recommended in an AI answer. Most brands that depend on AI visibility keep marketing content freely crawlable and reserve metering for archives, proprietary data, or high-cost endpoints. Crawler log analysis is what tells you which bots would actually be affected, and the free robots.txt generator for AI is the fastest way to draft rules that keep citation-worthy pages open while metering the rest.