Crawl-to-Refer Ratio

The crawl-to-refer ratio is the number of HTML pages a platform's crawlers request for every visit that platform refers back to websites. A ratio of 5:1 means five pages crawled per referred visitor. A ratio in the thousands means the platform reads far more of the web than it sends people to.

Cloudflare introduced the metric on Cloudflare Radar on July 1, 2025 (Cloudflare blog). It divides HTML requests from all user agents associated with a platform by HTML requests whose Referer header contains one of that platform's hostnames, normalized to one referral. Training crawlers, search crawlers and user-triggered fetchers are pooled under the platform name, so the figure describes a company's overall exchange with the web rather than one bot. Cloudflare notes a known bias: referrals from native apps, such as the Claude app, usually carry no Referer header, so ratios may overstate the imbalance by an unknown amount.

The numbers have moved a lot. For June 19 to 26, 2025, Cloudflare reported ratios from Anthropic's 70,900:1 down to Mistral's 0.1:1. In an August 2025 follow-up, Cloudflare showed Anthropic falling from about 286,900:1 in January 2025 to about 38,000:1 in July, after Claude gained web search with clickable citations, while OpenAI moved from about 1,200:1 to 1,100:1 and Google sat at 5.4:1 in July (Cloudflare data). In July 2026, Business Insider reported Cloudflare data for July 1 to 7 putting Anthropic at about 2,800:1, down from about 8,800:1 in early April. Third-party pulls of the Radar API for the full month of July 2026 put Anthropic near 1,900:1, Perplexity near 290:1, OpenAI near 250:1 and Google near 4.7:1. Anthropic has disputed Cloudflare's methodology, saying new search features are increasing referrals.

For publishers, the ratio is a rough price tag on letting a platform's AI crawlers in. Classic search engines crawl a few pages per visitor sent. AI platforms that answer in place send far fewer visitors, though the gap has narrowed as assistants added linked citations. That context feeds decisions on robots.txt rules and on pay per crawl, which Cloudflare announced the same day it published the metric.

Two cautions apply before blocking anything. First, the ratio is network-wide; your own site can look very different. Second, it mixes purposes. Blocking a training crawler and blocking a search crawler have different consequences, since only the search and user-fetch bots feed the answers that cite you (see citation crawlers vs. training crawlers).

You can calculate a site-level version. Count requests per platform in your AI crawler logs, count sessions referred by that platform in analytics (for example the GA4 AI Assistant channel plus a source filter), and divide. Track it monthly per platform and per section of the site. Because app referrals often arrive without a referrer, treat the result as an upper bound and compare it with AI-referred traffic and self-reported data. Our AI crawler traffic report shows the weekly share of IP-verified crawler requests by bot, a useful reference for the crawl side of the ratio.