Skip to main content

​
Capture every AI Crawler Log with Cloudflare

AI assistants now answer your buyers instead of sending them to you, and the crawlers behind them (GPTBot, ClaudeBot, PerplexityBot, Googlebot) are surging: AI crawling grew ~15× in 2025, now ~4.2% of all HTML requests (Cloudflare Radar). They rarely click through, so your referral analytics can’t show what AI reads on your site or whether it cites you.

​
Why your server logs aren’t enough

Those crawler visits do hit your site, but your server logs alone won’t surface them: you’d have to capture and stream every request somewhere it can be analyzed, then keep that pipeline running. Cloudflare will sit in front of your site, see every request, and ship the logs to Promptwatch with a small Worker (or Logpush), so there’s nothing to build and maintain.

​
Why Cloudflare

Cloudflare is a content delivery network (CDN) that sits in front of your site; Promptwatch’s small Worker runs there, before the cache, so it records every request. It’s also the easiest place to do this:
  1. Free to start. Cloudflare’s free plan is permanent (not a trial) and includes DNS, CDN, SSL, and DDoS protection.
  2. Easy. No app code changes — just a one-time DNS change, the proxy on, and a paste-once Worker. Reversible anytime.
  3. Global. 330+ cities in 125+ countries, so it adds no latency to your pages.
  4. Mainstream. In front of more than 1 in 5 websites (22.8%, W3Techs 2026); familiar, low-risk infrastructure.

​
What you’ll see in Promptwatch

  1. Which crawlers, how often. Per-bot frequency for GPTBot, ClaudeBot, PerplexityBot, and Googlebot.
  2. Which pages they read. Your most-crawled pages, not just totals.
  3. Whether citations turn into visitors. Your logs show the citation crawlers that fetch a page when ChatGPT, Perplexity, or Claude reference it; what they can’t show is whether that citation sent a real visitor. Promptwatch’s Visitor Analytics connects the two.

​
Setup

Cloudflare is the easiest way to get started. And if you’re already on Vercel, AWS CloudFront, Akamai, Fastly, Google Cloud CDN, or Netlify, Promptwatch integrates with those directly too.

Already on Cloudflare (~10 min)

Just deploy the Worker (no DNS change). See the Cloudflare Workers guide.

Not on Cloudflare yet (~30–60 min + propagation)

Loop in your DNS / infra owner and share this page. The FAQ for your team section below answers their questions. Then add your site to Cloudflare, point nameservers, turn on the proxy, and deploy the Worker.
The Worker is free up to 100k requests/day; higher-volume sites need a paid Workers plan. See Cloudflare’s Workers pricing.

​
Next step

Forward this to whoever owns your DNS or infrastructure. Setup is about 10 minutes if you’re already on Cloudflare, and it’s reversible. Once the Worker is live, crawler data starts appearing in your dashboard within about a day. Prefer we set it up with your team? Book a 15-minute call: team@promptwatch.com

​
FAQ for your team

One-time, reversible change with near-zero downtime when records are in place first; revert your nameservers anytime. Cloudflare auto-issues and renews free SSL, so HTTPS keeps working (set SSL/TLS mode to Full strict). Up front: disable DNSSEC at your registrar, and keep MX/email records DNS-only.
No. Logging is fire-and-forget (ctx.waitUntil) and runs after the response, so visitors never wait. Errors are swallowed, so a Promptwatch outage can’t affect your pages, and Workers add effectively zero latency.
The Worker sees each request (timestamp, status, method, path, query, content type, client IP, hostname, user agent, referrer), but only AI-crawler visits are kept; visitor requests are discarded server-side. So what’s stored is bot traffic, not your users’ personal data, and it’s all held in the EU, with a DPA available. An optional UA filter can send only crawler requests in the first place.
No. Remove the Worker route or revert DNS anytime; Promptwatch is provider-agnostic (see below).
None. The script is static; bot detection lives server-side at Promptwatch.
The one real feasibility check: Cloudflare can’t cleanly sit in front of another CDN; for the tracked hostname it replaces the public edge. Happy on your current edge? Use one of our other native integrations instead.
Cloudflare’s AI Crawl Control (free) shows who crawled and lets you block them, but not whether that turned into your brand being cited or visible across ChatGPT, Perplexity, and Claude. That correlation is what Promptwatch adds.