Skip to main content
Log ingestion streams your server or CDN access logs to Promptwatch, so the platform can show you exactly when GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers visit your site and which pages they fetch. This is the data source behind AI crawler insights.

​
Why logs, and not a tracking script?

AI crawlers don’t run JavaScript, so they’re invisible to normal analytics tools. The only place their visits show up is your server’s access logs. By sending those logs to Promptwatch, crawler activity becomes something you can chart, filter, and correlate with your visibility data: which pages get crawled most, whether crawl activity precedes citations, and which sections of your site the models ignore. For what to do with those answers, see Crawler use cases and Turning crawler logs into opportunities.

​
How the setup works

The setup is the same shape for every provider: create an API key, then configure your CDN or host to forward logs to Promptwatch’s ingestion endpoint.
  1. Go to Settings → API Keys in your project and create an API key. Each project needs its own key, so logs land in the right project.
  2. Pick your CDN or platform and follow its guide in the technical setup docs.
  3. Send traffic. Crawler visits start appearing in Crawler Logs as logs flow in.

​
Supported providers

There’s a step-by-step guide for each:
  • CDNs: Cloudflare (Logpush, or a Worker if your plan doesn’t include Logpush), Akamai, AWS CloudFront, Fastly, Google Cloud CDN, Netlify, and Vercel.
  • Platforms behind Cloudflare: Webflow, Shopify, Kinsta, and WP Engine, each with a guide for putting Cloudflare in front and streaming from there.
  • Everything else: a manual HTTP API, so any server or pipeline that can POST JSON can send logs.
The Promptwatch crawler log integration picker listing CDN providers and Cloudflare-backed platforms. Promptwatch keeps only what’s relevant: requests whose user agent doesn’t match a known AI crawler or search engine bot are dropped at ingestion, as are static assets, so your regular visitor traffic never accumulates in Promptwatch.

​
Verified crawler data

Anyone can fake a user agent string, and scrapers routinely pretend to be GPTBot. For crawlers whose providers publish IP ranges, Promptwatch checks the client IP against those ranges. Requests that claim to be a known bot but come from unverified IPs are left out of your charts unless you explicitly include them. Counts can be lower than a naive user-agent count from your raw logs, and more accurate.

​
After logs are flowing

Open Crawler Logs in the sidebar and you’ll see crawler trends over time, top crawled pages, and per-crawler breakdowns. Give it a few days of data before drawing conclusions: crawl behavior is bursty, and a week of logs tells you far more than an hour. From there, the interesting work is cross-referencing: pages that get crawled but never cited are optimization candidates, and pages that are never crawled may have a crawlability problem. The Promptwatch crawler trend chart with verified daily request volume by crawler over 90 days.