What Are Crawler Logs?
Crawler logs are server-side HTTP request logs that record when AI crawlers, such as GPTBot, ClaudeBot, PerplexityBot, and Google’s AI crawlers visit your website. They’re typically sourced directly from your CDN or hosting provider. Promptwatch correlates these logs with your AI visibility and citation data to go beyond simple tracking. You can analyze crawl trends in real time and over time, understand which pages are being crawled most, and connect crawler activity to actual brand mentions and citation patterns across AI platforms.Setup guide
Step 1: Create an API Key
Create an API key in Promptwatch to authenticate crawler log requests. Each project requires its own API key.- Go to Settings → API Keys in your project dashboard
- Click Create API Key
- Copy and securely store your API key
Step 2: Choose Your Integration
Select your CDN provider or platform below for step-by-step setup instructions.CDN Providers
Akamai
Stream logs from Akamai Edge
AWS CloudFront
Stream logs from CloudFront via Firehose
Fastly
Stream logs from Fastly CDN
Google Cloud CDN
Stream logs from GCP via Pub/Sub
Netlify
Stream logs from Netlify Log Drains (Enterprise)
Vercel
Stream logs from Vercel Log Drains
Vercel
Stream logs from Vercel Log Drains
Cloudflare (Logpush)
Push logs from Cloudflare (Enterprise)
Cloudflare (Workers)
Deploy a Worker to analyze your traffic
Manual Setup
Send HTTP request logs manually
Platform
WP Engine + Cloudflare
Deploy a Cloudflare Worker for WP Engine sites
Webflow + Cloudflare
Deploy a Cloudflare Worker for Webflow sites
Shopify + Cloudflare
Deploy a Cloudflare Worker for Shopify stores
For a full list of API response codes, see API Response Codes.