What Are Crawler Logs?
Crawler logs are server-side HTTP request logs that record when AI crawlers, such as GPTBot, ClaudeBot, PerplexityBot, and Google’s AI crawlers visit your website. They’re typically sourced directly from your CDN or hosting provider. Promptwatch correlates these logs with your AI visibility and citation data to go beyond simple tracking. You can analyze crawl trends in real time and over time, understand which pages are being crawled most, and connect crawler activity to actual brand mentions and citation patterns across AI platforms.Setup guide
Step 1: Create an API Key
Create an API key in Promptwatch to authenticate crawler log requests. Each project requires its own API key.- Go to Settings → API Keys in your project dashboard
- Click Create API Key
- Copy and securely store your API key
Step 2: Choose Your Integration
Select your hosting provider or CDN below for step-by-step setup instructions:
Vercel
Stream logs from Vercel Log Drains
Vercel
Stream logs from Vercel Log Drains
Cloudflare Logpush
Push logs from Cloudflare (Enterprise)
Cloudflare Workers
Deploy a Worker to analyze your traffic
AWS CloudFront
Stream logs from CloudFront via Firehose
Akamai
Stream logs from Akamai Edge
Fastly
Stream logs from Fastly CDN
Google Cloud CDN
Stream logs from GCP via Pub/Sub
Netlify
Stream logs from Netlify Log Drains (Enterprise)
Manual Setup
Send HTTP request logs manually