Promptwatch can capture crawler logs when your Shopify store proxies traffic through your own Cloudflare zone. This is called Origin-to-Origin (O2O) — traffic hits your Cloudflare zone first, then forwards to Shopify. Configure DNS first, then deploy the Cloudflare Worker below.
Auto-deploy is only available inside the Promptwatch app. Open Crawler Logs → Settings, select Shopify + Cloudflare, switch to the Auto Deploy tab, paste your Cloudflare API token, and click Deploy Worker.
Promptwatch identifies AI crawlers automatically and only stores AI crawler visits, all other traffic is discarded. You can safely send all your server logs without worrying about non-crawler data being retained.If you prefer to only forward AI crawler traffic, you can use the user agents listed below to pre-filter on your side. Keep in mind that you’ll need to maintain this list yourself as new crawlers emerge.
OpenAI (4)
Name
User Agent
Full User Agent
Description
GPT Bot
GPTBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot
Used to crawl content for training OpenAI’s generative AI foundation models.
SearchBot
OAI-SearchBot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Used by ChatGPT search to surface websites in search results.
ChatGPT Citations
ChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Used for user actions in ChatGPT when visiting web pages.
Ads Bot
OAI-AdsBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
Used to validate the safety of web pages submitted as ads on ChatGPT.
Anthropic (4)
Name
User Agent
Full User Agent
Description
Claude Bot
ClaudeBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Used to crawl content for training Anthropic’s generative AI models.
Claude Citations
Claude-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +claudebot@anthropic.com)
When individuals ask questions to Claude or use Claude Code, it may access websites using a Claude-User agent.
Claude Search Bot
Claude-SearchBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com)
Navigates the web to improve search result quality for users.
Verify your domain’s DNS records have the Cloudflare proxy enabled (orange cloud icon). Workers Routes only intercept traffic that flows through the Cloudflare network. If the proxy is off (gray cloud / “DNS only”), requests go directly to your origin and bypass the Worker entirely.Go to your Cloudflare dashboard → DNS → Records and make sure the Proxy status shows the orange cloud (“Proxied”) for your domain’s A/AAAA/CNAME records.