Send Cloudflare logs for HubSpot CMS sites
Promptwatch can capture crawler logs when your HubSpot site proxies traffic through your own Cloudflare zone. This is called Origin-to-Origin (O2O) — traffic hits your Cloudflare zone first, then forwards to HubSpot. Configure DNS first, then connect Cloudflare with Logpush or a Worker below. HubSpot serves sites from its own Cloudflare zone and doesn’t share request logs, so adding your own Cloudflare zone in front is how you get crawler logs for a HubSpot site. The Worker setup works on any Cloudflare plan, including Free.Crawler logs start flowing once setup is complete. Logs from before the setup can’t be imported.
Step 1: Set Up Proxied DNS for HubSpot
Set up proxied DNS for HubSpot in your Cloudflare zone. Follow Cloudflare’s HubSpot guide and HubSpot’s reverse proxy setup guide:- In HubSpot, go to Settings → Website → Domains & URLs and connect your domain as a reverse proxy: click Connect a domain, choose Primary or Secondary, and switch off Connect with HubSpot’s built-in content delivery network (CDN).
- Create a proxied CNAME record pointing your hostname to
[HubID].sites-proxy.hscoscdn[##].net(see the table below). - Set Proxy status to Proxied (orange cloud).
- Under Rules → Transform Rules, create a Modify Request Header rule for your hostname that sets
X-HS-Public-Hostto your domain (for examplewww.example.com) andX-HubSpot-Trust-Forwarded-Fortotrue. - In HubSpot, open the Edit menu next to your domain and select Verify Reverse Proxy Connection.
[HubID] is your HubSpot account ID, shown in the account menu in the top right of HubSpot. The rest of the target depends on your account’s data center:
For US East accounts,
[##] comes from the last two digits of your Hub ID: 00 for 00–10 and 50–59, 10 for 11–19 and 60–69, 20 for 20–29 and 70–79, 30 for 30–39 and 80–89, and 40 for 40–49 and 90–99. For example, Hub ID 123456 becomes 123456.sites-proxy.hscoscdn00.net.
Your domain’s nameservers must be managed at Cloudflare. Use the
sites-proxy target, not the group##.sites.hubspot.net CNAME from HubSpot’s SSL pre-provisioning step. If your Cloudflare zone has a zone hold enabled, release it before adding the CNAME record, otherwise HubSpot can’t verify the hostname.- Logpush
- Auto Deploy
- Manual
Step 2: Create an API Key
Create an API key in Promptwatch. Go to Settings → API Keys in your Promptwatch dashboard.Step 3: Create a Logpush Job
Go to your Cloudflare dashboard, select your domain and navigate to Analytics & Logs > Logpush > Create a Logpush job.Step 4: Configure HTTP Destination
For destination select HTTP destination. Enter the following HTTP endpoint URL:YOUR_API_KEY with your actual API key.Step 5: Configure Logpush Job
For dataset select HTTP Requests and configure the logpush job as follows:Name:promptwatch-crawler-logsIf logs match:- Field: ClientRequestHost
- Operator: Equals
- Value: YOUR_DOMAIN (example: app.promptwatch.com)
- ClientIP
- ClientRequestHost
- ClientRequestMethod
- ClientRequestReferer
- ClientRequestURI
- ClientRequestUserAgent
- WorkerSubrequest (allows us to filter out requests initiated from a Worker, since they will be seen as duplicate)
- EdgeStartTimestamp
- EdgeResponseContentType
- EdgeResponseStatus
rfc3339, which is the default format.Optional: Filter on known AI Crawler user agents
Promptwatch identifies AI crawlers automatically and only stores AI crawler visits, all other traffic is discarded. You can safely send all your server logs without worrying about non-crawler data being retained.If you prefer to only forward AI crawler traffic, you can use the user agents listed below to pre-filter on your side. Keep in mind that you’ll need to maintain this list yourself as new crawlers emerge.OpenAI (4)
OpenAI (4)
Anthropic (4)
Anthropic (4)
Google (3)
Google (3)
Perplexity (2)
Perplexity (2)
Cohere (1)
Cohere (1)
Mistral (2)
Mistral (2)
DeepSeek (1)
DeepSeek (1)
xAI / Grok (3)
xAI / Grok (3)
Exa (2)
Exa (2)
Step 6: Verify Cloudflare Proxy is Enabled
Verify your domain’s DNS records have the Cloudflare proxy enabled (orange cloud icon). Logpush for HTTP Requests only captures traffic that flows through the Cloudflare network. If the proxy is off (gray cloud / “DNS only”), Cloudflare never sees the requests and can’t push logs.Go to your Cloudflare dashboard → DNS → Records and make sure the Proxy status shows the orange cloud (“Proxied”) for your domain’s A/AAAA/CNAME records.
Step 7: Finish
Once submitted, your Cloudflare Logpush job will automatically send crawler logs to Promptwatch when an AI crawler visits your website. Depending on how active your website is, this can take a few minutes to appear.For a full list of API response codes, see API Response Codes.