> ## Documentation Index
> Fetch the complete documentation index at: https://promptwatch.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# HubSpot + Cloudflare

> Send Cloudflare logs for HubSpot CMS sites

![HubSpot](https://www.promptwatch.com/docs/integration-logos/card/hubspot.svg)

\+

![Cloudflare](https://www.promptwatch.com/docs/integration-logos/card/cloudflare.png)

Send Cloudflare logs for HubSpot CMS sites

[Don't have Cloudflare yet?](https://promptwatch.com/docs/crawler-logs/why-cloudflare)

Promptwatch can capture crawler logs when your HubSpot site proxies traffic through your own Cloudflare zone. This is called **Origin-to-Origin (O2O)** — traffic hits your Cloudflare zone first, then forwards to HubSpot. Configure DNS first, then connect Cloudflare with Logpush or a Worker below.

HubSpot serves sites from its own Cloudflare zone and doesn't share request logs, so adding your own Cloudflare zone in front is how you get crawler logs for a HubSpot site. The Worker setup works on any Cloudflare plan, including Free.

> [!NOTE]
>
> Crawler logs start flowing once setup is complete. Logs from before the setup can't be imported.

> [!WARNING]
>
> Cloudflare's HubSpot guide doesn't list a plan requirement, but HubSpot's reverse proxy docs mention Cloudflare Enterprise for this setup. If HubSpot's **Verify Reverse Proxy Connection** check fails on a non-Enterprise zone, contact support before continuing.

## Step 1: Set Up Proxied DNS for HubSpot

Set up proxied DNS for HubSpot in your Cloudflare zone. Follow [Cloudflare's HubSpot guide](https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/saas-customers/provider-guides/hubspot/) and [HubSpot's reverse proxy setup guide](https://developers.hubspot.com/docs/cms/best-practices/testing-staging-performance/reverse-proxies/setup):

1. In HubSpot, go to **Settings → Website → Domains & URLs** and connect your domain as a reverse proxy: click **Connect a domain**, choose **Primary** or **Secondary**, and switch off **Connect with HubSpot's built-in content delivery network (CDN)**.
2. Create a proxied **CNAME** record pointing your hostname to `[HubID].sites-proxy.hscoscdn[##].net` (see the table below).
3. Set **Proxy status** to Proxied (orange cloud).
4. Under **Rules → Transform Rules**, create a **Modify Request Header** rule for your hostname that sets `X-HS-Public-Host` to your domain (for example `www.example.com`) and `X-HubSpot-Trust-Forwarded-For` to `true`.
5. In HubSpot, open the **Edit** menu next to your domain and select **Verify Reverse Proxy Connection**.

`[HubID]` is your HubSpot account ID, shown in the account menu in the top right of HubSpot. The rest of the target depends on your account's data center:

| Data center | CNAME target |
| - | - |
| US East | `[HubID].sites-proxy.hscoscdn[##].net` |
| US West | `[HubID].sites-proxy.hscoscdn-na2.net` |
| Canada | `[HubID].sites-proxy.hscoscdn-na3.net` |
| European Union | `[HubID].sites-proxy.hscoscdn-eu1.net` |
| Australia | `[HubID].sites-proxy.hscoscdn-ap1.net` |

For US East accounts, `[##]` comes from the last two digits of your Hub ID: `00` for 00–10 and 50–59, `10` for 11–19 and 60–69, `20` for 20–29 and 70–79, `30` for 30–39 and 80–89, and `40` for 40–49 and 90–99. For example, Hub ID `123456` becomes `123456.sites-proxy.hscoscdn00.net`.

> [!NOTE]
>
> Your domain's nameservers must be managed at Cloudflare. Use the `sites-proxy` target, not the `group##.sites.hubspot.net` CNAME from HubSpot's SSL pre-provisioning step. If your Cloudflare zone has a zone hold enabled, release it before adding the CNAME record, otherwise HubSpot can't verify the hostname.

> [!WARNING]
>
> Your DNS must be set up via **CNAME** — A records are not supported for forwarding traffic through Cloudflare Workers. If your root domain uses A records pointing to HubSpot, connect the `www` hostname through Cloudflare and redirect the root domain to it.

> [!WARNING]
>
> If your HubSpot site is served on `www`, the Worker needs a route for `www.example.com/*`. Auto Deploy only creates a route for the root domain (`example.com/*`), so add a second route in Cloudflare under **Workers Routes** that points `www.example.com/*` to the same Worker. In the Manual tab, use `www.example.com/*` as the route in Step 6.

> [tab] Logpush
>
> ## Step 2: Create an API Key
>
> Create an API key in Promptwatch. Go to **Settings → API Keys** in your [Promptwatch dashboard](https://app.promptwatch.com).
>
> ## Step 3: Create a Logpush Job
>
> Go to your Cloudflare dashboard, select your domain and navigate to **Analytics & Logs** > **Logpush** > **Create a Logpush job**.
>
> ## Step 4: Configure HTTP Destination
>
> For destination select **HTTP destination**. Enter the following HTTP endpoint URL:
>
> ```
> https://logs.promptwatch.com/event/cloudflare?header_authorization=Bearer%20YOUR_API_KEY
> ```
>
> Replace `YOUR_API_KEY` with your actual API key.
>
> ## Step 5: Configure Logpush Job
>
> For dataset select **HTTP Requests** and configure the logpush job as follows:
>
> **Name:** `promptwatch-crawler-logs`
>
> **If logs match:**
>
> * Field: ClientRequestHost
> * Operator: Equals
> * Value: YOUR\_DOMAIN (example: app.promptwatch.com)
>
> **Send the following fields:**
>
> *Request*
>
> * ClientIP
> * ClientRequestHost
> * ClientRequestMethod
> * ClientRequestReferer
> * ClientRequestURI
> * ClientRequestUserAgent
>
> *General*
>
> * WorkerSubrequest (allows us to filter out requests initiated from a Worker, since they will be seen as duplicate)
>
> *Performance*
>
> * EdgeStartTimestamp
>
> *Response*
>
> * EdgeResponseContentType
> * EdgeResponseStatus
>
> **Advanced options:** Keep the default settings as is. The timestamp format should be `rfc3339`, which is the default format.
>
> ## Optional: Filter on known AI Crawler user agents
>
> Promptwatch identifies AI crawlers automatically and only stores AI crawler visits, all other traffic is discarded. You can safely send all your server logs without worrying about non-crawler data being retained.
>
> If you prefer to only forward AI crawler traffic, you can use the user agents listed below to pre-filter on your side. Keep in mind that you'll need to maintain this list yourself as new crawlers emerge.
>
> > [accordion] OpenAI (4)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | GPT Bot | `GPTBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot` | Used to crawl content for training OpenAI's generative AI foundation models. |
> > | SearchBot | `OAI-SearchBot` | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot` | Used by ChatGPT search to surface websites in search results. |
> > | ChatGPT Citations | `ChatGPT-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot` | Used for user actions in ChatGPT when visiting web pages. |
> > | Ads Bot | `OAI-AdsBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot` | Used to validate the safety of web pages submitted as ads on ChatGPT. |
>
> > [accordion] Anthropic (4)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Claude Bot | `ClaudeBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com)` | Used to crawl content for training Anthropic's generative AI models. |
> > | Claude Citations | `Claude-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +claudebot@anthropic.com)` | When individuals ask questions to Claude or use Claude Code, it may access websites using a Claude-User agent. |
> > | Claude Search Bot | `Claude-SearchBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com)` | Navigates the web to improve search result quality for users. |
> > | Claude Web | `claude-web` | `Mozilla/5.0 (compatible; claude-web/1.0; +http://www.anthropic.com/bot.html)` | Targeted crawler for recent web content, feeding the Claude browser agent with updated site data. |
>
> > [accordion] Google (3)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Gemini | `Google-Extended` | `Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)` | Controls whether content can be used for training Gemini AI models. |
> > | Google Mobile Agent | `Google-Agent` | `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)` | Used by Google AI agents to autonomously browse the web and complete tasks on behalf of users (mobile). |
> > | Google Desktop Agent | `Google-Agent` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36` | Used by Google AI agents to autonomously browse the web and complete tasks on behalf of users (desktop). |
>
> > [accordion] Perplexity (2)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Perplexity Bot | `PerplexityBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` | Used to surface and link websites in Perplexity search results. |
> > | Perplexity Citations | `Perplexity-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` | Used for user actions in Perplexity when visiting web pages to answer questions. |
>
> > [accordion] Cohere (1)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Cohere AI | `cohere-ai` | `Mozilla/5.0 (compatible; cohere-ai/1.0; +http://www.cohere.ai/bot.html)` | Collects textual data for Cohere's language models, helping refine large-scale text generation. |
>
> > [accordion] Mistral (2)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Mistral AI Citations | `MistralAI-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)` | Used for user actions in LeChat when visiting web pages to answer questions. |
> > | Mistral AI Index | `MistralAI-Index` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)` | Used to index content for Mistral AI's search engine, which helps answer user questions in Le Chat. |
>
> > [accordion] DeepSeek (1)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | DeepSeek | `DeepSeekBot` | `Mozilla/5.0 (compatible; DeepSeekBot/1.0; +http://www.deepseek.com/bot.html)` | Used to crawl content for training DeepSeek's generative AI models. |
>
> > [accordion] xAI / Grok (3)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Grok Bot | `GrokBot` | `GrokBot/1.0 (+https://x.ai)` | Used for training Grok AI. |
> > | Grok Search | `xAI-Grok` | `xAI-Grok/1.0 (+https://grok.com)` | Used for Grok's search capabilities. |
> > | Grok Deep Search | `Grok-DeepSearch` | `Grok-DeepSearch/1.0 (+https://x.ai)` | Used for Grok's advanced search capabilities. |
>
> > [accordion] Exa (2)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Exa Search Bot | `ExaSearchBot` | `Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)` | Crawls the public web to build the index behind Exa's search API, which AI agents and apps use to find and cite pages. |
> > | Exa Citations | `ExaUser` | `Mozilla/5.0 (compatible; ExaUser/1.0; +https://exauser.crawler.exa.ai)` | Fetches a page live when a user, or an application built on the Exa API, asks for its current content. |
>
> ## Step 6: Verify Cloudflare Proxy is Enabled
>
> Verify your domain's DNS records have the **Cloudflare proxy enabled** (orange cloud icon). Logpush for HTTP Requests only captures traffic that flows through the Cloudflare network. If the proxy is off (gray cloud / "DNS only"), Cloudflare never sees the requests and can't push logs.
>
> Go to your Cloudflare dashboard → **DNS** → **Records** and make sure the **Proxy status** shows the orange cloud ("Proxied") for your domain's A/AAAA/CNAME records.
>
> ![Cloudflare DNS proxy enabled](https://mintlify.s3.us-west-1.amazonaws.com/promptwatch/crawler-logs/cloudflare-proxy-enabled.jpeg)
>
> ## Step 7: Finish
>
> Once submitted, your Cloudflare Logpush job will automatically send crawler logs to Promptwatch when an AI crawler visits your website. Depending on how active your website is, this can take a few minutes to appear.

> [tab] Auto Deploy
>
> Create an access token so Promptwatch can deploy the crawler log Worker for you
>
> We'll automatically discover your Cloudflare zone, upload the script, and configure the route.
>
> The token is only used for this deployment and is never stored.
>
> ### Step 1: Create an API token on Cloudflare
>
> Go to 
> [dash.cloudflare.com → My Profile → API Tokens](https://dash.cloudflare.com/profile/api-tokens) 
> and create a new Custom Token.
>
> Enable the following permissions:
>
> * **Account / Workers Scripts / Edit** — Upload the crawler-log Worker script
> * **Zone / Zone / Read** — Locate your Cloudflare zone by domain name
> * **Zone / Workers Routes / Edit** — Attach the script to your domain route
>
> ### Step 2: Paste the token and deploy
>
> We'll take it from here — the rest is automatic.
>
> > [!NOTE]
> >
> > Auto-deploy is only available inside the Promptwatch app. Open **Crawler Logs → Settings**, select **HubSpot + Cloudflare**, switch to the **Auto Deploy** tab, paste your Cloudflare API token, and click **Deploy Worker**.

> [tab] Manual
>
> ## Step 2: Create an API Key
>
> Create an API key in Promptwatch. Go to **Settings → API Keys** in your [Promptwatch dashboard](https://app.promptwatch.com).
>
> ## Step 3: Create an Application
>
> Create a new application in your Cloudflare account:
>
> 1. Go to your [Cloudflare dashboard](https://dash.cloudflare.com) → **Workers & Pages**
> 2. Click **Create application**
> 3. Click **Start with Hello World!**
> 4. Give it a name (e.g., `promptwatch-analytics`)
> 5. Click **Deploy**
>
> ## Step 4: Paste the Worker Script
>
> On the Worker page, click **Edit** in the top right corner and paste in the code below:
>
> ```javascript theme={null} theme={null} theme={null} theme={null}
> /**
>  * Cloudflare Worker - stream page views to analytics server
>  *
>  * Sends JSON in the expected crawler logs schema.
>  * Content type filtering is handled server-side.
>  */
> const ANALYTICS_ENDPOINT = "https://logs.promptwatch.com/event";
>
> export default {
>   async fetch(request, env, ctx) {
>     const { pathname, search } = new URL(request.url);
>
>     // 1. Make the upstream request (cache/origin)
>     const response = await fetch(request);
>
>     const contentType = response.headers.get("content-type")?.split(";")[0];
>     const queryString = search.startsWith("?") && search.length > 1 ? search.slice(1) : null;
>
>     // 2. Build the log object
>     const log = {
>       timestamp: new Date().toISOString(),
>       status_code: response.status,
>       request_method: request.method,
>       request_path: pathname,
>       query_string: queryString,
>       content_type: contentType,
>       client_ip: request.headers.get("CF-Connecting-IP") || null,
>       hostname: request.headers.get("Host") || null,
>       user_agent: request.headers.get("User-Agent") || null,
>       referrer: request.headers.get("Referer") || null,
>     };
>
>     // 3. Fire-and-forget upload - never delays the visitor
>     ctx.waitUntil(
>       fetch(ANALYTICS_ENDPOINT, {
>         method: "POST",
>         headers: {
>           "Content-Type": "application/json",
>           "X-API-Key": env.ANALYTICS_KEY,
>         },
>         body: JSON.stringify(log),
>       }).catch(() => {}), // swallow network errors
>     );
>
>     return response;
>   },
> };
> ```
>
> Promptwatch identifies AI crawlers automatically and only stores AI crawler visits, all other traffic is discarded. You can safely send all your server logs without worrying about non-crawler data being retained.
>
> If you prefer to only forward AI crawler traffic, you can use the user agents listed below to pre-filter on your side. Keep in mind that you'll need to maintain this list yourself as new crawlers emerge.
>
> > [accordion] OpenAI (4)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | GPT Bot | `GPTBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot` | Used to crawl content for training OpenAI's generative AI foundation models. |
> > | SearchBot | `OAI-SearchBot` | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot` | Used by ChatGPT search to surface websites in search results. |
> > | ChatGPT Citations | `ChatGPT-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot` | Used for user actions in ChatGPT when visiting web pages. |
> > | Ads Bot | `OAI-AdsBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot` | Used to validate the safety of web pages submitted as ads on ChatGPT. |
>
> > [accordion] Anthropic (4)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Claude Bot | `ClaudeBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com)` | Used to crawl content for training Anthropic's generative AI models. |
> > | Claude Citations | `Claude-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +claudebot@anthropic.com)` | When individuals ask questions to Claude or use Claude Code, it may access websites using a Claude-User agent. |
> > | Claude Search Bot | `Claude-SearchBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com)` | Navigates the web to improve search result quality for users. |
> > | Claude Web | `claude-web` | `Mozilla/5.0 (compatible; claude-web/1.0; +http://www.anthropic.com/bot.html)` | Targeted crawler for recent web content, feeding the Claude browser agent with updated site data. |
>
> > [accordion] Google (3)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Gemini | `Google-Extended` | `Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)` | Controls whether content can be used for training Gemini AI models. |
> > | Google Mobile Agent | `Google-Agent` | `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)` | Used by Google AI agents to autonomously browse the web and complete tasks on behalf of users (mobile). |
> > | Google Desktop Agent | `Google-Agent` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36` | Used by Google AI agents to autonomously browse the web and complete tasks on behalf of users (desktop). |
>
> > [accordion] Perplexity (2)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Perplexity Bot | `PerplexityBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` | Used to surface and link websites in Perplexity search results. |
> > | Perplexity Citations | `Perplexity-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` | Used for user actions in Perplexity when visiting web pages to answer questions. |
>
> > [accordion] Cohere (1)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Cohere AI | `cohere-ai` | `Mozilla/5.0 (compatible; cohere-ai/1.0; +http://www.cohere.ai/bot.html)` | Collects textual data for Cohere's language models, helping refine large-scale text generation. |
>
> > [accordion] Mistral (2)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Mistral AI Citations | `MistralAI-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)` | Used for user actions in LeChat when visiting web pages to answer questions. |
> > | Mistral AI Index | `MistralAI-Index` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)` | Used to index content for Mistral AI's search engine, which helps answer user questions in Le Chat. |
>
> > [accordion] DeepSeek (1)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | DeepSeek | `DeepSeekBot` | `Mozilla/5.0 (compatible; DeepSeekBot/1.0; +http://www.deepseek.com/bot.html)` | Used to crawl content for training DeepSeek's generative AI models. |
>
> > [accordion] xAI / Grok (3)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Grok Bot | `GrokBot` | `GrokBot/1.0 (+https://x.ai)` | Used for training Grok AI. |
> > | Grok Search | `xAI-Grok` | `xAI-Grok/1.0 (+https://grok.com)` | Used for Grok's search capabilities. |
> > | Grok Deep Search | `Grok-DeepSearch` | `Grok-DeepSearch/1.0 (+https://x.ai)` | Used for Grok's advanced search capabilities. |
>
> > [accordion] Exa (2)
> >
> > | Name | User Agent | Full User Agent | Description |
> > | - | - | - | - |
> > | Exa Search Bot | `ExaSearchBot` | `Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)` | Crawls the public web to build the index behind Exa's search API, which AI agents and apps use to find and cite pages. |
> > | Exa Citations | `ExaUser` | `Mozilla/5.0 (compatible; ExaUser/1.0; +https://exauser.crawler.exa.ai)` | Fetches a page live when a user, or an application built on the Exa API, asks for its current content. |
>
> ## Step 5: Environment Variables
>
> In your Worker settings, add a secret for your API key:
>
> 1. Open the **Settings** tab for your Worker
> 2. Under **Variables and secrets**, click **Add**
> 3. Select **Secret**
> 4. Set the name to `ANALYTICS_KEY`
> 5. Set the value to the API key you created in step 2
>
> ## Step 6: Deploy to a Route
>
> Deploy your Worker to a route:
>
> 1. Navigate to **Domains**
>
> 2. Select the domain you're setting up
>
> 3. In the left-hand navigation, click **Worker Routes**
>
> 4. Add a new route: `yourdomain.com/*`
>
> 5. Select your Worker from the dropdown
>
> 6. Save and deploy
>
> > [!WARNING]
> >
> > The Worker will intercept all requests to your domain. It transparently passes through all traffic while logging HTML page views to Promptwatch.
>
> ## Step 7: Verify Cloudflare Proxy is Enabled
>
> Verify your domain's DNS records have the **Cloudflare proxy enabled** (orange cloud icon). Workers Routes only intercept traffic that flows through the Cloudflare network. If the proxy is off (gray cloud / "DNS only"), requests go directly to your origin and bypass the Worker entirely.
>
> Go to your Cloudflare dashboard → **DNS** → **Records** and make sure the **Proxy status** shows the orange cloud ("Proxied") for your domain's A/AAAA/CNAME records.
>
> ![Cloudflare DNS proxy enabled](https://mintlify.s3.us-west-1.amazonaws.com/promptwatch/crawler-logs/cloudflare-proxy-enabled.jpeg)
>
> ## Step 8: Finish
>
> Once deployed, your Worker will automatically send crawler logs to Promptwatch.

> [!NOTE]
>
> For a full list of API response codes, see [API Response Codes](https://promptwatch.com/docs/crawler-logs/api-responses.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.