> ## Documentation Index
> Fetch the complete documentation index at: https://promptwatch.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS CloudFront

> Configure crawler log ingestion via Amazon Kinesis Firehose and CloudFront Logging v2

<div style={{ display: "flex", alignItems: "center", justifyContent: "center", marginBottom: 0 }}>
  <img alt="AWS CloudFront" height="40" src="https://www.promptwatch.com/docs/integration-logos/card/aws-cloudfront.svg" width="40" />
</div>

<p style={{ textAlign: "center", color: "#6b7280", marginTop: 8, marginBottom: 24 }}>
  Stream logs from CloudFront via Firehose
</p>

This integration uses **Amazon Kinesis Data Firehose** and **CloudFront Logging v2** to forward CloudFront logs to the Promptwatch ingestion endpoint.

## Prerequisites

* You have a CloudFront distribution with Standard Logging v2 enabled or available
* You have access to the AWS Console with permissions to create Kinesis Firehose delivery streams

## Step 1: Create an API Key

Create an API key in Promptwatch. Go to **Settings → API Keys** in your [Promptwatch dashboard](https://app.promptwatch.com).

## Step 2: Create Firehose Delivery Stream

In the AWS Console, navigate to Amazon Data Firehose and create a new delivery stream.

<img src="https://mintcdn.com/promptwatch/juWgwR7Z8y8ygRMF/cf_setup_guide/cf1.png?fit=max&auto=format&n=juWgwR7Z8y8ygRMF&q=85&s=0762996aba09124fa826763c81a7031d" alt="Navigate to Kinesis Firehose" width="2450" height="398" data-path="cf_setup_guide/cf1.png" />

* Go to **Firehose** in the AWS Console
* Click **Create Firehose stream**

## Step 3: Configure Delivery Stream

Configure the delivery stream to send logs to Promptwatch via HTTP endpoint.

<img src="https://mintcdn.com/promptwatch/juWgwR7Z8y8ygRMF/cf_setup_guide/cf2.png?fit=max&auto=format&n=juWgwR7Z8y8ygRMF&q=85&s=030910de0f90cb50e98d4216eefa59eb" alt="Create delivery stream" width="3444" height="1506" data-path="cf_setup_guide/cf2.png" />

* Set **Source** to **Direct PUT**
* Set **Destination** to **HTTP endpoint delivery**
* **Optional:** Name the stream something descriptive

**Within Destination settings:**

Enter the following **HTTP endpoint URL**:

```
https://logs.promptwatch.com/event/cloudfront
```

* **Set the access key:** Enter your API key from Step 1

<Info>
  We recommend storing your API key in [AWS Secrets Manager](https://docs.aws.amazon.com/firehose/latest/dev/secrets-manager-whats-secret.html).
</Info>

* **Set Content Encoding to GZIP**

<img src="https://mintcdn.com/promptwatch/juWgwR7Z8y8ygRMF/cf_setup_guide/cf3.png?fit=max&auto=format&n=juWgwR7Z8y8ygRMF&q=85&s=967363f75b7e4ac6136a184b8860b725" alt="Configure HTTP endpoint delivery" width="3452" height="1616" data-path="cf_setup_guide/cf3.png" />

## Step 4: Create S3 Bucket for Failed Logs

* **Create or select an S3 bucket** to store failed delivery logs (required by AWS)

<img src="https://mintcdn.com/promptwatch/juWgwR7Z8y8ygRMF/cf_setup_guide/cf4.png?fit=max&auto=format&n=juWgwR7Z8y8ygRMF&q=85&s=f9986bf830605f39953132ebf69ce888" alt="Configure CloudFront logging" width="3010" height="1152" data-path="cf_setup_guide/cf4.png" />

## Step 5: Attach Firehose to CloudFront

Now attach the Firehose delivery stream to your distribution.

* Navigate to your CloudFront distribution → **Logging** tab → **Add** → **Amazon Kinesis Data Firehose**

<img src="https://mintcdn.com/promptwatch/juWgwR7Z8y8ygRMF/cf_setup_guide/cf5.png?fit=max&auto=format&n=juWgwR7Z8y8ygRMF&q=85&s=6664a1b353515a78bffe2eaad720617a" alt="CloudFront add Firehose" width="2284" height="710" data-path="cf_setup_guide/cf5.png" />

## Step 6: Select Delivery Stream

Select the Firehose delivery stream you created in Step 2.

<img src="https://mintcdn.com/promptwatch/juWgwR7Z8y8ygRMF/cf_setup_guide/cf5.5.png?fit=max&auto=format&n=juWgwR7Z8y8ygRMF&q=85&s=8bf6845d887ed053c26abc109360fad9" alt="Select Firehose stream" width="2086" height="712" data-path="cf_setup_guide/cf5.5.png" />

## Step 7: Configure Log Format

Configure the log format and field selection in CloudFront.

* Select **Additional settings**
* For **Field selection**, select exactly these 10 fields:

1. `timestamp`
2. `c-ip`
3. `sc-status`
4. `cs-method`
5. `cs-uri-stem`
6. `cs-uri-query`
7. `cs(Host)`
8. `cs(User-Agent)`
9. `cs(Referer)`
10. `sc-content-type`

<Warning>
  Promptwatch requires these specific fields to parse crawler logs. Omitting any of them will cause logs to be rejected.
</Warning>

## Optional: Filter on known AI Crawler user agents

Promptwatch identifies AI crawlers automatically and only stores AI crawler visits, all other traffic is discarded. You can safely send all your server logs without worrying about non-crawler data being retained.

If you prefer to only forward AI crawler traffic, you can use the user agents listed below to pre-filter on your side. Keep in mind that you'll need to maintain this list yourself as new crawlers emerge.

<AccordionGroup>
  <Accordion title="OpenAI (4)">
    | Name              | User Agent      | Full User Agent                                                                                                                                                                       | Description                                                                  |
    | ----------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
    | GPT Bot           | `GPTBot`        | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot`                                                                              | Used to crawl content for training OpenAI's generative AI foundation models. |
    | SearchBot         | `OAI-SearchBot` | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot` | Used by ChatGPT search to surface websites in search results.                |
    | ChatGPT Citations | `ChatGPT-User`  | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot`                                                                           | Used for user actions in ChatGPT when visiting web pages.                    |
    | Ads Bot           | `OAI-AdsBot`    | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot`                                                                          | Used to validate the safety of web pages submitted as ads on ChatGPT.        |
  </Accordion>

  <Accordion title="Anthropic (5)">
    | Name              | User Agent         | Full User Agent                                                                                                  | Description                                                                                                    |
    | ----------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
    | Claude Bot        | `ClaudeBot`        | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com)`       | Used to crawl content for training Anthropic's generative AI models.                                           |
    | Claude Citations  | `Claude-User`      | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +claudebot@anthropic.com)`     | When individuals ask questions to Claude or use Claude Code, it may access websites using a Claude-User agent. |
    | Claude Search Bot | `Claude-SearchBot` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com)` | Navigates the web to improve search result quality for users.                                                  |
    | Claude Web        | `claude-web`       | `Mozilla/5.0 (compatible; claude-web/1.0; +http://www.anthropic.com/bot.html)`                                   | Targeted crawler for recent web content, feeding the Claude browser agent with updated site data.              |
    | Anthropic AI      | `anthropic-ai`     | `Mozilla/5.0 (compatible; anthropic-ai/1.0; +http://www.anthropic.com/bot.html)`                                 | Primary Anthropic crawler that collects broad web data for Claude model development.                           |
  </Accordion>

  <Accordion title="Google (3)">
    | Name                 | User Agent        | Full User Agent                                                                                                                                                                                                                                | Description                                                                                              |
    | -------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
    | Gemini               | `Google-Extended` | `Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)`                                                                                                                                                               | Controls whether content can be used for training Gemini AI models.                                      |
    | Google Mobile Agent  | `Google-Agent`    | `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)` | Used by Google AI agents to autonomously browse the web and complete tasks on behalf of users (mobile).  |
    | Google Desktop Agent | `Google-Agent`    | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36`                                                       | Used by Google AI agents to autonomously browse the web and complete tasks on behalf of users (desktop). |
  </Accordion>

  <Accordion title="Perplexity (2)">
    | Name                 | User Agent        | Full User Agent                                                                                                               | Description                                                                      |
    | -------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
    | Perplexity Bot       | `PerplexityBot`   | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)`     | Used to surface and link websites in Perplexity search results.                  |
    | Perplexity Citations | `Perplexity-User` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` | Used for user actions in Perplexity when visiting web pages to answer questions. |
  </Accordion>

  <Accordion title="Cohere (1)">
    | Name      | User Agent  | Full User Agent                                                           | Description                                                                                     |
    | --------- | ----------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
    | Cohere AI | `cohere-ai` | `Mozilla/5.0 (compatible; cohere-ai/1.0; +http://www.cohere.ai/bot.html)` | Collects textual data for Cohere's language models, helping refine large-scale text generation. |
  </Accordion>

  <Accordion title="Mistral (2)">
    | Name                 | User Agent        | Full User Agent                                                                                                        | Description                                                                                         |
    | -------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
    | Mistral AI Citations | `MistralAI-User`  | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)`  | Used for user actions in LeChat when visiting web pages to answer questions.                        |
    | Mistral AI Index     | `MistralAI-Index` | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)` | Used to index content for Mistral AI's search engine, which helps answer user questions in Le Chat. |
  </Accordion>

  <Accordion title="DeepSeek (1)">
    | Name     | User Agent    | Full User Agent                                                                | Description                                                         |
    | -------- | ------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------- |
    | DeepSeek | `DeepSeekBot` | `Mozilla/5.0 (compatible; DeepSeekBot/1.0; +http://www.deepseek.com/bot.html)` | Used to crawl content for training DeepSeek's generative AI models. |
  </Accordion>

  <Accordion title="xAI / Grok (3)">
    | Name             | User Agent        | Full User Agent                       | Description                                   |
    | ---------------- | ----------------- | ------------------------------------- | --------------------------------------------- |
    | Grok Bot         | `GrokBot`         | `GrokBot/1.0 (+https://x.ai)`         | Used for training Grok AI.                    |
    | Grok Search      | `xAI-Grok`        | `xAI-Grok/1.0 (+https://grok.com)`    | Used for Grok's search capabilities.          |
    | Grok Deep Search | `Grok-DeepSearch` | `Grok-DeepSearch/1.0 (+https://x.ai)` | Used for Grok's advanced search capabilities. |
  </Accordion>
</AccordionGroup>

## Step 8: Finish

Set **Output format** to `JSON`.

Setup is now complete. CloudFront will stream access logs to your Firehose delivery stream, which delivers them to Promptwatch. Crawler logs will usually appear within a few minutes.

<Note>
  For a full list of API response codes, see [API Response Codes](/docs/crawler-logs/api-responses).
</Note>