Skip to main content
This integration ingests Google Cloud CDN (HTTP Load Balancer) request logs into Promptwatch using a Pub/Sub topic, push subscription, and a Log Router sink. Make sure to only route logs from your project’s website.

​
Prerequisites

  • You are using Google Cloud CDN / HTTP(S) Load Balancer
  • You have access to the GCP project and permissions to create Pub/Sub resources and Log Router sinks

​
Step 1: Service Account (Verification)

If you already know the service account email you’ll use to authorize this setup, note it down. You’ll need it when creating the push subscription in Step 3. In Promptwatch, navigate to Crawler Logs → Settings, click the Google Cloud button, and enter the service account email you select when creating the push subscription. This lets Promptwatch validate incoming requests.
The service account email is used for verification. During the creation of the Pub/Sub push subscription, you’ll select a service account for authentication.

​
Step 2: Create Pub/Sub Topic

In Google Cloud Console, create a Pub/Sub topic for your Cloud CDN logs. Create Pub/Sub topic
  • Navigate to Pub/Sub → Topics
  • Click Create topic, name it, and create it

​
Step 3: Create Push Subscription

Create a push subscription for the topic you created. Create Pub/Sub push subscription
  • Navigate to Pub/Sub → Subscriptions
  • Click Create subscription
  • Give your subscription an ID
  • Select the topic you created in Step 2
  • Set Delivery type to Push
  • Set the endpoint URL to:
https://logs.promptwatch.com/event/gcp
  • Click Enable authentication and select a service account
  • Enable payload unwrapping
Use the same service account email here as the one you’ll save in Promptwatch (Step 1).
You can also adjust retention, retry policy, and throughput settings in this setup.

​
Step 4: Add Message Transform

Add a Pub/Sub message transform to reduce the payload before sending to Promptwatch. Add a subscription transform
  • In your subscription, click Add a transform
  • Set the function name to log_reduce
  • Paste this function and click Validate:
/* This is used to filter the unused fields before sending to the analytics service.
 *
 * @param {Object} message - The message to reduce.
 * @param {Object} metadata - The metadata of the message.
 * @returns {Object} The reduced message.
 */

function log_reduce(message, metadata) {
  // parse the message
  const data = JSON.parse(message.data);

  // remove unused fields
  delete data['resource'];
  delete data['severity'];
  delete data['spanId'];
  delete data['trace'];
  delete data['insertId'];
  delete data['jsonPayload'];
  delete data['logName'];
  delete data['receiveTimestamp'];

  message.data = JSON.stringify(data);

  return message;
}
This transform drops unneeded fields before sending to Promptwatch. It reduces export volume, improves performance, and enhances data security.

​
Step 5: Create Log Router Sink

Create a Log Router sink to export the HTTP Load Balancer logs into your Pub/Sub topic. Create a Log Router sink
  • Search for Log Router (or go to Logging → Log Router)
  • Click Create sink
  • Give your sink a name
  • For destination, choose Cloud Pub/Sub topic and select the topic from Step 2
  • Add a filter similar to the following:
resource.type="http_load_balancer"
resource.labels.forwarding_rule_name="MY-FORWARDING-RULE-NAME"
resource.labels.url_map_name="MY-URL-MAP-NAME"
Replace MY-FORWARDING-RULE-NAME and MY-URL-MAP-NAME with your real values. Use “Preview logs” in Log Router to confirm what will and won’t be exported.
If you have multiple load balancers / URL maps, expand the filter like this: 
resource.type="http_load_balancer"
AND (
  (resource.labels.forwarding_rule_name="MY-FORWARDING-RULE-NAME"
   AND resource.labels.url_map_name="MY-URL-MAP-NAME")
  OR
  (resource.labels.forwarding_rule_name="MY-FORWARDING-RULE-NAME-2"
   AND resource.labels.url_map_name="MY-URL-MAP-NAME-2")
)
If you are registering multiple domains, create a separate service account per domain and save the correct service account email in Step 1 for each project.

​
Step 6: Save Service Account in Promptwatch

In Promptwatch, navigate to Crawler Logs → Settings, click the Google Cloud button, and enter the service account email you selected when creating the push subscription in Step 3. Google Cloud CDN settings Once your first crawler traffic hits your site, logs should begin appearing in Promptwatch.