Skip to main content
Crawler logs and visitor analytics are the two ends of your AI pipeline: bots reading your site in, humans clicking through from answers out. Here are the scenarios where that data earns its keep, each with the steps and the reasoning. They all assume crawler logs and visitor analytics are set up.

​
Catch a block before it costs you visibility

Bot-protection rules block AI crawlers silently: human traffic looks normal while a provider gets 403s for weeks, and your citations on that platform quietly decay.
  1. Open Crawler Logs weekly. The comparison cards cover the ChatGPT, Claude, Perplexity, and Mistral citation bots; scan the crawl trend or Top Crawlers for the other bots. A crawler down sharply against the previous period is your trigger.
Promptwatch citation-crawler comparison cards showing ChatGPT, Claude, and Mistral down versus the previous period.
  1. Filter by that crawler plus status codes 403 and 500. Concentrated 403s mean a firewall or bot-fight rule; check your CDN’s security settings against the bot’s published identity.
  2. After fixing, watch the trend with the 24-hour preset, minute-level granularity shows the crawler returning the same day rather than making you wait a week.
Why weekly: crawl loss precedes citation loss, so this check buys you lead time that visibility metrics can’t.

​
Validate a site migration

After a URL restructure, AI crawlers keep requesting the old paths they have indexed.
  1. In the Top Pages tab, sort by error rate right after the migration. Old paths returning 404 rise to the top.
  2. Redirect every 404’d path that used to earn crawls; the crawler is telling you it still values that content.
  3. Over the following weeks, check the Top Pages tab for the new URLs with a range starting at the migration; First Crawled there is the first request in the selected range.
Why this beats a generic redirect audit: it prioritizes by actual AI crawler demand, so you fix the paths that protect citations first.

​
Measure whether new content gets picked up

You published a cluster of pages and want to know if AI models noticed.
  1. Make sure the sitemap includes the new URLs, sitemap updates are a common crawl-spike trigger.
  2. Filter Crawler Logs by the new section’s path. On the Top Pages tab, set the range to start at publish and read First Crawled: that’s the first request in the selected range, so a range from publish day gives you discovery time per page.
The Promptwatch Top Pages table sorted by First Crawled, with the most recently discovered pages at the top.
  1. Add the pages to the Page Tracker so crawls, citations, and clicks for the cluster sit in one view as they develop.
Why: “crawled but never cited” and “never crawled” need different fixes (content versus discoverability), and this is how you tell them apart per page.

​
Compare providers and spot the missing one

  1. Set Last 90 days and read Top Crawlers for the highest-volume bots. It’s a top-10 list, so use the crawler filter or the crawl trend to check a specific low-volume provider.
  2. If a provider’s search or citation bots never appear, they either can’t reach you or haven’t found you. Check robots.txt for an accidental block of its user-agent tokens (Crawlability lists the bot types), then check your CDN rules. Training-only bots like GPTBot are a separate question.
  3. Cross-reference with your visibility per model: weak visibility on a model whose bots never crawl you is an access problem before it’s a content problem.

​
See which AI platforms send humans, not just bots

Crawl volume is attention from machines; the business case needs humans.
  1. In visitor analytics, read top sources and referrer stats: visits per AI platform.
  2. Compare against crawl share. A provider that crawls heavily but refers few visitors either cites you poorly or cites pages that don’t invite clicks; check those pages’ positioning in actual responses.
  3. Compare landing pages against your cited pages to find citations that never convert to traffic.

​
Prove the channel to stakeholders

For a recurring report, combine three numbers per key page: crawl trend, citations, and AI-referred visits. That chain, machine attention to answer presence to human traffic, is the argument that AI search is a channel and not a curiosity, and Attributing AI traffic to revenue extends it to the revenue step. For turning crawl patterns into content decisions, see Turn crawler logs into opportunities.