Which AI Crawlers Visit the Open Web?

This report shows, week by week, how verified requests from OpenAI, Anthropic, Google, Perplexity, and Mistral are split across the last three months of production crawler logs. See which bots are doing the crawling, how that mix has shifted, and where to look if a provider is missing.

Klaas FoppenSeptember 9, 2026

Share of verified AI crawler requests

Each bar is one week (Monday–Sunday, UTC) and totals 100%. Segments show the share of verified requests per provider, so a growing segment means that provider accounted for a larger slice that week, not necessarily more in absolute terms. Hover a bar for the exact split.

Top provider, August 31–September 6, 2026

OpenAI 79.8%

share of verified requests

Group by

What this means for you

OpenAI accounted for 79.8% of verified AI crawler requests in August 31–September 6, 2026, down from 94.8% in June 8–June 14, 2026.

The biggest mover over the period was OpenAI, which went from 94.8% to 79.8% of the mix. Provider totals hide the split between training crawlers and answer-time fetchers; switch back to Crawler to see which bot inside each provider is doing the work.

How to act on it

Check robots.txt, CDN, and WAF rules for every provider in this chart. A block that felt harmless a year ago now removes you from the providers that dominate the mix.
Compare your own crawler logs against this mix. If a provider is a large share here but near zero in your logs, you are likely blocked or unreachable for that provider.
Switch back to Crawler to see which bot inside OpenAI is doing the work. A large provider share can be training traffic, answer-time fetches, or both.

How to read this data: requests are counted from the raw server and CDN logs connected to Promptwatch crawler analytics. Only requests whose source IP matches the crawler operator's published IP ranges are included, so spoofed user agents are excluded. Weeks are Monday–Sunday in UTC; the current incomplete week is excluded. Shares are across all tracked logs for that week.

How we collect this data

We collect millions of prompt responses, citations, and click data from the actual user interfaces of major AI platforms: over 26 billion data points and growing. This gives us one of the largest datasets on how AI search engines cite sources and recommend brands.

Real UI monitoring

Data straight from the interfaces of ChatGPT, Gemini, Perplexity, Claude, AI Overviews, and more.

26B+ data points

Over 26 billion analyzed citations, prompts, and responses, one of the largest AI search datasets available.

Continuously updated

Refreshed constantly so the trends you see reflect the latest behavior of AI search engines.

Aggregated & public

Published freely for the GEO community, based on aggregated, non-identifiable trends.

Want to start tracking your own AI search data? Get started with Promptwatch

Track AI Crawler Visits to Your Site

See exactly which AI crawlers reach your pages, which ones are blocked, and how your mix compares to your industry. Promptwatch crawler analytics works with Cloudflare, Vercel, Fastly, and raw server logs.