Definition
AI crawler verification is the process of confirming that a request claiming to come from an AI crawler, such as GPTBot or PerplexityBot, really came from that provider. The user-agent string is only a claim that anyone can type. Verification checks the request against evidence the provider controls.
Three methods are in use as of October 2026:
- Matching the source IP against the provider's published list. OpenAI publishes separate JSON files for GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. Perplexity publishes files for PerplexityBot and Perplexity-User. Google publishes files for its common crawlers, special-case crawlers, user-triggered fetchers and user-triggered agents such as Google-Agent. Microsoft publishes bingbot.json and Apple publishes Applebot ranges. Anthropic, which previously published no ranges, now links a single list from its crawler help page (the current file is dated October 7, 2026). It covers ClaudeBot, Claude-SearchBot and Claude-User, and proves a request came from Anthropic but not which of its bots sent it.
- Running a reverse DNS lookup on the IP, checking that the hostname ends in the provider's domain (googlebot.com or google.com for Google, search.msn.com for Bing, applebot.apple.com for Apple), then running a forward lookup on that hostname to confirm it resolves back to the same IP. Google documents this method step by step.
- Checking a cryptographic signature. With Web Bot Auth, the agent signs each request using HTTP Message Signatures (RFC 9421) and publishes its public keys. ChatGPT Work's cloud browser signs its requests with a Signature-Agent header of https://chatgpt.com, and Google says it is experimenting with Web Bot Auth for Google-Agent.
Verification matters because spoofing is routine. Scrapers and SEO tools often send a GPTBot or ClaudeBot user agent to slip past rules that allow AI bots. That inflates crawler numbers and, worse, turns allow rules into open doors. Checking identity protects both your analytics and your access policy. Perplexity's own firewall guidance tells site owners to combine user-agent matching with its published IP ranges rather than relying on either alone. Crawlers whose operators publish no list can only be matched on user agent, so their counts deserve more caution.
The lists change, so fetch them automatically. Bing tells site owners to refresh its list daily, and Perplexity recommends automated updates. Map lists per bot rather than per company: an OpenAI IP proves OpenAI, but only the ChatGPT-User file proves ChatGPT-User. Anthropic also warns that blocking its IP addresses is not a reliable opt-out, because it stops the bot from reading robots.txt. Use robots.txt for access preferences and IP or signature checks for identity.
Promptwatch checks crawler visits against published ranges and leaves unverified requests out of its charts by default, so its counts can be lower than a raw user-agent search of your logs, and more accurate; see our verified AI crawler data announcement. Verification affects measurement and access control only. It does not change whether a page gets cited. For what each bot does once identified, see AI crawlers and AI crawler logs.
Examples of AI Crawler Verification
- A site's logs show a burst of requests labeled GPTBot, but about a third come from IPs outside openai.com/gptbot.json and trace to a cloud host used by a scraper, so those requests are excluded from crawler reports.
- A security team replaces a Cloudflare rule that allowed any request containing 'PerplexityBot' with one that also requires an IP from Perplexity's published list.
- An engineer checks a spike of Googlebot traffic with a reverse DNS lookup, gets a hostname on googlebot.com, and confirms it with a forward lookup back to the same IP.
- A retailer hosted on Vercel lets signed ChatGPT Work cloud browser sessions through its bot protection, because Vercel verifies the Web Bot Auth signature automatically.
