Web Bot Auth

Web Bot Auth is an emerging, IETF-track standard for cryptographic bot identity on the web. Instead of asking a site to trust a self-declared user-agent string, it has the bot operator sign each HTTP request with a private key using HTTP Message Signatures. The receiving site validates the signature against a public key published in a registry, so a request claiming to come from a given AI company either verifies or it does not.

The problem it solves is old and widespread. User-agent strings are trivially spoofable, so any traffic analysis based on them alone is unreliable, and reverse-DNS verification—the traditional workaround—is inconsistent across operators and expensive at scale. As agent traffic has grown, so has the incentive to impersonate well-behaved crawlers in order to bypass rate limits or bot rules.

Adoption has been driven by CDNs and large AI operators, with companies including OpenAI, Visa, and Mastercard signing agent traffic, and Cloudflare and Google publishing verification tooling. The protocol establishes provider-level identity—which company operates a bot—and is often paired with a user-level authorization layer such as x402 or AP2, which prove that a specific person authorized the agent to act.

For GEO and analytics teams the practical value is measurement quality. Signed requests make AI crawler log analysis trustworthy: you can separate genuine GPTBot or ClaudeBot activity from impersonators, attribute crawl volume correctly, and apply access policies such as pay per crawl or RSL terms to the bots they were actually written for.