Web Bot Auth

Web Bot Auth is a protocol, still in draft at the IETF, that lets an automated client prove who operates it by cryptographically signing its HTTP requests. Instead of trusting a self-declared user-agent string or an IP address, a website or CDN checks the signature against a public key the operator publishes. If the signature verifies, the request came from that operator; if not, it did not.

It builds on HTTP Message Signatures (RFC 9421). The operator generates a key pair, typically Ed25519, and publishes the public keys as a JSON Web Key Set at /.well-known/http-message-signatures-directory on its own domain. Each request then carries three headers: Signature-Agent, which points to that directory, Signature-Input, which lists the signed components and includes a tag of web-bot-auth, a key ID, creation and expiry timestamps and usually a nonce, and Signature itself. The verifier fetches and caches the key, checks the signature, and rejects expired messages. Cloudflare's documentation recommends short expiry windows, around a minute, as replay protection.

Status as of October 2026: the IETF chartered the Web Bot Auth working group on October 23, 2025. Its main working group draft, draft-ietf-webbotauth-httpsig-protocol-00, was published on September 1, 2026 by authors from Cloudflare and Google and is intended for the Standards Track, but it has not yet been sent to the IESG, which the charter had planned for April 2026. Related individual drafts cover a bot registry and signature agent cards, anonymous bot authentication and a JSON format for publishing crawler IP ranges. Deployed implementations also trail the drafts: Cloudflare still implements the Signature-Agent format from an earlier directory draft and rejects the dictionary form used in later ones.

Adoption so far is strongest for agents rather than classic crawlers:

• OpenAI signs requests from ChatGPT's cloud browser with Signature-Agent set to https://chatgpt.com, and Cloudflare, Akamai, HUMAN and Vercel verify it automatically.
• Google is testing it experimentally for some agents on its infrastructure and signs only a subset of Google-Agent requests, as https://agent.bot.goog, advising sites to keep IP, reverse DNS and user-agent checks.
• Amazon Bedrock AgentCore Browser offers signing in preview, and AWS WAF Bot Control has verified signatures since November 2025.
• Visa's Trusted Agent Protocol and Mastercard's Agent Pay use it as the authentication layer for shopping agents in agentic commerce.

OpenAI's search and training crawlers, OAI-SearchBot and GPTBot, are still verified through published IP ranges, so AI crawler verification by IP remains necessary for most AI crawlers.

For GEO and analytics teams, the value is trustworthy identity. Verified requests let AI crawler logs separate real agent traffic from impersonators, and they let sites apply policies such as allow, rate limit, pay per crawl or block to the right operator. Web Bot Auth only establishes which company runs a bot; proving that a specific user authorized an agent's purchase is a separate layer handled by protocols such as AP2 and x402. One practical check: make sure proxies and bot rules pass the Signature, Signature-Input and Signature-Agent headers through untouched, or signed agents will fail verification and get challenged like any unknown bot.