How can I verify ExaSearchBot is genuine?

Requests are signed with Web Bot Auth. Validate the signature against the key directory at `https://crawler.exa.ai/.well-known/http-message-signatures-directory`.

How do I remove a page from Exa's index?

Use a `noindex` robots meta tag or an `X-Robots-Tag: noindex` header. Exa drops the page after its next re-fetch.

What is the difference between ExaSearchBot and ExaUser?

ExaSearchBot crawls on its own to build the index. ExaUser fetches a page live when a user or an app asks for it.

ExaSearchBot: AI Crawler Guide & robots.txt

What is ExaSearchBot?

ExaSearchBot is the crawler behind Exa's index. The Exa crawler page says its purpose is search and retrieval: connecting people and applications to your content and sending them back to your site. Exa documents the user agent as Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/), with ExaSearchBot as the robots.txt product token.

Every request is signed with HTTP Message Signatures (RFC 9421) following the Web Bot Auth scheme, with Signature, Signature-Input and Signature-Agent headers. Cloudflare, Akamai and other providers that support Web Bot Auth validate it automatically.

ExaSearchBot respects the Robots Exclusion Protocol. It looks for a group addressed to ExaSearchBot first, then follows the rules set for major search engines, and otherwise the * group. Exa also says it limits request rates per site and does not try to bypass logins, paywalls or CAPTCHAs.

Pages fetched live for a user or an app go out as ExaUser instead. robots.txt controls crawling, not indexing: to remove a page already in the index, serve a noindex meta tag or X-Robots-Tag: noindex header and let ExaSearchBot re-fetch it.

Relevant for AI searchExa

Is ExaSearchBot relevant for AI search?

Yes. ExaSearchBot builds the index behind Exa's search API, which AI agents and apps use to find and cite web pages.

Exa is a search engine built for AI applications. ExaSearchBot discovers and indexes public pages so users and apps can find, retrieve and cite them. A page it has never fetched cannot be returned by Exa, so blocking it removes you from every agent that searches through Exa.

How to handle ExaSearchBot

Keep ExaSearchBot allowed if you want AI agents that search through Exa to find and cite your pages. To block it entirely:

User-agent: ExaSearchBot
Disallow: /

To keep it out of one area, give it its own group, for example Disallow: /private/. A group that names ExaSearchBot takes precedence over your rules for other search engines. Verify genuine traffic with the Web Bot Auth signature rather than the user-agent string.

Examples

After publishing a comparison page, a marketer checks that a signed ExaSearchBot request fetched it, since only indexed pages can be returned by Exa.
A site owner disallows `/private/` for ExaSearchBot and adds a `noindex` header to remove an old page from the index.

Frequently asked questions about ExaSearchBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Yes. Exa says it follows the Robots Exclusion Protocol and uses the group addressed to `ExaSearchBot` when one exists.

Related bots

ExaUser

Exa's on-demand fetcher that retrieves a page live when a user or an app built on the Exa API asks for it.

AI Assistant

ShapBot

Crawls and indexes web content to power Parallel's search and content extraction APIs for AI applications.

AI Crawler

KeenableBot

Keenable's crawler for public pages in its search index, which answers queries from AI agents with linked sources.

AI Search

OctenSearchBot

Octen's search crawler, which indexes the web for the search service AI applications use.