Does BraveBot respect robots.txt?

Yes, BraveBot is classified as respecting crawl rules. Brave's own documentation says its crawler will not visit a page that Googlebot cannot crawl, so Googlebot directives are also relevant to Brave's behavior.

Will a robots.txt block remove a page from Brave Search?

No. Brave says robots.txt prevents fetching but does not itself prevent indexing. Use a `noindex` directive for delisting and allow Brave to refetch the page so it can process that change.

Does BraveBot collect training data?

The documented purpose is to build the Brave Search index, which also grounds Leo answers. The available operator details do not say that BraveBot's crawl is used to train a general AI model.

Can the BraveBot user agent be authenticated?

Not from the header alone. Cloudflare's bot directory records a `Bravebot/1.0` string, but Brave says its crawler does not consistently advertise a differentiated user agent. No Web Bot Auth key directory or verified IP identity is listed here.

BraveBot: AI Crawler Guide & robots.txt

What is BraveBot?

BraveBot crawls pages for the Brave Search index. That index supplies conventional search results and can ground answers from Brave's Leo assistant, so this is both a search crawler and part of an AI answer workflow. It is not a user agent that appears only after one person asks Leo to open a particular URL.

Robots.txt controls whether Brave fetches a page, and BraveBot is classified as respecting crawl directives. Brave also warns that blocking a crawl is not the same as removing a URL from its index. For delisting, it directs site owners to use a noindex directive and let Brave fetch the updated page.

Brave describes its crawler as a discovery system for finding new pages and indexing their content. The company also says the crawl is partially informed by its Web Discovery Project, an optional feature in the Brave browser. A visit may therefore be part of initial discovery, a refresh of known content, or another indexing pass.

Brave's current crawler documentation has an unusual identity policy. It says the crawler does not advertise a differentiated user agent and will not crawl a page that Googlebot cannot crawl. The catalog nevertheless has a stable BraveBot token, while Cloudflare's bot directory records Bravebot/1.0 inside a full Chrome-style user agent. Cloudflare's bot directory marks that recorded identity as not following robots.txt, so this page's positive classification relies on Brave's own stated Googlebot boundary. A rule for the token is useful for traffic that presents it, but it may not describe every Brave Search request.

Pages that Brave can fetch are candidates for Brave Search and for grounded Leo responses. Blocking crawl access can limit updates and reduce the chance that current material is available on those surfaces, but a successful fetch does not guarantee a ranking, quotation, or citation. There is no evidence in the operator record that BraveBot collects this content for general model training.

Cloudflare's recorded user agent is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36. No Web Bot Auth directory or verified IP identity is listed for BraveBot. Combined with Brave's nondifferentiated user-agent policy, the header should be treated as a classification clue rather than conclusive authentication.

Relevant for AI searchBraveBrave Leo

Is BraveBot relevant for AI search?

Yes. BraveBot feeds Brave, Brave Leo, so the pages it can reach shape what those AI products say about you.

BraveBot crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle BraveBot

Keep pages crawlable if you want their current content represented in Brave Search and available to Leo's retrieval layer. Brave says its crawler uses Googlebot crawlability as a boundary, so check the Googlebot rules as well as any Brave-specific entry.

The stable catalog token can be addressed with:

User-agent: BraveBot
Disallow: /

This rule stops matching crawl requests, but it is not a delisting mechanism. Add noindex when the goal is removal from Brave Search, then allow a refetch so Brave can see the directive. Because Brave says it may not identify every request with a distinct token, do not use the BraveBot header as the sole basis for a security decision.

Examples

A news site allows crawling of article pages so Brave Search can refresh them, while excluding an internal preview path.
A publisher that wants an old URL removed adds `noindex` and permits Brave to fetch the change instead of blocking the URL first.
An analyst sees `Bravebot/1.0` in logs but avoids calling it authenticated traffic because the request has no signature or verified IP match.

Frequently asked questions about BraveBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

BraveBot gathers pages for Brave Search. The same search index can ground responses from Brave's Leo AI assistant, which gives the crawler relevance to both search listings and AI answers.