Does Googlebot respect robots.txt?

Yes. Google classifies it as a common crawler, and automatic common-crawler requests obey robots.txt rules for the relevant host.

How does Googlebot affect AI Overviews and AI Mode?

Those features draw supporting links from pages already indexed and snippet-eligible in Search. Blocking Googlebot can prevent new or changed content from reaching that index.

Does allowing Googlebot also allow Gemini model training?

Google provides Google-Extended as the separate control for certain Gemini training and grounding uses. Googlebot controls Search crawling, so the two policies can be set independently.

Will blocking Googlebot immediately erase an indexed URL?

No. Robots.txt stops crawling, not necessarily the display of a URL Google learned elsewhere. Removal and indexing directives are separate controls.

How can I tell whether a Googlebot request is genuine?

Do not trust the user-agent alone. Verify the source with Google's published crawler IP ranges or a forward-confirmed reverse DNS check.

Googlebot: AI Crawler Guide & robots.txt

What is Googlebot?

Googlebot is the main crawler for Google Search. It fetches pages for Google's search indexes, and rules for the Googlebot token also affect Discover and other Search features. Google publishes smartphone and desktop user agents, both of which contain the same stable token.

The crawler discovers URLs, checks the host's robots.txt, fetches allowed resources, and returns as Google decides content may need refreshing. Requests come from distributed Google infrastructure, so one site can see several source addresses. Googlebot can use ETag or Last-Modified responses when recrawling, which saves sending an unchanged page again.

Google's Search AI features reuse this Search workflow. A page must be indexed and eligible to appear in Google Search with a snippet before it can be shown as a supporting link in AI Overviews or AI Mode. Allowing Googlebot makes that possible, but it does not guarantee ranking or citation.

Googlebot access and Gemini training controls are not the same decision. Google-Extended is the separate robots token for managing certain uses of crawled content in future Gemini training and for grounding Gemini from the Search index. Blocking Google-Extended does not remove a site from Search, while blocking Googlebot can prevent fresh content from being crawled for both standard and AI Search features.

Relevant for AI searchGoogle SearchAI OverviewsGemini

Is Googlebot relevant for AI search?

Yes. Googlebot feeds Google Search, AI Overviews, Gemini, so the pages it can reach shape what those AI products say about you.

Googlebot feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle Googlebot

Keep public pages crawlable if they should appear in Google Search. Use path rules for faceted or otherwise duplicate URLs that Google does not need. Normal cache validators can reduce repeat transfer costs.

To stop Googlebot across the host, use its documented token:

User-agent: Googlebot
Disallow: /

Googlebot obeys robots.txt for automatic crawls. A crawl block is not a reliable removal instruction for URLs Google already knows, since an uncrawled URL can still be referenced from external links. Use the appropriate Search removal or indexing control for that separate goal. Snippet controls such as nosnippet and max-snippet also affect what Search can show in its AI features.

Examples

A documentation team submits a sitemap after a release, then confirms that Googlebot revisited the changed URLs before checking their Search status.
A publisher allows Googlebot for Search eligibility but adds a separate Google-Extended rule because its policy for Gemini-related reuse is different.

Frequently asked questions about Googlebot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Google says the rules affect Google Search, including Discover and Search features, along with products that rely on its crawling such as Google Images, Video, and News.