Google-Extended

Google-Extended is a robots.txt user-agent token that lets site owners decide whether content Google crawls from their site may be used to train Gemini models and to ground answers in the Gemini app and Vertex AI. It is a control token, not a separate crawler: Google fetches pages with its usual user agents and applies the Google-Extended rules when deciding how that content may be used.

Google announced the token on September 28, 2023, as a way for publishers to manage whether their sites 'help improve Bard and Vertex AI generative APIs'. Bard has since been renamed Gemini, and the wording has been tightened. Google's crawler documentation now says Google-Extended covers two uses:

• training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini
• grounding, meaning content from the Google Search index supplied to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI

The same page states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. To opt out completely, a site adds a group with User-agent: Google-Extended and Disallow: /. Path rules work the same way as for any other token.

Blocking Google-Extended does not remove a site from AI Overviews or AI Mode. Google has said since 2023 that its AI features in Search follow Search controls: Googlebot rules in robots.txt for crawling, and noindex or snippet controls for what is shown. Since August 31, 2026 there is also the Search generative AI control in Search Console, which excludes a whole site from AI Overviews, AI Mode and generative AI features in Discover. Google's help page for that control says it does not affect AI training and points back to Google-Extended 'to limit training of the models used to generate responses in Search generative AI features'. That extension of the token's scope matches what the UK Competition and Markets Authority asked of Google in its June 2026 publisher conduct requirement.

In short, the two controls split the work. Google-Extended governs training and grounding in Gemini products. The Search Console control governs whether a site appears in, and helps ground, AI features inside Google Search.

For AI visibility, the decision is a trade-off. A site that disallows Google-Extended keeps its future content out of Gemini training data, which shapes the model's parametric knowledge. It also stops its pages from grounding Gemini app answers, so the Gemini app can no longer draw on them when it searches. For a publisher whose business depends on licensing, that may be the point. For a brand that wants Gemini to describe its products accurately, blocking it can mean Gemini relies on third-party pages instead. The distinction between crawlers used for training and those used for answers is covered in citation crawlers vs training crawlers.

Because Google-Extended has no user agent string of its own, it never appears in server logs or AI crawler logs. Auditing it means reading the robots.txt file itself, for example with the robots.txt report in Search Console, and then checking Gemini answers for the prompts that matter before and after a change. Promptwatch keeps a reference page for the Google-Extended token alongside its other crawler profiles.