Promptwatch Logo

Google-CloudVertexBot

Google-CloudVertexBot is the crawler Google documents for site-owner-requested crawls used to build Vertex AI Agents.
Google-CloudVertexBot
AI Assistant

What is Google-CloudVertexBot?

Google-CloudVertexBot is the crawler Google documents for site-owner-requested crawls used to build Vertex AI Agents. Google lists it among its common crawlers, while Cloudflare classifies it as an AI crawler and identifies Google as the operator in its operator reference.

The request starts with a Vertex AI workflow in which a site owner asks Google to crawl that owner's site. This is narrower than a crawler that discovers arbitrary domains for a public search index. A visit should correspond to content selected for a particular Vertex AI Agent project.

Cloudflare's bot directory records both mobile and desktop Chromium-style User-Agent forms. Each contains the Google-CloudVertexBot substring, and its recorded matching pattern is CloudVertexBot. Chrome version numbers can change, so log rules should match the stable bot token rather than a complete browser string.

Google says preferences for this crawler have no effect on Google Search or any other product. Content it can reach may be available to the Vertex AI Agent being built, but access does not improve organic ranking or make the page visible across public AI search services. Cloudflare describes the purpose as targeted AI training; in this bot that means a site owner supplying its own site for an agent-building task, not documented training of Google's general foundation models.

There is a source conflict about robots.txt. Google's crawler documentation places Google-CloudVertexBot in the common-crawler list, whose automatic crawls always obey robots.txt, and the page metadata records support as true. Cloudflare's bot directory marks followsRobotsTxt as false. A named rule is still the operator-documented control, but sites that require enforcement should confirm the result in logs.

The exact token offers useful classification, not authentication. No Web Bot Auth directory is recorded, and this page does not mark the bot as IP verified. If a request needs privileged access, validate it through the Vertex project or another access mechanism instead of trusting the User-Agent header.

Relevant for AI search

Is Google-CloudVertexBot relevant for AI search?

Yes. Google-CloudVertexBot collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Google-CloudVertexBot crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Google-CloudVertexBot

If you are building a Vertex AI Agent from your website, allow the public documentation or support paths that belong in that agent. Exclude account pages, administration routes, and any content the agent should not ingest. Authentication remains the proper control for private material.

Google publishes a stable robots.txt token. To refuse the crawler across the site, use:

User-agent: Google-CloudVertexBot
Disallow: /

Google documents this as a common crawler that obeys robots.txt, although Cloudflare currently records it as not following robots.txt. Check for follow-up requests after a rule change. Apply a WAF or application rule as well if a robots preference alone is not sufficient.

Do not use this token to manage Google Search or general Gemini model training. Google states that the token affects only owner-requested Vertex AI Agent crawls; Googlebot and Google-Extended are separate controls.

Examples

  • A software company allows `/docs/` while creating a support agent in Vertex AI, but keeps `/account/` disallowed because customer records do not belong in that agent.
  • A site owner who has never requested a Vertex crawl investigates an unexpected Google-CloudVertexBot visit instead of assuming it came from ordinary Google Search.
  • A publisher blocks the token and verifies that organic Google Search traffic is unchanged, as Google says this crawler does not affect Search.

Frequently asked questions about Google-CloudVertexBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Google's official crawler documentation and Cloudflare's bot directory identify Google as the operator. The bot object's operator field itself remains blank.

Related bots

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant

Amazon Q

Amazon Q Business is a generative artificial intelligence (generative AI)-powered assistant that you can tailor to your business needs.

AI Assistant

ChatGPT-Operator

Handles user-initiated requests from ChatGPT operator accessing external content; not used for automated crawling or AI training.

AI Assistant

ChatGPT-User

Handles user-initiated requests in ChatGPT, accessing external content to provide real-time information; not used for automated crawling or AI training.

AI Assistant

Claude-SearchBot

Claude-SearchBot is Anthropic's crawler for improving search results shown to Claude users.

AI Assistant

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard