Does Google-Agent respect robots.txt?

Not always. Because Google-Agent performs user-initiated or real-time fetches, it can bypass robots.txt. Watch your access logs to confirm how it behaves on your site.

Should I block Google-Agent?

It depends on whether you rely on the service behind Google-Agent. If you do, keep it allowed; otherwise you can disallow it in robots.txt.

Can I see when Google-Agent crawls my site?

Yes. Promptwatch classifies Google-Agent from your server and CDN logs in real time, showing which pages it hits, the response codes it gets, and how its crawls relate to AI citations.

Book a Demo

Google-Agent

Google-Agent navigates the web and performs actions upon user request, used by agents hosted on Google infrastructure such as Project Mariner.

GoogleGoogle-Agent

AgentUser Initiated

What is Google-Agent?

Google-Agent is Google's user-initiated fetcher. Google-Agent navigates the web and performs actions upon user request, used by agents hosted on Google infrastructure such as Project Mariner.

Google-Agent fetches pages in real time on behalf of a person using an AI product, so a visit usually means a user is actively trying to reach your content through AI. These user-driven fetches are often the moments that turn into citations and referral traffic, which makes them valuable signals for GEO.

Tracking which AI crawlers and agents reach your site, and what they do once there, is the foundation of generative engine optimization. See our guides to AI crawlers and robots.txt to control automated access and protect your AI search visibility.

Promptwatch tracks Google-Agent in real time from your server and CDN logs, so you can see exactly when it visits, which pages it requests, and how those crawls relate to citations. Learn more in AI crawler logs.

Track Google-Agent with Promptwatch

Promptwatch classifies Google-Agent (Google-Agent) in real time from your server and CDN logs. See exactly when it visits, which pages it requests, the status codes it gets, and how those crawls map to AI citations. Requests are validated against the operator's published IP ranges to filter out spoofed user agents.

How to handle Google-Agent

Whether to allow Google-Agent depends on whether you rely on the service behind it. Use a robots.txt rule to allow or disallow it.

To control Google-Agent, add a rule for its user agent to your robots.txt:

User-agent: Google-Agent
Disallow: /

Because Google-Agent performs user-initiated or real-time fetches, it may not always honor robots.txt, confirm its behavior by watching your access logs.

Examples

A site owner spots Google-Agent in access logs and confirms it against the operator's documented user agent.
An operations team allows Google-Agent so the service it powers keeps working as expected.

Related bots

Google-Extended

Google-Extended is a standalone product token that web publishers can use to manage whether their sites help improve Gemini Apps and Vertex AI generative APIs, including future generations of models...

AI Crawler

Google Image Proxy

Google's image caching proxy service used by Gmail and other Google services to cache and serve images.

Preview

Frequently asked questions about Google-Agent

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Google-Agent is operated by Google. It functions as Google's user-initiated fetcher.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.