Who operates google-xrawler?

google-xrawler is operated by Google. It functions as Google's search engine crawler.

Does google-xrawler respect robots.txt?

Google has not published a robots.txt commitment for google-xrawler. Treat its behavior as unconfirmed and watch your access logs to see whether your rules are honored.

Should I block google-xrawler?

Usually no. Blocking google-xrawler removes your pages from the search results it powers, and increasingly from the AI answers built on that index. Only block it to deliberately opt out.

google-xrawler - What It Is & How to Handle It

What is google-xrawler?

google-xrawler is Google's search engine crawler. Google's user-triggered fetcher for merchant product feeds. Fetches XML product feeds from e-commerce sites for syncing with Google Merchant Center.

google-xrawler powers a search engine, and search indexes increasingly feed AI answers and overviews alongside classic results. The pages it can crawl are the pages eligible to rank and to be pulled into AI-generated summaries, so allowing it protects both your SEO and your AI search visibility.

Like any automated client, google-xrawler consumes crawl budget and appears in your server and CDN logs. Reviewing those logs alongside your robots.txt rules helps you keep automated traffic deliberate and easy to reason about.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle google-xrawler

Most sites keep google-xrawler allowed, because blocking it can remove your pages from the search results and AI answers it powers. Only disallow it if you specifically want to opt out of that surface.

To control google-xrawler, add a rule for its user agent to your robots.txt:

User-agent: google-xrawler
Disallow: /

Google does not publish a robots.txt commitment for google-xrawler, so confirm how it behaves by watching your access logs.

Examples

After publishing a new comparison page, a marketer checks whether google-xrawler has fetched it so the content can appear in search results.
An SEO lead keeps google-xrawler allowed in robots.txt to preserve visibility in search and AI-powered answers.

Related bots

Google-Agent

Google-Agent navigates the web and performs actions upon user request, used by agents hosted on Google infrastructure such as Project Mariner.

Agent

Google-Extended

Google-Extended is a standalone product token that web publishers can use to manage whether their sites help improve Gemini Apps and Vertex AI generative APIs, including future generations of models...

AI Crawler

Google Image Proxy

Google's image caching proxy service used by Gmail and other Google services to cache and serve images.

Preview

Google NotebookLM

Google NotebookLM fetches web sources that a user adds to a notebook so the assistant can summarize, answer questions, and cite them. Because fetches are user-initiated, it may bypass robots.txt.

AI Assistant

Google-AdWords-Express

Google-AdWords-Express is a bot for a Google Ads product aimed at small businesses. It crawls advertiser websites to assist with ad creation and to verify site information.

SEO

google-xrawler