Does google-xrawler crawl the whole storefront?

That is not its documented purpose. The supplied record describes a feed fetcher, so expected traffic should center on configured XML product-feed URLs.

Does google-xrawler respect robots.txt?

The metadata leaves the answer unknown, and Cloudflare's bot directory marks it false. Google also says user-triggered fetchers generally ignore robots.txt, so use an enforced rule when the feed must be protected.

Will blocking google-xrawler remove ordinary pages from Google Search?

The supplied evidence only connects this identity to Merchant Center feed retrieval. It does not establish that `google-xrawler` maintains Google's general page index.

Is google-xrawler an AI training crawler?

No AI training use is documented for this fetcher. Its stated function is moving merchant product-feed data into Google Merchant Center.

How should I verify a google-xrawler request?

Check the exact token and expected feed path, then apply Google's reverse DNS or published IP-range verification. this bot provides no HTTP signature URL and is not marked as IP verified.

google-xrawler: AI Crawler Guide & robots.txt

What is google-xrawler?

google-xrawler fetches XML product feeds from ecommerce sites for synchronization with Google Merchant Center. The supplied facts classify it as user triggered. Its job begins with a merchant feed workflow, rather than discovering arbitrary product pages by following links across a storefront.

A normal request should point to a feed URL that a merchant expects Google to retrieve. The response can contain the product data being synchronized into Merchant Center. Changes to that URL, its authentication, or its response status can interrupt the feed even when the storefront itself remains available.

This distinction helps when reading logs. A request for a known XML source fits the recorded purpose. A long sequence across category pages, checkout, or customer accounts does not look like the narrow feed fetch described here and should not be attributed to Merchant Center merely because the header says google-xrawler.

The fetcher is relevant to ecommerce data delivery, not general web indexing. The supplied record does not connect it to AI search answers, Gemini, or model training. Successful retrieval means Google Merchant Center can receive the feed; it does not prove that a product appeared in a particular Google surface or generated answer.

Cloudflare links this entry to Google's general crawler overview, records the exact user agent google-xrawler, and marks followsRobotsTxt as false. The Promptwatch metadata leaves compliance unknown. Google's broader documentation says user-triggered fetchers generally ignore robots.txt, so a robots rule should not be the only protection for a feed.

No Web Bot Auth signature directory is recorded, and this entry is not marked as IP verified. Google documents DNS and IP-list checks for its crawler classes, which can provide stronger evidence than the header alone. Also compare the requested URL with the feed configured for the merchant account before allowing access to a protected resource.

Relevant for AI search

Is google-xrawler relevant for AI search?

Yes. google-xrawler collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

google-xrawler feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle google-xrawler

If Merchant Center is supposed to fetch the feed, keep that specific URL stable and return the intended XML response. Do not grant the fetcher broader access to an administration area just to make one feed work. If the feed is obsolete, remove or update the corresponding Merchant Center source and retire the endpoint.

The known token permits this declaration:

User-agent: google-xrawler
Disallow: /

Cloudflare says the fetcher does not follow robots.txt, and the catalog does not assert that it does. Use authentication, an unguessable feed URL where appropriate, or a narrowly scoped edge rule for actual access control. Verify Google origin with the operator's published process before IP allowlisting.

Examples

A merchant changes its product-feed URL and confirms that `google-xrawler` begins requesting the new XML location instead of the retired path.
An ecommerce platform permits the fetcher on one feed route without exposing the dashboard used to generate that feed.
A log analyst separates an expected feed retrieval from a browser-like crawl that happens to copy the same user-agent string.

Frequently asked questions about google-xrawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Google uses it to fetch XML product feeds from ecommerce sites for synchronization with Google Merchant Center.

Related bots

Google-Agent

Google-Agent navigates the web and performs actions upon user request, used by agents hosted on Google infrastructure such as Project Mariner.

Agent

Google-Extended

Google-Extended is a robots. txt product token, not a separate crawler that identifies itself in HTTP requests.

AI Crawler

Google Image Proxy

Google's image caching proxy service used by Gmail and other Google services to cache and serve images.

Preview

Google NotebookLM

Google NotebookLM fetches a web page when a person adds that URL as a notebook source.

AI Assistant

Google-AdWords-Express

Google-AdWords-Express is associated with a Google Ads product for small businesses.

SEO

google-xrawler