Does Google Site Verifier respect robots.txt?

Google lists it as a user-triggered fetcher and says fetchers in that group generally ignore robots.txt. Use an origin, CDN, or firewall rule if you need to stop the request.

How can I confirm that the request came from Google?

Check the source against Google's published user-triggered fetcher IP ranges, or perform the forward and reverse DNS procedure Google documents. The user agent alone can be spoofed.

Why does the verifier return after ownership was approved?

Search Console periodically checks some verification tokens. Keep the file or meta tag in place for as long as the verified owner should retain access.

What happens if I block Google Site Verifier?

A new HTML file or tag verification can fail, and a later token check may no longer confirm the associated owner. Blocking it does not affect ordinary Googlebot crawling.

Does Google Site Verifier affect AI search or model training?

No. It reads an ownership token for Search Console. It is separate from Google search crawlers and from clients that retrieve content for AI products.

Google Site Verifier Bot Guide

What is Google Site Verifier?

Google Site Verifier is a user-triggered fetcher used for Search Console ownership checks. When someone chooses HTML file verification, Google requests the unique file placed at the site's root. With HTML tag verification, it fetches the public homepage and looks for the account-specific meta tag. Search Console can check the token again later, so the fetcher may return after the first successful verification.

Its documented user agent is Mozilla/5.0 (compatible; Google-Site-Verification/1.0). A user agent can be copied, so it is not proof that a request came from Google. Google publishes IP ranges for user-triggered fetchers and documents forward and reverse DNS checks for validating individual requests.

Google says user-triggered fetchers generally ignore robots.txt. A robots rule is therefore not a dependable way to stop this verification request. Server, CDN, or firewall rules can reject it, but doing so can prevent a requested ownership check from completing. Removing a verification file or tag can also cause the associated owner to lose verified status.

This fetcher checks for an ownership token. Google uses separate crawlers for search indexing, and Site Verifier supplies no content to AI search or model training systems. Its presence in access logs records a Search Console verification action, not an AI visibility signal.

Not relevant for AI search

Is Google Site Verifier relevant for AI search?

No. Google Site Verifier is not part of AI search or training, so allowing or blocking it does not change your AI visibility.

Google Site Verifier fetches specific files to verify something you configured, such as domain control. These are small, targeted requests that have no role in search rankings or AI answers.

How to handle Google Site Verifier

Allow the requested verification file or public homepage while you are adding or retaining a verified Search Console owner. Make sure redirects, authentication, and bot challenges do not replace the token-bearing response. If a WAF needs an exception, scope it to the verification path and validate the source against Google's published user-triggered fetcher ranges rather than trusting the user agent alone.

For an unexpected request, check the source IP or forward and reverse DNS before treating it as Google traffic. You can block the request at the origin, CDN, or firewall when no ownership check should be running. Do not rely on robots.txt for this decision because Google classifies Site Verifier as a user-triggered fetcher.

Examples

A developer uploads the Search Console HTML verification file, clicks Verify, and sees Google Site Verifier request that exact root-level path.
Search Console revisits a homepage to confirm that an existing owner's verification meta tag is still present.
A security team finds the documented user agent coming from an address outside Google's published fetcher ranges and treats it as spoofed.

Frequently asked questions about Google Site Verifier

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

For web-based Search Console verification, it requests either the unique HTML file placed at the site's root or the public homepage containing the verification meta tag.

Related bots

Google Association Service

Verifies associations between apps and websites for Digital Asset Links.

Verification

Google Business Link Verification

Verifies that business links in Google Business Profile are accessible and return valid HTTP status codes.