Who operates meta-externalfetcher?

meta-externalfetcher is a user-initiated fetcher; its operator is identified in its user-agent string and public documentation.

Does meta-externalfetcher respect robots.txt?

Not always. Because meta-externalfetcher performs user-initiated or real-time fetches, it can bypass robots.txt. Watch your access logs to confirm how it behaves on your site.

Should I block meta-externalfetcher?

It depends on whether you rely on the service behind meta-externalfetcher. If you do, keep it allowed; otherwise you can disallow it in robots.txt.

meta-externalfetcher - What It Is & How to Handle It

What is meta-externalfetcher?

meta-externalfetcher is a user-initiated fetcher. The Meta-ExternalFetcher crawler performs user-initiated fetches of individual links to support specific product functions. Because the fetch was initiated by a user, this crawler may bypass robots.txt rules.

meta-externalfetcher fetches pages in real time on behalf of a person using an AI product, so a visit usually means a user is actively trying to reach your content through AI. These user-driven fetches are often the moments that turn into citations and referral traffic, which makes them valuable signals for GEO.

Like any automated client, meta-externalfetcher consumes crawl budget and appears in your server and CDN logs. Reviewing those logs alongside your robots.txt rules helps you keep automated traffic deliberate and easy to reason about.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle meta-externalfetcher

Whether to allow meta-externalfetcher depends on whether you rely on the service behind it. Use a robots.txt rule to allow or disallow it.

To control meta-externalfetcher, add a rule for its user agent to your robots.txt:

User-agent: meta-externalfetcher
Disallow: /

Because meta-externalfetcher performs user-initiated or real-time fetches, it may not always honor robots.txt, confirm its behavior by watching your access logs.

Examples

A site owner spots meta-externalfetcher in access logs and confirms it against the operator's documented user agent.
An operations team allows meta-externalfetcher so the service it powers keeps working as expected.

Related bots

Google Read Aloud

Upon user request, Google Read Aloud fetches and reads out web pages using text-to-speech (TTS).

User Initiated

meta-externalfetcher