Does meta-externalfetcher respect robots.txt?

Only partially. Meta says it may bypass robots.txt because the request was initiated by a user, so a disallow rule expresses a preference but is not reliable access control.

What Meta features use this fetcher?

Meta describes product functions that work with user-supplied links. Its documentation specifically mentions evaluating and improving agentic AI capabilities and helping AI navigate sites to complete user tasks.

Does one request mean Meta is training on my site?

No. The documentation does not say that each fetched page enters model training. The supported conclusion from a log entry is that a Meta product requested that individual URL for a user-driven function.

Can I use robots.txt to protect account pages?

No. Protect account pages with authentication and authorization. Robots.txt is public guidance for crawlers and this fetcher may bypass it.

What breaks if I block meta-externalfetcher?

A Meta product may be unable to use the submitted link for the requested task. Blocking this token does not, by itself, document any change to unrelated Meta crawlers.

meta-externalfetcher AI Crawler Guide

What is meta-externalfetcher?

meta-externalfetcher retrieves an individual link after a Meta user asks a product feature to work with that URL. It is not described as an autonomous crawler that follows links across a domain or builds a standing copy of the open web.

Meta says these fetches can support product functions that evaluate and improve agentic AI capabilities, including helping an AI navigate a website to complete a task for the user. A request should still be interpreted narrowly: it shows that one URL was needed for a user-initiated function, not that Meta began a site-wide crawl.

The recorded user agent is meta-externalfetcher/1.1, sometimes followed by Meta's crawler documentation URL. Since the fetch exists to carry out a user's request, Meta says it may bypass robots.txt. Authentication and application authorization remain the effective boundaries for nonpublic actions or content.

This bot is connected to Meta product features that can involve agentic AI, but a log entry is not proof of an AI search citation or model training. Meta's crawler documentation does not say that every fetched page enters a search index or training corpus. Measure the event as a requested page access unless another source establishes a broader use.

Relevant for AI searchMeta AI

Is meta-externalfetcher relevant for AI search?

Yes. meta-externalfetcher feeds Meta AI, so the pages it can reach shape what those AI products say about you.

meta-externalfetcher fetches, indexes, or retrieves pages for an AI product, so the pages it can reach shape how that system describes your brand and products.

How to handle meta-externalfetcher

Allow meta-externalfetcher on public pages if Meta users should be able to use those links in supported product tasks. Keep authenticated operations protected with normal user authorization, since a crawler rule is not a substitute for permission checks.

You can publish a preference for the exact token:

User-agent: meta-externalfetcher
Disallow: /

Meta explicitly says this user-initiated fetcher may bypass robots.txt, so the stanza cannot guarantee denial. Use a server or edge rule for a hard block, knowing that the related Meta feature may then fail for a user who supplied that link.

Examples

A Meta user gives an agent one product page while completing a task. `meta-externalfetcher/1.1` requests that page, but the event does not imply that adjacent catalog URLs will be crawled.
A private dashboard appears in a user-supplied link. The application requires a valid session and returns an unauthorized response to the fetcher, preserving the same access boundary used for other clients.

Frequently asked questions about meta-externalfetcher

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Meta describes it as a fetcher for individual links supplied at a user's request. The documented workflow does not authorize a site-wide crawl.

Related bots

Google Read Aloud

Upon user request, Google Read Aloud fetches and reads out web pages using text-to-speech (TTS).

User Initiated

meta-externalfetcher