Definition
User-triggered fetchers are bots that request a web page because a person asked for it, instead of crawling the web on their own schedule. In AI search, examples include ChatGPT-User, Perplexity-User and Google-Agent. They differ from the crawlers that build search indexes or collect training data.
Vendors describe them similarly. OpenAI says ChatGPT-User visits pages for certain user actions in ChatGPT and Custom GPTs, is not used to crawl the web automatically, and that because the user starts the request, robots.txt rules may not apply. Perplexity says Perplexity-User supports user actions and, since a user requested the fetch, generally ignores robots.txt. Google states that its user-triggered fetchers generally ignore robots.txt for the same reason, and lists Google-Agent among them for agents acting on a user's request. Check each vendor's current documentation, as policies change and some vendors may behave differently.
This changes how you manage access. Rules in robots.txt are the standard control for crawlers, but they are not a dependable way to block a user-triggered fetch. To control search appearance, OpenAI points to OAI-SearchBot, and for OpenAI crawlers the three roles (training, search, user actions) are separate. See citation crawlers vs training crawlers for that split, and Anthropic crawlers for how another vendor handles it.
If you want to block them anyway, the tools are server-side: firewall or CDN rules based on user agent and published IP ranges. User agent strings can be spoofed, so verify against the vendor's IP lists or a signature scheme such as Web Bot Auth. Blocking can stop live answers from reading your page, so weigh it against the visibility cost.
These fetchers also matter for measurement. A visit by ChatGPT-User in your logs means someone's question led the assistant to read your page, which is a useful signal that you appear in live answers. Our guides to AI crawler bots and the LLM crawler user agents list the agents to watch, and the AI crawler URL inspector can test whether a page is reachable.
Examples of User-Triggered Fetchers
- A user asks ChatGPT about a product, and ChatGPT-User fetches the vendor's pricing page to read it before answering.
- A site owner disallows a bot in robots.txt, then still sees Perplexity-User requests in server logs because the fetch was user initiated.
- A publisher adds a firewall rule that allows ChatGPT-User only from OpenAI's published IP ranges, to filter spoofed user agents.
- A marketing team counts weekly ChatGPT-User hits on its comparison pages as a sign those pages are being read for live answers.
