AIWebIndex-Agent is the on-demand fetch type in Lyrenth's crawler policy. It runs when an agent requests a page that is not yet indexed or wants a fresh copy, whereas AIWebIndex crawls on a schedule.
Lyrenth says every fetch honors robots.txt, so a disallowed URL is not fetched, not indexed and not served on request. The effect is prospective only: content already in the index remains until you request removal.
Yes. AIWebIndex-Agent fetches pages into Lyrenth's index when an AI agent asks for one that is missing or stale.
When an agent requests a page that Lyrenth has not indexed, or wants a fresh copy, AIWebIndex-Agent fetches it. Lyrenth says it does no training. Blocking it keeps those on-demand fetches from reaching your pages, so agents get older copies or none.
To keep AIWebIndex-Agent out, add a group for its user-agent token:
User-agent: AIWebIndex-Agent
Disallow: /
See the Lyrenth crawler policy for the full set of directives and the removal request process.
Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.
AIWebIndex crawls on a schedule. AIWebIndex-Agent fetches a page when an agent requests it.
Lyrenth's scheduled crawler that indexes the public web and serves pages to AI agents as attributed documents.
Exa's on-demand fetcher that retrieves a page live when a user or an app built on the Exa API asks for it.
Keenable's on-demand fetcher that reads a single page in real time for a user or an agent acting for one.
Parallel's user-initiated fetcher that retrieves pages on behalf of users of its web APIs.
Handles user-initiated requests in ChatGPT, accessing external content to provide real-time information; not used for automated crawling or AI training.
Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.
