Live session Nov 5: Fireflies' AI Search Playbook
Promptwatch Logo

AIWebIndex

Lyrenth's scheduled crawler that indexes the public web and serves pages to AI agents as attributed documents.
Lyrenth
AI SearchAI Crawler

What is AIWebIndex?

AIWebIndex is the scheduled crawler for Lyrenth. The Lyrenth crawler policy describes two fetch types: scheduled crawling by AIWebIndex, and on-demand fetches by AIWebIndex-Agent. Lyrenth says both honor your robots.txt.

Cloudflare lists no user-agent pattern for AIWebIndex. It signs requests with Web Bot Auth and publishes keys at https://api.lyrenth.com/.well-known/http-message-signatures-directory. Lyrenth also supports ownership verification through a DNS TXT record on your domain.

A robots.txt disallow stops future scheduled and on-demand fetching of those paths. Content already in the index stays until you also request removal.

Relevant for AI searchLyrenth

Is AIWebIndex relevant for AI search?

Yes. AIWebIndex builds Lyrenth's index of the web, which it serves to AI agents as clean, attributed documents.

Lyrenth indexes the public web and serves pages to AI agents with a link back to the source. Lyrenth says it does no model training. If AIWebIndex cannot fetch a page, that page will not be served to agents that query Lyrenth.

How to handle AIWebIndex

Because AIWebIndex has no stable user-agent token on record, check Lyrenth's crawler policy for the current robots.txt group name before writing a rule. A disallow rule stops future fetching of the paths, but already-indexed content remains until you ask Lyrenth to remove it.

Verify genuine traffic with the Web Bot Auth signature at your CDN or edge.

Examples

  • A documentation site checks signed AIWebIndex requests to confirm new pages reach Lyrenth's index.
  • A publisher disallows a path in robots.txt and then asks Lyrenth to remove content that was already indexed.

Frequently asked questions about AIWebIndex

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Yes. Lyrenth says every fetch it makes honors your robots.txt, scheduled or on demand.

Related bots

AIWebIndex-Agent

Lyrenth's on-demand fetcher that pulls a page into its index when an agent requests one or wants a fresh copy.

AI Assistant

ExaSearchBot

Exa's search crawler, which discovers and indexes public web pages for its search and retrieval API.

AI Search

KeenableBot

Keenable's crawler for public pages in its search index, which answers queries from AI agents with linked sources.

AI Search

OctenSearchBot

Octen's search crawler, which indexes the web for the search service AI applications use.

AI Search

ShapBot

Crawls and indexes web content to power Parallel's search and content extraction APIs for AI applications.

AI Crawler

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard