Algolia

Algolia Crawler is configured by an Algolia customer to collect content from domains that the customer has verified. Configured actions turn selected page content into records in that customer's Algolia index. The resulting index usually supports search within the customer's own website or application.

This is not automatic inclusion in a public web search engine. If the customer later uses the Algolia index as retrieval material for an AI answer feature, crawled records can be part of that private workflow. The crawl alone does not expose a page to unrelated AI services, and Algolia does not describe it as general model-training collection.

The official request identity is Algolia Crawler/xx.xx.xx, with a changing version suffix. This catalog stores the shorter token Algolia, while Algolia's robots examples use the full stable product name Algolia Crawler. A second observed identity, Algolia Crawler Renderscript, is associated with rendering work.

Algolia respects robots.txt by default, which agrees with this bot's metadata. Compliance is configurable, however: a crawler owner can set ignoreRobotsTxtRules to true. The matching Cloudflare record also marks robots support as false, so publishers should not treat the directory as an unconditional access guarantee.