How does Alphalens Bot discover pages?

It reads the sitemap when one is available. Otherwise it begins at the homepage and follows links using a breadth-first process.

Does Alphalens Bot follow robots.txt and crawl delays?

Yes. The operator says the crawler checks robots.txt before retrieval, skips blocked URLs, and pauses for a published crawl delay.

Does the crawler interact with forms or bypass bot protection?

No. Alphalens says it retrieves HTML from approved pages without interacting with forms or buttons and without bypassing CAPTCHAs or other security measures.

Is Alphalens Bot used for AI model training?

The operator does not document a model-training purpose. Its stated use is building a detailed company and offerings index for business discovery.

Does allowing Alphalens Bot guarantee search visibility?

No. Access makes the page available for indexing, but Alphalens does not guarantee a ranking, appearance, or referral for any crawled URL.

Alphalens Bot: AI Crawler Guide & robots.txt

What is Alphalens Bot?

Alphalens Bot builds an index of companies and the products or services they offer. Alphalens says it wants business discovery to reflect more than a short company description, so the crawler reads company websites for the detail needed by semantic search. Its job is commercial indexing, not general web archiving.

Before fetching pages, the crawler checks the site's robots.txt file and sitemap. When a sitemap is available, Alphalens begins with its listed URLs and considers the supplied priority values. Without one, it starts at the homepage and follows links in breadth-first order to discover other pages.

The operator describes a restrained retrieval process. Alphalens Bot fetches approved HTML pages but does not submit forms, click buttons, solve CAPTCHAs, or try to pass security controls. It says it honors rate limits, crawl restrictions, and Crawl-delay values published by the site.

A successful crawl can make accurate product and service detail available to Alphalens's business discovery index. Pages that are blocked or absent from navigable links and sitemaps may be missing from that index. The operator does not promise a particular ranking, lead, quotation, or referral from any one crawl.

Alphalens describes this work as indexing for discovery, not collecting a corpus for generative model training. There is no documented training-data purpose for alphalens-bot. Site owners can therefore evaluate it as an AI search crawler without assuming that allowing it grants a separate AI training use.

Every request is documented with the exact user agent alphalens-bot. The operator's site links to public IP information, although this entry is not marked as IP verified and has no request-signing directory. Robots.txt remains the stated control: Alphalens says it checks the file before crawling and skips any URL that the file blocks.

Relevant for AI searchAlphaLens

Is Alphalens Bot relevant for AI search?

Yes. Alphalens Bot feeds AlphaLens, so the pages it can reach shape what those AI products say about you.

Alphalens Bot crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Alphalens Bot

Allow Alphalens Bot on pages that accurately describe your company and offerings if participation in its business index is useful. A complete, current sitemap helps the crawler find those pages. Use path-level disallows for private, administrative, duplicate, or irrelevant sections.

To exclude the entire site, publish this exact group:

User-agent: alphalens-bot
Disallow: /

Alphalens states that it strictly follows robots.txt, skips blocked URLs, and observes crawl delays. If request volume is the concern, set an appropriate crawl delay or contact the operator at the address on its bot page instead of removing useful product pages from the index.

Examples

A software vendor adds its integration catalog to the sitemap so Alphalens can index specific capabilities that a one-line company profile would miss.
A consultancy allows service and case-study pages but disallows client portals and proposal directories for `alphalens-bot`.
A large product site keeps the crawler enabled and publishes a crawl delay to reduce request pressure without leaving the business discovery index.

Frequently asked questions about Alphalens Bot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

It indexes companies and the details of their products and services. Alphalens uses that material for semantic business discovery rather than reducing each company to a short summary.