Crawling and Indexing

Crawling and indexing are the two steps that make a web page findable. Crawling is discovery and fetching: automated bots request URLs they learn about from links, sitemaps and earlier crawls, and download the content. Indexing is processing and storage: the system parses the fetched page, works out what it is about, decides whether it is worth keeping, and stores it so it can be retrieved for a query. A page can be crawled and not indexed, and a page that is never crawled cannot be indexed at all.

In classic search, the cycle is well documented. Googlebot and Bingbot follow links and XML sitemaps, respect robots.txt, render pages (Google runs JavaScript in a second pass), and then index the content, consolidating duplicates under a canonical URL. Search Console's Page Indexing report lists why pages are excluded, for example 'Crawled, currently not indexed', noindex tags, soft 404s or duplicates. On large sites, how much gets crawled is limited by crawl budget.

AI systems crawl for three different purposes, and each kind of bot needs a separate decision:

• Training crawlers such as GPTBot and ClaudeBot collect content that may be used to train future models. Google uses a robots.txt token, Google-Extended, rather than a separate bot to let sites opt out of Gemini training.
• Search index crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes those assistants search when they answer with web results.
• Answer-time fetchers such as ChatGPT-User, Claude-User and Perplexity-User load a specific page when a user's question needs it.

Blocking a training crawler opts future content out of training. Blocking a search crawler or a fetcher removes your pages from that assistant's cited answers, which is rarely what a brand wants. Some assistants also rely on outside indexes: Google's AI Overviews and AI Mode use Google's own index, Microsoft Copilot uses Bing (see Bing index), and TechCrunch reported in March 2025 that Claude's web search appeared to use Brave Search.

Indexing for AI looks different from a classic search index. Retrieval systems often split pages into passages and store them as embeddings for semantic matching, then pull individual passages into a model's context (retrieval-augmented generation). Clear headings, self-contained sections and facts in plain HTML text make those passages easier to match and quote. Most AI crawlers fetch raw HTML without executing JavaScript, so content that only appears after client-side rendering may never be indexed by them.

A common misconception is that llms.txt works like robots.txt for AI. It does not. llms.txt is a proposed Markdown file listing a site's key pages for language models; it grants or blocks nothing, and Google's AI features documentation says sites do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. To be eligible there, a page must simply be indexed and eligible for a snippet. Our analysis found llms.txt had no measurable impact on AI search.

To check your own site, confirm robots.txt and CDN rules allow the search and answer-time bots you want, then read server logs per bot for 403, 404 and 5xx responses (AI crawler logs). Verify bot identity against published IP ranges where available, since user-agents can be spoofed (AI crawler verification). Use Search Console and Bing Webmaster Tools for indexing status in those engines, and IndexNow to notify Bing of changed URLs. To see what an AI crawler extracts from a single page, run it through the AI crawler URL inspector.