AI Search Index

An AI search index is the collection of crawled and processed web pages that an AI assistant searches when it needs fresh information to answer a question. It works like a classic search engine index, but it is tuned to return passages a language model can read and cite rather than a page of ten links. Which index a platform uses largely decides which of your pages it can find.

The major platforms do not share one index:

• Google's AI Overviews and AI Mode use Google Search's index, crawled by Googlebot. Google says a page must be indexed and eligible for a snippet to appear as a supporting link.
• ChatGPT search uses OpenAI's own crawler, OAI-SearchBot, along with third-party search providers and partner content. OpenAI's crawler documentation says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• Perplexity runs its own index. When it opened its Search API in September 2025 it said the index tracks more than 200 billion unique URLs and processes tens of thousands of index updates per second.
• Microsoft Copilot builds on the Bing index.
• Meta crawls with Meta-WebIndexer to supply Meta AI's answers. In Promptwatch's crawler data, Meta-WebIndexer went from about 2% of tracked AI crawler requests in mid-July 2026 to 37.8% on August 9, 2026, consistent with reports since October 2024 that Meta is building its own web search index (Meta-WebIndexer report).

These indexes are separate from training data. A citation crawler such as OAI-SearchBot or PerplexityBot fetches pages so they can be retrieved and cited at answer time. A training crawler such as GPTBot or Google-Extended collects content for future model training. Most providers let you control them separately in robots.txt, so a site can stay in an AI search index while opting out of training.

For GEO and SEO teams, multiple indexes mean more places where a page can be missing. A product page that Google indexes may be absent from Perplexity's index, blocked for OAI-SearchBot by a CDN firewall rule, or never crawled by Meta-WebIndexer. The fix starts with access: check AI crawler logs to see which bots fetch which pages and with what status codes, confirm that robots.txt and bot-management settings allow the citation crawlers you want, and use AI crawler verification against each provider's published IP ranges, since user-agent strings are easy to fake.

After access, freshness. Assistants pull from their index at answer time, so an outdated price or spec on a crawled page can be repeated in answers until the crawler returns. Server-rendered HTML, accurate sitemaps with lastmod dates and IndexNow for Bing-based systems help new content get picked up faster. Our OAI-SearchBot page lists the user agent, IP ranges and robots.txt rules for OpenAI's citation crawler.