> ## Documentation Index > Fetch the complete documentation index at: https://promptwatch.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Crawlability: can AI bots read your site? > Crawlability is whether AI crawlers can fetch and read your pages, and Promptwatch surfaces blocks in Crawler Logs alongside what the latest page crawl returned. ### What is crawlability? Crawlability is whether AI crawlers can fetch your pages and read what's on them. It sits at the very start of the chain that ends in visibility: a page that OAI-SearchBot or PerplexityBot can't fetch can't enter that model's search index, and if ChatGPT-User or Perplexity-User also can't fetch it at answer time, it is much less likely to be cited. Everything else you do for AI visibility assumes this step works, which is why it's worth verifying rather than assuming. Crawlability fails quietly. Your site looks fine in a browser, human traffic is normal, and meanwhile a bot-protection rule has been returning 403 to ClaudeBot for a month. The only place that failure is visible is in your server logs, which is what [AI crawler insights](/docs/academy/ai-crawler-insights) surfaces. ### Not all AI bots do the same job AI providers run separate crawlers for separate purposes, and blocking each has a different consequence. The main types, using OpenAI's bots as the example: * **Training crawlers** (GPTBot, ClaudeBot): collect content for training foundation models. Blocking them opts your content out of future training data, and only that. Google-Extended is a robots.txt token rather than a bot (Google still crawls with its usual bots); disallowing it opts you out of Gemini training and Gemini Apps grounding, not Google Search. * **Search index crawlers** (OAI-SearchBot, Claude-SearchBot, PerplexityBot): index your site so it can appear in that model's search features. Blocking OAI-SearchBot keeps you out of ChatGPT search; blocking Claude-SearchBot or PerplexityBot does the same for Claude and Perplexity, which for most brands is the opposite of what they want. * **Citation fetchers** (ChatGPT-User, Claude-User, Perplexity-User): fetch a page live when a user's question needs it. Blocking these prevents the model from retrieving your content at answer time, which directly reduces your chances of being cited. The common mistake is copying a robots.txt block of GPTBot from a "block AI" tutorial and accidentally extending it to the search and citation bots. If your goal is AI visibility, the search and citation bots are the ones that must stay open; whether to allow training crawlers is a separate business decision. ### Where blocks happen * **robots.txt**: each bot has a user-agent token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and so on) you can allow or disallow per token. Well-behaved AI crawlers respect it, and you can watch them fetch `/robots.txt` in your crawler logs, so a robots change shows its effect quickly. * **CDN and bot protection**: firewall rules, bot-fight modes, and rate limits at Cloudflare, Akamai, or similar often block AI crawlers by default or by heuristic. These show up as 403 responses in crawler logs, and they override whatever robots.txt says. * **Server errors and broken paths**: 404s from moved pages and 500s from failing routes waste the crawl on pages that return nothing. * **Content the crawler can't see**: most AI crawlers read the HTML they fetch without executing JavaScript the way a browser does, so content that only appears after client-side rendering may be invisible to them. [Site health](/docs/academy/site-health) is not that check: its word count comes from a headless-browser crawl that does run JavaScript, so use its thin-content flag for pages that are thin even after rendering, not as a proxy for what a no-JS bot saw. ### How to check your site in Promptwatch * **Watch status codes per bot**: in [Crawler Logs](/docs/academy/ai-crawler-insights), filter by status code. A healthy site shows mostly 200s; a cluster of 403s for one crawler is a block, and 404 spikes point at moved or deleted paths. The Promptwatch Crawler Logs table filtered to SearchBot and 403 status codes, with a red 403 badge on each row. * **Compare providers**: if GPTBot crawls daily but PerplexityBot never appears, Perplexity either can't reach you or doesn't know about you. A missing provider is as much a crawlability signal as an error code. * **Trust the verified numbers, where they exist**: anyone can send a request claiming to be GPTBot. Promptwatch checks crawler IPs against the ranges some providers publish (OpenAI, Perplexity, Google, and Anthropic's user and search fetchers), and requests that fail those checks are kept out of your numbers by default. Crawlers without a published IP list, like ClaudeBot, can't be verified this way, so treat their numbers with a little more care. * **Check what a crawl returns**: [Site health](/docs/academy/site-health) reports what the latest crawl of your pages actually found, including pages with thin content, missing titles, and slow responses. Crawl activity is the input; being cited is the output. Once bots can read your site, the question becomes whether they use what they read, which is where [citations](/docs/academy/citations-vs-mentions) and the [Page Tracker](/docs/academy/page-tracker) take over.