Crawl Budget

Crawl budget is the number of URLs a search engine crawler can and wants to crawl on a site in a given period. Google defines it as the combination of two things: the crawl capacity limit, how much crawling a site's servers can handle without slowing down, and crawl demand, how much Google wants to crawl based on a site's size, popularity, update frequency and quality. If either is low, fewer pages get crawled, and pages that are not crawled cannot be indexed or refreshed.

Most sites never need to think about it. Google's crawl budget guide says it is mainly relevant for sites with over 1 million unique pages that change weekly, sites with over 10,000 pages that change daily, and sites where Search Console shows a large share of URLs as 'Discovered, currently not indexed'. If new pages are crawled the day they are published, crawl budget is not your problem.

Google treats each hostname as a separate site with its own budget, so www.example.com and shop.example.com are budgeted separately. The capacity limit rises when the server responds quickly and consistently and falls when responses slow down or return 5xx errors or 429 rate-limit responses. Google also notes that the capacity limit is shared across all of its crawlers, so heavy demand from one can leave less for others.

The main cause of wasted budget is a bloated URL inventory: faceted navigation and filter combinations, session or tracking parameters, internal search results, infinite calendars, duplicate versions of the same page, soft 404s and long redirect chains. Google's recommended fixes are specific. Consolidate duplicates with canonical URLs or redirects, block URLs you never want crawled in robots.txt, return 404 or 410 for removed pages, keep XML sitemaps current with accurate lastmod dates, support 304 Not Modified responses, and make pages fast to load. Google also warns against two common workarounds: using noindex to save budget (Google still fetches the page) and temporarily blocking pages in robots.txt hoping budget shifts elsewhere.

AI crawlers add a second layer. OpenAI, Anthropic, Perplexity, Meta, Apple and others run their own bots, each with its own schedule and no published budget model, and they share your server capacity with Googlebot and Bingbot. Cloudflare reported in September 2026 that mixed-use crawlers, which collect content for both search and AI training, made up 36.6% of verified crawler traffic on its network, the largest single category. Many AI crawlers fetch raw HTML without running JavaScript, so pages that depend on client-side rendering cost a fetch and may still return little content. Some operators honor the non-standard Crawl-delay directive (Anthropic says its bots do), which can reduce load but also slows how quickly they see changes.

For AI visibility, the useful questions are which bots reach your important pages, how often, and with what status codes. Server or CDN logs answer that; Search Console's Crawl Stats report only covers Google. Look for answer-time fetchers such as ChatGPT-User or Claude-User hitting errors, training crawlers spending most requests on low-value URLs, and important pages that no AI bot has requested in weeks. Promptwatch's AI crawler traffic data shows how the mix of AI bots has shifted across sites, and AI crawler logs are the way to see it for your own. For Bing, which also feeds Microsoft Copilot, IndexNow can notify the engine of changed URLs directly instead of waiting for a recrawl.