Promptwatch Logo

CCBot

CCBot is operated by the Common Crawl Foundation to crawl web content for AI training and research.
the Common Crawl FoundationCCBot
AI CrawlerAI Training

What is CCBot?

CCBot is the crawler for the Common Crawl Foundation, a nonprofit that makes large web-crawl datasets available at no cost for research and analysis. The crawler gathers pages for that shared repository. It is not itself a search engine or a language model.

Common Crawl's technical FAQ describes CCBot as Nutch based. Crawl candidates are extracted from its crawl database with Hadoop MapReduce, sorted by host, and distributed to crawler servers. Common Crawl samples the web rather than promising a complete archive of every site.

For an allowed page, CCBot makes an HTTP GET request. It supports HTTP/1.1 and HTTP/2 over TLS, can connect over IPv4 or IPv6, and follows a limited redirect chain. It does not currently execute JavaScript or use cookies, so content that exists only after client-side rendering may not be present in the fetched response.

Common Crawl data is reused by outside researchers and companies, including for AI training. That makes CCBot relevant to training-data policy, but the relationship is indirect: Common Crawl publishes crawl data, while downstream users decide how to filter and use it. Allowing a page does not guarantee model inclusion, a citation, or appearance in an AI search answer.

CCBot checks robots.txt before fetching a page and honors the CCBot group. It also supports Crawl-delay, slows down after HTTP 429 or 5xx responses, and periodically revisits robots.txt after a full-site block. Common Crawl says the crawler waits a few seconds between requests to the same site by default.

A sitemap declared in robots.txt can help CCBot discover eligible URLs. This may suit sites that want public research material represented while keeping private, licensed, or expensive application routes outside the crawl. Since CCBot does not log in or use cookies, authentication remains the stronger boundary for nonpublic content.

The current published user agent is CCBot/2.0 (https://commoncrawl.org/faq/); older logs may contain CCBot/1.0 (+https://commoncrawl.org/bot.html). Common Crawl also publishes current CIDR blocks at https://index.commoncrawl.org/ccbot.json and documents forward-confirmed reverse DNS under crawl.commoncrawl.org. This entry is verified, although its metadata does not mark Promptwatch traffic as IP verified.

Relevant for AI search

Is CCBot relevant for AI search?

Yes. CCBot collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

CCBot gathers public web content that can end up in the training data for large language models. Once your pages are in that set, they influence how the operator's models talk about you for that model generation. Allowing it lets your own writing carry weight in those answers; blocking it means the models learn about you from third parties instead.

How to handle CCBot

Use Common Crawl's documented token to stop future CCBot fetches across the site:

User-agent: CCBot
Disallow: /

Common Crawl says it follows this robots.txt rule and will continue checking the file periodically for changes. If load rather than collection is the concern, keep the desired paths open and place a Crawl-delay value under the same CCBot group. The operator gives Crawl-delay: 2 as an example of limiting requests to no more than one every two seconds.

For network-level verification, compare the source with https://index.commoncrawl.org/ccbot.json or complete a reverse lookup followed by a forward lookup for a hostname under crawl.commoncrawl.org. Keep access controls on login-only material even when robots rules are present.

Examples

  • An open-data project lists its sitemap in robots.txt so CCBot can discover public records that are licensed for broad research reuse.
  • A magazine allows article summaries but disallows its licensed archive because Common Crawl datasets can be downloaded and reused by third parties.
  • An operations team keeps CCBot enabled and adds `Crawl-delay: 2` after a crawl begins competing with normal traffic.

Frequently asked questions about CCBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

The Common Crawl Foundation operates CCBot. Common Crawl is a nonprofit that publishes web-crawl data for research and analysis.

Related bots

AI2Bot

AI2Bot is operated by the Allen Institute for Artificial Intelligence (Ai2) to crawl the web for content to train open-source AI models.

AI Crawler

aiHitBot

aiHitBot collects and maintains historical information about companies.

AI Crawler

Amazonbot

Amazonbot is Amazon's general web crawler for improving Amazon products and services.

AI Crawler

Applebot

Applebot powers search features in Apple's ecosystem (Spotlight, Siri, Safari) and may be used to train Apple's foundation models for generative AI features.

AI Crawler

Atlassian Rovo

Crawls and indexes web content for Atlassian Rovo's AI-powered search, chat, and agents.

AI Crawler

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard