Definition
Common Crawl is a nonprofit foundation that crawls the public web and publishes the results as a free, open archive of web pages, metadata and extracted text. It has collected data since 2008, and its archive, measured in petabytes, is one of the most widely used sources of AI training data for large language models. The Atlantic reported in November 2025 that OpenAI, Google, Anthropic, Nvidia, Meta and Amazon have all used it to train models.
What it collects
The crawler is CCBot, which identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/), follows robots.txt, and runs from dedicated IP ranges with reverse DNS under crawl.commoncrawl.org, published as JSON so site owners can verify it. Common Crawl warns that other crawlers impersonate CCBot. A new crawl is released roughly every month. The September 2026 crawl (CC-MAIN-2026-39), gathered between September 4 and 17, holds 2.17 billion pages from 33.2 million registered domains, or 361.4 TiB uncompressed, including 587.2 million URLs not seen in earlier crawls.
Each crawl ships as WARC files (raw HTTP responses), WAT files (metadata) and WET files (extracted plain text), plus URL indexes. The data is free to use on AWS's public datasets program, and selected crawls are mirrored on Hugging Face. AI developers rarely train on the raw archive. They filter, deduplicate and quality-score it into their own datasets. The New York Times' 2023 lawsuit against OpenAI cited Common Crawl as 60% of GPT-3's training data, and a 2024 Mozilla Foundation report argued that generative AI in its current form would probably not have been possible without it.
The dispute with publishers
That role has made Common Crawl a target. In November 2025 The Atlantic reported that the archive exposed paywalled news articles to AI developers and that content from publishers who had asked for removal was still available. Common Crawl denied scraping paywalled material or misleading publishers, and said its WARC archives cannot be edited, so removals apply to future crawls and its public indexes. Since September 2025 it has published an Opt-Out Registry of the legal removal requests it receives, which lists publishers such as the New York Times, the BBC, The Guardian and The Washington Post.
Pressure grew in 2026. The News/Media Alliance sent a letter on April 29 asking Common Crawl to remove publisher content on request and to prohibit unauthorized AI use in its terms. In June, Digital Content Next, whose members include the AP, the New York Times and Bloomberg, sent a cease-and-desist letter alleging copyright infringement and demanding removal of members' content from existing datasets.
What it means for AI visibility
For GEO, Common Crawl matters through parametric knowledge: what a model knows about a brand without searching comes largely from web text collected before its knowledge cutoff, and Common Crawl feeds that pipeline for many models. Blocking CCBot keeps future pages out of its archive and out of models trained on later crawls. It does not remove pages already captured, and it has no effect on the search and citation crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot that fetch pages for live answers.
The choice is a trade-off. Publishers that license their content or want to limit training usually block CCBot in robots.txt with User-agent: CCBot and Disallow: /, and some also file a legal opt-out. Brands that want models to know their products, pricing and positioning generally leave it allowed and make sure key pages are crawlable HTML, since training data optimization depends on appearing in corpora like this one. In both cases, verify CCBot traffic by IP or reverse DNS rather than by user agent. Common Crawl's CCBot page documents the user agent and IP ranges, and our CCBot bot profile explains how to recognize it in logs.
Examples of Common Crawl
- A news publisher blocks CCBot in robots.txt and files a legal request to join Common Crawl's Opt-Out Registry, while keeping OAI-SearchBot and PerplexityBot allowed so its articles can still be cited in AI search.
- A B2B software company searches the Common Crawl URL index for its domain and finds its new pricing pages are missing because their content renders client-side, so it adds server-side rendering.
- A log review shows requests claiming to be CCBot from cloud IPs outside the ranges in Common Crawl's ccbot.json. Reverse DNS does not resolve to crawl.commoncrawl.org, so the team blocks them as impersonators.
- A research lab builds a pretraining set from the WET text files of several 2026 crawls and removes every domain listed in the Opt-Out Registry before training.
