AI Training Data

AI training data is the content used to train an AI model: for large language models, mostly text and code from crawled web pages, books, academic papers, code repositories, licensed archives and, increasingly, synthetic data generated by other models. Training turns this content into the model's parametric knowledge, the facts and associations it can recall without looking anything up. Frontier models are trained on many trillions of tokens, and providers rarely disclose exact sources.

Training data matters for AI visibility in a different way from retrieval. When an assistant searches the web, it cites what it finds at that moment, which is governed by its search index. When it answers without searching, or decides which brands to look up in the first place, it relies on what it learned in training. A brand that was well and consistently described across the web before a model's knowledge cutoff is more likely to be named from memory; a brand that was thin or contradictory in the training data may be omitted or described with outdated or invented details.

Site owners have some control over whether their pages are collected for training, separately from search:

• OpenAI's GPTBot collects content that may be used to train its models; blocking it in robots.txt does not remove a site from ChatGPT search, which depends on OAI-SearchBot. See OpenAI crawlers.
• Google-Extended is a robots.txt token, not a separate crawler. Google says it is the control for limiting training of the models used in Search's generative AI features, and that the Search generative AI control in Search Console, which removes a site from AI Overviews and AI Mode, does not affect training (Google Search Console Help).
• Other training-related agents include Anthropic's ClaudeBot, Apple's Applebot-Extended token and Common Crawl's CCBot, whose open archive feeds many training datasets.

Law and contracts are adding more structure. Under the EU AI Act, providers of general-purpose AI models must, since August 2, 2025, publish a summary of their training content using a mandatory Commission template and maintain a policy that respects machine-readable opt-outs from text and data mining, the TDM rights reservation. In the US, courts are still working through copyright claims. In Bartz v. Anthropic, a judge ruled in 2025 that training on lawfully acquired books was fair use but that downloading pirated copies was not, and the resulting $1.5 billion settlement, roughly $3,000 per book, received final approval on July 20, 2026. Many publishers now license content to AI companies directly; see publisher licensing.

An llms.txt file is sometimes described as a way to steer what models learn, but it is not a training-control standard, and our analysis found it had no measurable impact on AI search.

For GEO teams, the practical levers are indirect. Publish accurate, crawlable, consistent information about your brand, products and expertise on your own site; earn coverage on the reputable third-party sources that are widely crawled and licensed; keep entries like Wikipedia and Wikidata correct where you qualify; and decide crawler by crawler whether the value of being learned by future models outweighs your reasons to opt out. Then check what models say about you without web search, which reveals the training-data picture directly. See training data optimization.