Content Signals

Content Signals is an extension to robots.txt that lets a website state how its content may be used after a crawler fetches it, separately for three purposes: search, AI input and AI training. Classic robots.txt only says whether a bot may crawl a path. A Content-Signal line adds what the bot may do with what it collects.

Cloudflare launched the format as the Content Signals Policy on September 24, 2025 and released it under a CC0 license so anyone can use it. The policy text defines the three signals:

• search: building a search index and providing search results, meaning links and short excerpts. Cloudflare's definition explicitly excludes AI-generated search summaries.
• ai-input: feeding content into AI models at answer time, such as retrieval-augmented generation and grounding for generative AI search answers.
• ai-train: training or fine-tuning AI models.

Each signal is set to yes or no, for example Content-Signal: search=yes, ai-input=yes, ai-train=no. Leaving a signal out means no preference is expressed for that use. The policy text also states that restrictions are express reservations of rights under Article 4 of the EU copyright directive, which links it to TDM rights reservation.

The split matters because a robots.txt disallow is all or nothing. Blocking a crawler to stop training can also stop it from indexing or fetching your pages for answers, which removes you from those AI answers. Content Signals lets a site say 'cite me in AI answers, but do not train on me'. Note the detail in the definitions: since search does not cover AI summaries, a site that wants to be cited in generated answers needs ai-input=yes or no ai-input statement at all, not just search=yes.

Adoption grew mostly through Cloudflare's own defaults. At launch, Cloudflare added search=yes, ai-train=no (with no ai-input signal) to the managed robots.txt it served for more than 3.8 million domains. On September 15, 2026 it replaced its 'Block AI Bots' switch with separate controls for search, AI training and AI agents, and replaced managed robots.txt with Bot Preference Sync, which writes those choices into robots.txt. Cloudflare is also testing a fourth field, use, with values immediate, reference and full to describe what a crawler may keep and reuse. Independent crawl data from Keywords Everywhere's PoweredBy found that 4.74% of 20.47 million domains with a readable robots.txt declared content signals in September 2026, down from 7.12% in August, and only 4,202 declared ai-input=no.

Content Signals is a stated preference, not an enforcement mechanism. Compliance is voluntary, as with robots.txt, and Google Search Console may report the line as 'Syntax not understood'. Cloudflare pairs it with edge blocking for crawlers that ignore preferences, and publishers wanting formal terms add RSL or pay per crawl. A standard vocabulary for AI preferences is also being developed at the IETF.

For GEO teams, the practical step is to check what your robots.txt actually says today, especially if a CDN writes part of it, and make sure the signals match your goals: usually search and AI input allowed so you stay citable, with training decided as a separate business question. Then confirm in your AI crawler logs that search and answer-time fetchers still reach your pages. Our guide on seeing which AI bots crawl your site shows how to check.