Every day, Promptwatch processes roughly 50 million crawler events across properties, including traffic from Google, Bing, ChatGPT browsing agents, Perplexity, and others. In those logs, we do see occasional requests for llms.txt at the site root. Importantly, these requests are treated just like any other text asset: they are fetched and indexed, but we do not observe any preferential crawling, ranking, or citation behavior tied to the presence of llms.txt.
TL;DR
- Current state: Google says normal SEO is what matters for AI Overviews, and that llms.txt isn’t used today (Search Engine Land). Some bots fetch it, but we don’t see evidence it influences inclusion, ranking, or citations (based on Promptwatch logs).
- Agent‑friendly: Browser agents and RAG pipelines can benefit from a curated starting map of high‑value pages.
- Future‑leaning and low‑effort: Adoption takes minutes, does no harm when used carefully, and positions you for future support.
How llms.txt can add value
llms.txt can act as a “treasure map” for AI systems—more like a curated sitemap than a robots.txt. It can:
- Enumerate high-value, LLM-friendly pages (docs, FAQs, how-tos, research, benchmarks) for easier ingestion and citation.
- Provide lightweight hints about preferred canonical versions, update cadence, and important sections.
- Complement, not replace, sitemaps and robots.txt by offering semantic cues rather than crawl directives.
Today, it functions as an early‑stage, optional hint: most major systems don’t use it yet.
Reality check: Crawled, but not used
- Google: Public guidance is consistent—follow normal SEO best practices. llms.txt signals are not used for AI Overviews or ranking, and Google has no plans to rely on it.
- Industry commentary: Even when AI bots fetch the file, it does not translate to preferential crawling, indexing, ranking, or citation. Crawling logs alone are not evidence of influence.
Agent-first use cases (where llms.txt could help)
For autonomous and semi-autonomous agents, a compact, human-curated map can be useful:
- ChatGPT-style browsing agents that combine search with site navigation could read llms.txt to jump directly to definitive resources (docs, changelogs, pricing, comparison pages) instead of discovering them indirectly.
- Retrieval-augmented flows inside assistants could use it as an initial seed list for crawling, chunking, and indexing.
- Enterprise agents and internal knowledge assistants could standardize on llms.txt to unify where to start, especially across microsites.
Today, we do see browser agents fetch llms.txt in our logs, but the behavior matches generic asset fetches. There is no observable boost in inclusion, ranking, or citation.
Why it hasn’t impacted results yet
- Voluntary and unenforceable: AI providers are not obligated to honor it.
- Redundant: High-quality pages are already discoverable via HTML, sitemaps, and links.
- Unverifiable impact: No official analytics or feedback loops from major AI search experiences.
- Risky if misconfigured: Can unintentionally highlight sensitive or low-quality URLs.
What would make it useful
- Clear, stable spec and semantics that multiple providers agree to honor.
- Native support in major AI platforms and popular agent frameworks.
- Basic telemetry: fetch/use signals, validation, and warnings in webmaster consoles.
- Incentives for adoption (e.g., citation quality improvements, crawl efficiency) that publishers can measure.
Should you still add it?
If your CMS or plugin auto-generates it (e.g., some SEO plugins have experimented with support), adding a minimal, harmless version won’t hurt. Just don’t expect results. If you do publish one, keep it short, accurate, and aligned with your public sitemap. Avoid listing anything you wouldn’t want highlighted to crawlers.
Quick reasons to consider adding it now:
- Set a clear starting map for agents and internal assistants.
- Prepare for future platform support with minimal effort.
- Takes minutes to set up; low maintenance if you mirror your sitemap and key docs.
- Do no harm: safe when you avoid sensitive or ephemeral URLs.
Minimal llms.txt quick‑start (example):
version: 1 urls:
- https://example.com/docs/
- https://example.com/pricing/
- https://example.com/blog/ notes: canonical resources; updated monthly
For background, see our practical explainer: How to use llms.txt (and why you probably don’t need it) and the data study on browsing behavior: How ChatGPT searches the web.
What actually works for AI Overviews and GEO
- Answer-centric content: Clear, concise sections that directly answer common questions with evidence.
- Comparison-first pages: AI assistants surface comparisons constantly—build “X vs Y” and alternative roundups with transparent criteria.
- Citations and sources: Link to primary sources and include factual, attributable statements that models can quote.
- Structured signals: Clean titles, headings, summaries, tables, and schema where appropriate.
- Technical foundation: Fast, crawlable pages with solid internal linking and canonicalization.
See also: What is Generative Engine Optimization (GEO)? and Why GEO matters more than ever.
Simple checklist
- Clarify intent: Informational vs. commercial vs. comparative.
- Write for retrieval: Skimmable headings, answer blocks, and structured data.
- Include comparisons: Position your pages to be cited in "X vs Y" contexts.
- Cite sources: Provide verifiable facts and references.
- Measure visibility: Track where and how AI assistants mention your brand.
Bottom line
llms.txt has no meaningful impact on AI search or GEO today. Focus on content and technical practices that AI assistants demonstrably use. When major AI platforms publish support and telemetry for llms.txt, we’ll revisit—until then, it’s optional housekeeping at best.
—
Want to monitor how your brand shows up in AI answers? Promptwatch tracks your visibility across AI platforms, including when you’re cited and in what context.
