Definition
llms.txt is a proposed convention for a Markdown file, usually served at /llms.txt, that gives AI agents and language models a short, curated summary of a website plus links to its most useful pages. Jeremy Howard of Answer.AI published the proposal on September 3, 2024 and released a revised v2 at llmstxt.org on August 10, 2026.
The file is a reading aid, not a permission system. It contains no allow or disallow rules, no licensing terms and no attribution requirements. Crawler access is still controlled through robots.txt, and training opt-outs through mechanisms such as Google-Extended or a TDM rights reservation. Earlier descriptions of llms.txt as a 'robots.txt for AI' with directives like User-LLM or License do not match the spec.
The format is plain Markdown in a fixed order, so it can be parsed by code as well as read by a model:
- An H1 with the site or project name (the only required element)
- A blockquote with a one or two sentence summary
- Optional paragraphs explaining how to read the rest
- H2 sections with Markdown link lists, each link optionally followed by a short note
- An optional section titled 'Optional' for links an agent can skip when context is tight
The links should point to LLM-friendly versions of pages, typically the same URL with .md appended. Version 2 added three things: discovery through standard link relations (rel="alternate" type="text/markdown" for a page's Markdown version and rel="describedby" for the covering llms.txt, sent as HTML link elements or an HTTP Link header), acceptance of both page.md and page.html.md URLs, and defined scope for files in subfolders, so /docs/llms.txt covers everything under /docs/. A separate convention, llms-full.txt, bundles full page text into one file.
Use is real in software documentation. Mintlify, GitBook, Yoast SEO, AIOSEO and Wix can generate the file automatically, and OpenAI, Anthropic and Google publish llms.txt files for their developer docs. Chrome's Lighthouse includes an llms.txt check in its experimental Agentic Browsing category (a missing file is marked not applicable). Coding AI agents are the clearest consumers.
The evidence for search visibility points the other way. Google's Gary Illyes said in July 2025 that Google does not support llms.txt and is not planning to. In June 2026 Google added a line to its guide to generative AI features in Search stating that Google Search ignores llms.txt files and that publishing one will neither help nor harm rankings, including in AI Overviews and AI Mode. As of October 2026, no major AI search product has said publicly that llms.txt influences which pages it retrieves or cites. Promptwatch's crawler logs show AI crawlers occasionally requesting the file, with no preferential crawling, ranking or citation behavior attached, as covered in our post on why llms.txt has no impact on AI search and GEO.
Treat it as cheap housekeeping for agents, not a GEO tactic. If you run developer documentation or expect browser agents to work on your site, a short, accurate llms.txt that mirrors your sitemap's key sections is worth an afternoon. Keep sensitive or temporary URLs out, check that every link resolves, and test it by giving an agent only the file and asking questions about your product. If AI search visibility is the goal, the work that moves citations is still crawlable HTML, clear answers and third-party coverage. Our free llms.txt generator builds a starting file from your sitemap.
Examples of LLMs.txt
- A developer tools company publishes /docs/llms.txt with one link per API reference page in .md form. A coding agent asked to add the SDK reads the file, follows two links and writes the integration without scraping the HTML docs.
- An online retailer turns on Yoast's llms.txt generator. Three months later its crawler logs show a few dozen requests for the file, while its citation share in ChatGPT and AI Overviews has not moved, which matches Google's statement that Search ignores the file.
- A SaaS team runs Lighthouse's experimental Agentic Browsing audit and fails the llms.txt check because the file returns a 500 error. Fixing the server error clears the audit; deleting the file would also have produced a not applicable result.
- A news publisher adds 'Disallow: /' lines to llms.txt to stop AI training, then learns the spec has no such directive. It moves the rules for GPTBot, ClaudeBot and Google-Extended into robots.txt instead.
