llms.txt: what it does and what it doesn't do
llms.txt is a proposed convention, not a standard the big AI engines have committed to. What the file is, what it can realistically do for you, what it can't, and how to check whether anything reads it.
Few things in AI SEO get pitched as hard as llms.txt. Add one file, the pitch goes, and ChatGPT will understand your site. We get asked about it on nearly every first call, so here's the plain version of what it is and where it fits.
What llms.txt is
llms.txt is a proposal published in September 2024 by Jeremy Howard of Answer.AI, documented at llmstxt.org. The idea is simple. Put a Markdown file at the root of your site, at /llms.txt, that gives a language model a short, curated guide to your content.
The proposed format goes like this:
- An H1 with the name of the site or project
- A short summary in a blockquote
- Optional paragraphs with extra context
- H2 sections, each with a list of links to important pages and a one-line note on what each covers
- An optional section for secondary links a model can skip when it's short on context
The proposal also suggests offering clean Markdown versions of key pages at the same URL with .md appended. Some documentation platforms go further and publish an llms-full.txt with the full text of the docs in one file. That's a convention layered on top, not part of the original proposal.
What it doesn't do
Here's where the sales pitch and the facts part ways.
It isn't an access control file. robots.txt tells crawlers what they may fetch. llms.txt tells nobody anything about permissions. If you want to allow or block GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot, that still happens in robots.txt and at your CDN.
No major AI engine has publicly committed to using it. As far as we can find, OpenAI, Google, Anthropic, and Perplexity haven't documented llms.txt as something their search crawlers read or rank on. Google's own guidance for AI Overviews and AI Mode says the standard SEO basics apply and doesn't list it. If that changes, we'll change this post. Until then, anyone promising citations from an llms.txt file is promising something nobody has confirmed.
It doesn't fix a page that can't be quoted. If your pricing page is vague or your docs only render client-side, a tidy index pointing at them won't help. The engine still has to fetch and understand the page itself.
What it can do
That doesn't make it useless. It just narrows where it earns its keep.
It helps when someone points a model at it on purpose. Developers paste docs URLs into coding assistants and chat tools all the time. Some tools and agent frameworks can be told to read llms.txt, and a clean Markdown map of your docs is easier for them to use than a JavaScript-heavy site.
It forces a useful editorial exercise. Writing a good llms.txt means deciding which twenty pages actually explain your product. That list is worth having for its own sake. Often it shows that the page you'd want a model to read doesn't exist yet, or that three pages say slightly different things about the same feature.
It's cheap. A first version takes an hour or two. The cost of shipping one is close to zero, so the bar for value is low.
How we'd write one
If you decide to ship one, keep it short and honest.
- Write the H1 and a two-sentence summary of what the company does and who it's for. Plain language, no slogans.
- Group links under a few H2 headings that mirror how buyers think: product overview, pricing, integrations, docs, comparisons.
- Give every link a one-line description that says what the page answers, not what it sells.
- Link only to pages that load as clean HTML and say something specific.
- Put a date in the file or your release process so it gets updated when pages move. A map full of redirects is worse than no map.
Then make sure the file returns a 200, isn't blocked at the CDN, and is served as plain text.
How to tell whether anything reads it
This is the part almost nobody does. You don't have to take anyone's word about llms.txt, including ours. Your server logs will tell you whether AI crawlers ever request the file.
We check it for clients with Agent Analytics in Promptwatch. It reads AI crawler logs through integrations with Cloudflare, Vercel, Fastly, AWS CloudFront, and other CDNs, so you can filter for requests to /llms.txt and the .md pages and see which bots asked for them, how often, and with what response code. If nothing requests the file over a couple of months, you've learned something real about its value for your site. If a bot does fetch it, the same view shows whether that bot went on to fetch the pages it links.
That's a better basis for a decision than a vendor promise or a conference slide.
Where it sits on the list
In our audits, llms.txt sits well below crawler access, page content, and offsite sources. If your important pages are blocked, empty without JavaScript, or missing the facts buyers ask about, fix those first. Once they're in good shape, adding an llms.txt is a reasonable small task, especially for developer products with large docs. Just ship it as a tidy experiment, measure whether anything reads it, and keep expectations where the evidence is.