/llms.txt is a proposed convention: a markdown file at the root of your site that tells a language model what the site is and where its most useful pages live. It is short, human-readable, and deliberately unlike robots.txt — it is not a permission file. It grants nothing and forbids nothing. It is a table of contents written for a reader that arrived with no context.
The format is plain markdown, and it has four parts:
- an H1 with the site's name;
- a blockquote summarising the site in a sentence or two;
- H2 sections grouping the pages that matter;
- under each, a linked list — one line per page, with a short note about what that page contains.
A companion /llms-full.txt is sometimes served alongside it, containing the actual content rather than links, for a reader that would rather not make fifty more requests.
What problem it is trying to solve
A model working from your website has a context window and a fetch budget, and your sitemap helps with neither. A sitemap is a machine-readable list of every URL, optimised for a crawler that will visit all of them over days. It has no notion of importance, no summary, and no prose. If an assistant can only afford to read five pages of your site before answering a question, a sitemap does not help it choose which five.
That is a real problem, and llms.txt is a plausible answer to it: here is what we are, here are the five pages, here is a sentence about each. Written well, it is also a useful exercise for the humans — a site that cannot summarise itself in a blockquote usually has a positioning problem rather than a markup problem.
Who actually reads it
This is where honesty is required, and where most coverage of the format stops being reliable.
The format is a community proposal. It is not a standard published by a standards body, it is not part of any search engine's documented crawl behaviour, and as of this writing no major model provider has publicly committed to reading it as a ranking, retrieval or citation input. Files are being served and files are being fetched — but "somebody requested the file" is a long way from "the file changed what a model said about you", and the second claim is the one that would justify a project.
Both sides of the decision are worth stating fairly, so here they are next to each other:
- The case for serving one
- Adding the file is cheap; it is a static document. If a convention like this does settle, the sites that already serve one lose nothing by being early. And the file is useful today in a way that has nothing to do with models: it is a decent entry point for anyone — human or agent — trying to understand a site quickly, and you can point people at it directly.
- The case against
- Every file you serve is a file that can go stale, and a stale llms.txt is strictly worse than none: it is an authoritative-looking document confidently describing a version of your site that no longer exists. If you cannot keep it current, do not publish it.
The comparison people keep making, and why it misleads
It is often described as "robots.txt for AI". That comparison is wrong in the way that matters. robots.txt is honoured because crawler operators chose, decades ago, to honour it, and because ignoring it carries a reputational and occasionally a legal cost. Its effect does not depend on anyone finding it useful — it depends on a norm. llms.txt has no such norm behind it. Its effect depends entirely on someone deciding the file is worth fetching and trusting.
If your goal is to control which AI crawlers may fetch your site, the file for that is still robots.txt, and that mechanism does work today, with the usual caveat that it is a request and not a fence.
How we treat it
We generate and serve llms.txt and llms-full.txt for sites we operate, and our audit flags a missing or unhealthy file as a finding. We serve one for this site too, at /llms.txt — generated from the same page and article lists that build the site's sitemap, so it cannot describe a page we stopped serving. We do that because the cost is close to zero and the file is genuinely useful to a human reader — not because we can show you that it moved anything. We cannot, and we have not seen anybody else demonstrate it either.
So treat it accordingly. It is a reasonable thing to serve, it is a bad thing to build a strategy around, and if a vendor's pitch leans on llms.txt as a primary mechanism, ask them for the evidence. The correct current answer to "does this work" is that nobody knows yet, and a file that costs an afternoon is a fine bet under that uncertainty. A quarter of engineering time is not.