Who Actually Serves an llms.txt? 184 Top Sites Measured

What the file is supposed to do

The idea behind llms.txt is straightforward. A site puts a markdown file at its root that introduces itself in plain language and then lists the URLs worth reading, grouped under headings. It is meant for a language model arriving with a question, in the same spirit that robots.txt is meant for a crawler arriving with a request. A companion llms-full.txt holds the expanded version with the content inlined rather than linked.

Whether any assistant reads either file is a separate question, and one I have written about before without being able to answer it. What I can answer is how many sites bothered, which nobody seems to have counted.

What I measured

Same sample as the other measurements this month: the 184 hosts from the top of the Tranco list that return a 200 and HTML to an ordinary browser. For each one I requested five paths once each and kept the response.

A hit has to be a real text file. That rules out two things that would otherwise inflate the count: an HTML page served at that path, which is how most sites answer a request for a file they do not have, and a catch all that returns something for every path you ask for. Two sites in the sample did exactly that, one answering with a JSON error object and one with binary, and both were excluded after inspection rather than counted.

Two of those results are worth pausing on. /.well-known/llms.txt, which is the location a standards minded person would expect, is used by nobody in this sample. And AGENTS.md turns up once at the web root, which fits what it is actually for: it lives in a code repository to instruct coding agents, not on a marketing site. I compared the three conventions in AGENTS.md vs llms.txt vs llms-full.txt and this is the measurement that backs up the distinction.

Who actually did it

Here is the whole list of 31 adopters from the sample. Read it as a group rather than one by one, because the pattern is the finding.

Developer tools, cloud infrastructure, security vendors, payment platforms, hosting companies and a handful of consumer brands that run large documentation estates. Cloudflare, GitHub, Stripe, Slack, HubSpot, Zendesk, Ubuntu, NVIDIA, Dell, Calendly, Plesk, Hostinger, DigiCert, Palo Alto Networks, Check Point.

What is absent says as much. No newspaper, no magazine, no broadcaster. No retailer. No social network. No government or institutional site. For a convention that gets discussed as a general web practice, adoption is sitting almost entirely inside companies whose customers are engineers and whose docs were already structured well enough to list.

That is not a criticism of the file. It is the most plausible reason it exists in those places: if you already maintain a documentation index, publishing a markdown version of it costs almost nothing. If your content is a news archive or a product catalogue, there is no obvious index to publish and the work is real.

What is inside them

These are not placeholder files. The median is 12,490 bytes and three of them exceeded the 200,000 byte ceiling I set on the reader, so the real top end is larger than I measured.

The ones I opened follow the published format closely: a level one heading with the company name, a single blockquote paragraph summarising what the company does, then sections of markdown links with a sentence of description on each. GitHub goes further and uses the file to point at its own APIs, telling an arriving agent to query the search endpoint rather than scrape the site, which is a genuinely clever use of the format.

Only 6 sites serve both llms.txt and llms-full.txt. The index version is the one people are willing to maintain. The inlined full text version, which is much larger and has to be regenerated whenever anything changes, mostly is not.

The part that keeps getting merged

Adoption and effect are two different questions and the gap between them is where most of the confusion sits. This measurement answers the first one. It says nothing whatsoever about the second.

I have not seen a controlled test showing that publishing one of these files changes how often a site gets cited, and I have run an unserious version of the experiment on a site of my own that suggested a file at the root is not obviously being read. So the honest position is that the file is cheap, harmless, plausibly useful, and unproven. All four of those at once.

That is a fine reason to publish one if your documentation is already indexed. It is a poor reason to sell it as a lever, or to put it ahead of things that are measurable. The first of those measurable things is whether the crawlers can reach your pages at all, which I found a fifth of top sites failing, and the second is whether your robots.txt names crawlers that still exist, where three of the ten most blocked names are retired. Both of those are worth doing before a new file goes at the root.

If you want to publish one anyway

It is an hour of work for a site with existing documentation and the downside is close to zero, so the case for doing it is fine as long as nobody is promised a result.

Put it at /llms.txt and nowhere else. The well known directory reads as more correct and is used by nobody, so a file there will be looked for by nothing. Serve it as text/plain or text/markdown, and check what your server actually sends, because a file served as text/html is indistinguishable from a soft 404 to anything reading it programmatically.

Follow the shape the adopters use: the company name as a heading, one paragraph saying plainly what you do, then grouped links with a sentence each. Link to pages that answer questions rather than to your navigation. Then put a reminder in whatever process owns your documentation, because a stale index of moved URLs is worse than no index, and this is exactly the kind of file that gets published once and never touched again.

Questions people ask about this

Method: five paths requested once per host on 18 September 2026 from a single residential connection in the United States, reading at most 200,000 bytes per response. A hit requires HTTP 200, a non HTML content type, a body of at least 200 bytes, and text that is not a JSON error object or binary. Two hosts were excluded after inspection for answering every path with the same error payload. Sample is the 184 hosts that returned 200 and HTML to a browser in the September crawler access study.