llms.txt

What llms.txt is

llms.txt is a proposed convention: a Markdown file at your site root that hands language models a curated map of your most important content. Proposed by Jeremy Howard of Answer.AI in September 2024, it has one honest problem, no major AI engine has confirmed it actually reads the file, and one honest virtue: it costs almost nothing to publish and can't hurt you.

What one looks like in the wild

This site publishes a live one at seoprocheck.com/llms.txt. The opening lines:

# SEO ProCheck

> Technical SEO and AI-search consulting by Claude Vincent. Receipt-backed,
> no-hype guidance on how search engines and AI answer engines actually
> read, render, and cite websites.

## The AI-readability series (start here)
- [The Machine-Readable Web: What robots.txt, llms.txt, Schema, WebMCP and
  Entity Signals Actually Do in 2026](https://seoprocheck.com/research/ai-search/machine-readable-web-standards/)

That's the whole grammar: an H1 with the site name, a blockquote summary, then H2 sections containing curated link lists with one-line descriptions. Plain Markdown, served as text.

llms.txt vs robots.txt vs sitemap.xml

People keep filing these three under "root files for bots," which misses that they answer three different questions:

The honest adoption picture

As of now, none of the large AI providers has committed to reading llms.txt, and Google's John Mueller has publicly noted that server logs show AI systems aren't even requesting the file. Treat it as a cheap bet on a possible future, not a visibility tactic with measurable returns today. The work that actually moves AI representation, crawlable server-rendered HTML, clean structure, entity clarity, is covered in The Machine-Readable Web, which puts llms.txt in context alongside the standards that do have teeth.

What belongs in a good one

Since curation is the whole point, apply an editor's ruthlessness. Lead with the content that defines your site's expertise, cornerstone guides, original research, reference pages, not whatever published last. Give each link the one-line description the spec allows; a bare URL list wastes the format's only advantage over a sitemap. Group by intent ("start here," "reference," "tools") rather than by your CMS's category tree, because the file's imagined reader is a model deciding what to fetch next, not a human browsing your nav. And cap it: a dozen or two links with clear descriptions beats two hundred without. If everything is important, the file says nothing.

How to validate your llms.txt

1. Fetch https://yourdomain.com/llms.txt directly. You want a 200 status and a text response, not a 404, not an HTML error page in disguise.
2. Check the structure: exactly one H1, an optional blockquote summary, then H2-sectioned link lists. It's Markdown, so any renderer will show you if it's malformed.
3. Click every link. A curated file full of 404s and redirects is worse than no file, it's a map that lies.
4. Grep your access logs for llms.txt requests over a month. This tells you, with receipts, whether anything is actually consuming it.
5. Diff it against your current site priorities each quarter. Curation is the entire value proposition; a stale file has none.

Common mistakes and fixes

• Expecting it to affect rankings or AI citations. There's no evidence it does. Fix: publish it as a low-cost hedge and put your real effort into crawlable HTML.
• Auto-dumping every URL into it. That's a sitemap wearing a costume. Fix: pick the pages you'd show a smart stranger in five minutes, a generator like the site's llms.txt generator gives you the skeleton, but the curation is on you.
• Thinking it controls crawling. It has zero blocking power. Access control lives in robots.txt and your firewall, full stop.
• Confusing llms.txt with llms-full.txt. The companion convention inlines full page content into one giant file rather than linking out. Different file, different (equally unconfirmed) consumption story.
• Publishing once and forgetting it. If your best content changes and the file doesn't, you've built a monument to your 2024 site. Fix: tie updates to your content calendar.

FAQ

Does any major AI engine read llms.txt?

No confirmed consumption from OpenAI, Anthropic, Google, or Perplexity as of this writing. Some developer-facing tools and smaller agents do fetch it. Your own logs are the ground truth for your site.

So why publish one at all?

Ten minutes of work, zero downside, and a small option on the future, if an engine starts reading it, you're already there. Also, writing one forces a useful editorial exercise: deciding what your site's actual best content is.

Will it hurt anything?

No. It's advisory text at a URL nothing depends on. The only failure mode is letting it rot into a list of dead links.

Does it replace sitemap.xml?

Not remotely. Sitemaps are exhaustive, machine-consumed by search engines today, and tied to crawl scheduling. llms.txt is a curated pitch with no confirmed audience yet.

Where can I go deeper?

The full breakdown, origin, spec details, and whether your site specifically needs one, is in llms.txt Explained.