llms.txt Explained: What It Is and Whether You Need One

Few things in AI search are as misunderstood as llms.txt. Half the internet treats it as a magic ticket to AI citations; the other half confuses it with robots.txt. Neither is right. There are actually two files, they do different jobs, and the evidence on whether AI systems use them at all is… not flattering. Let me clear it up — with receipts.

Two files, two jobs

• llms.txt, the index. A short Markdown file (kept under ~10KB) with a curated list of links to your most important pages, organized by topic. Think of it as a table of contents or sitemap written for an LLM, so an agent can navigate to the right page.
• llms-full.txt, the whole thing. A single large Markdown file containing your entire documentation/content concatenated, so a model can ingest everything in one fetch. Many sites that adopt the idea publish both.

Proposed by Jeremy Howard of Answer.AI in September 2024 (spec at llmstxt.org), the format solves a real problem: model context windows cannot swallow a whole HTML site full of nav, ads and JavaScript, so a clean Markdown map/dump is easier to consume. Crucially, it is aimed at inference time — an agent you point at your docs — not at the big training or search crawlers.

Does anything actually use it? The receipts.

This is where the hype collides with reality:

• Google says no. In June 2025, Google's John Mueller stated plainly that "no AI system currently uses llms.txt," adding: "It's super-obvious if you look at your server logs… none of them fetch the llms.txt file." He compared it to the old keywords meta tag — something a site owner claims, that engines ignore (Search Engine Roundtable). Google has separately said normal SEO, not llms.txt, is what wins AI Overviews (Search Engine Land).
• The server logs agree. Multiple log analyses through 2025 found zero requests for llms.txt from GPTBot, ClaudeBot, PerplexityBot or Google-Extended — the very crawlers it is meant for.
• No citation lift. A large 2025 study of hundreds of thousands of domains found no correlation between having an llms.txt and being cited by AI systems (PPC Land).

The nuance everyone misses: published ≠ consumed

Here is the twist that fuels the confusion. The file is widely published — Anthropic's own developer docs serve one, and documentation platforms like Mintlify auto-generate it for thousands of sites. So you see llms.txt everywhere and assume it must matter. But publishing a file is not the same as a major AI provider reading it. No major LLM provider — OpenAI, Anthropic, or Google — has confirmed consuming it in production search/answers, and the objective measures (logs, citation studies) show negligible use.

The one place it genuinely helps today: developer tools and agents you explicitly point at your docs (Cursor, Claude Code, Copilot and similar) — they can ingest a clean llms-full.txt far more easily than crawling your HTML.

So should you add one?

• If you publish documentation a clean llms.txt / llms-full.txt is a reasonable, low-cost nicety for agent ingestion.
• Do not expect AI search citations from it. As of 2026 the evidence says it does not drive AI visibility.
• Spend your effort where it counts first: being crawlable and in the raw HTML (not client-rendered), not blocking the AI bots you want, and publishing genuinely useful, well-structured content. Those are what actually get you cited.

Bottom line: llms.txt is a tidy idea with real utility for agents and dev docs — and, as of now, no demonstrated impact on AI search visibility. Add it if it is cheap; never let it distract from the fundamentals.