What an llms.txt file actually is
The llms.txt file is a plain text, Markdown formatted file you place at the root of your domain, so https://yourdomain.com/llms.txt. It was proposed in September 2024 by Jeremy Howard of Answer.AI. The idea is simple: give a language model a short, curated map of the pages on your site that you actually want it to read and cite, instead of forcing it to crawl a mess of navigation, ads, cookie banners, and JavaScript to find the good stuff.
The problem it targets is real. Model context windows are finite, and raw HTML is a terrible way to feed a page to an LLM. A single product page can be ten thousand tokens of markup wrapping two hundred tokens of useful text. llms.txt is a proposed shortcut: you hand the model a clean list of links with a one line description for each, and optionally point at Markdown versions of those pages so the model reads prose instead of parsing your theme.
I want to be blunt about status, because there is a lot of hype. This is a community proposal. It is not part of any search engine guideline, and as of now the major model providers have not committed to fetching it at inference time the way crawlers fetch robots.txt. Treat it as low cost hygiene for content heavy sites, not as a growth lever.
llms.txt vs robots.txt vs sitemap.xml
People conflate these three because they all live at the root. They do different jobs.
| File | Audience | Format | Job |
|---|---|---|---|
robots.txt | Crawlers | Directives | Says what may or may not be crawled |
sitemap.xml | Search engines | XML | Lists every indexable URL for discovery |
llms.txt | Language models | Markdown | Curates your most valuable pages for AI reading |
The key mental shift: a sitemap is exhaustive and machine ranked, an llms.txt is short and human curated. You do not dump five thousand URLs into it. You pick the twenty or fifty pages you would want quoted back to someone asking about your domain.
The structure the spec expects
The format is opinionated in a good way. A valid file follows this shape:
- An H1 with the site or project name.
- A blockquote summary, one or two sentences on what the site is.
- Optional prose giving context a model should know.
- H2 sections, each holding a bullet list of links. Every link gets a short description after a colon.
A section named Optional is a convention for links a model can skip if it is tight on context. Here is a minimal example:
# Acme Docs > Developer documentation for the Acme API and SDKs. ## Docs - [Quickstart](https://acme.com/docs/quickstart.md): Get a first request working in five minutes - [Auth](https://acme.com/docs/auth.md): API keys, OAuth, and token rotation ## Optional - [Changelog](https://acme.com/changelog.md): Release history
Note the .md links. A common companion pattern is to publish a Markdown twin of each page so the model reads clean prose. Some teams also ship a fatter llms-full.txt that inlines the entire content of key docs into one file.
How the flow works
How to build and validate one
The work is editorial, not technical. Steps that hold up:
- List your genuinely high value pages: docs, pillar guides, product references, pricing, policies. Cut the fluff.
- Write one honest sentence per link. This is the part that helps a model choose correctly, so do not phone it in.
- Group links under clear H2 headings. Park anything skippable under an
Optionalsection. - Serve the file as
text/plain; charset=utf-8at the exact root path. A 200 status matters, a redirect chain does not count. - Verify it renders by loading the URL in a browser and by fetching it with curl. If you built Markdown twins, spot check that those
.mdURLs actually return the clean content.
Detection and monitoring: check your server logs for hits on /llms.txt to see whether anything is actually requesting it. Screaming Frog can crawl the URL to confirm it returns 200 and the right content type, and you can list it as a custom URL to keep an eye on it alongside your other root files.
DO and DON'T
- Curate ruthlessly: your best twenty to fifty pages, not everything.
- Write a real one line description for each link.
- Serve it at the root with a 200 and text/plain.
- Keep it in sync when pages move or get retired.
- Consider Markdown twins for docs heavy sites.
- Treat it as a ranking factor or a traffic tactic.
- Dump your whole sitemap into it.
- Assume every AI system fetches it. Most do not yet.
- Let it rot: a stale map is worse than none.
- Use it to hide or cloak content from crawlers.
FAQ
Does Google use llms.txt for rankings?
Is it llms.txt or llm.txt?
Do I still need robots.txt and a sitemap?
Who benefits most from one?
Can it leak content I do not want scraped?
We audit robots.txt, sitemaps, llms.txt, and how crawlers and AI systems actually see your site, then hand you a prioritized fix list.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
