AI Summary
AGENTS.md, llms.txt and llms-full.txt are three plain files that look similar but serve different readers. AGENTS.md is a README for coding agents working inside your code repository. llms.txt and llms-full.txt live at your website root and give answer engines a clean version of your published content.
- AGENTS.md: instructions for coding agents (build, test, conventions). Lives in a code repo, not on a website. Read by Claude Code, Codex, Cursor, Copilot and others.
- llms.txt: a curated markdown index of your key pages, at
/llms.txt. A map for language models, not a ranking file. - llms-full.txt: the same idea but with the full page text inlined into one file, for text heavy documentation.
- Underneath all of them sits the access and licensing layer:
robots.txt, RSL and AI bot controls decide who may crawl, train, and pay.
These three files get lumped together because they are all plain text you drop somewhere and hope a machine reads. They are not interchangeable. One is for agents that write code, two are for models that read your website, and confusing them wastes effort on files nobody in your audience will ever fetch. Here is what each one actually does, how they differ, and what else in the same family is worth knowing about.
AGENTS.md: a README for coding agents
AGENTS.md is a markdown file you place in the root of a code repository. Its job is simple: tell an AI coding agent how to work in this project. The official description calls it "a README for agents". Where README.md explains a project to a human, AGENTS.md explains it to the agent that is about to build, test, and commit code.
A typical AGENTS.md covers the things a new engineer would ask on day one: how to install and build, how to run tests, the code style to follow, commit and pull request conventions, and any security notes. There is no required schema. It is ordinary markdown with whatever headings you find useful, so an agent simply parses the text you give it.
The format matters because of who backs it. AGENTS.md started as an OpenAI initiative in August 2025, and by December 2025 it moved under the Linux Foundation's Agentic AI Foundation, alongside Anthropic's Model Context Protocol. Backers include OpenAI, Anthropic, Google and AWS. By mid 2026 it is read natively by more than thirty tools, including Claude Code, OpenAI Codex, Cursor, GitHub Copilot, Devin, Aider, Gemini CLI, Windsurf and Amazon Q, and it appears in tens of thousands of public repositories. That breadth is the point: before AGENTS.md, every tool wanted its own file (one for Cursor, one for Claude, one for the next tool), and teams drowned in near duplicate instruction files. AGENTS.md is the consolidation.
In a monorepo you can place nested AGENTS.md files inside individual packages. Agents read the nearest file in the directory tree, so the closest one wins and each subproject can ship tailored instructions. OpenAI's own repository reportedly carries dozens of them.
The key thing for a website owner: AGENTS.md is not a website file. It is read from a code checkout, not fetched over HTTP by a search crawler. If you do not ship code that agents operate on, you do not need one.
llms.txt: a curated map for language models
llms.txt is a markdown file at your website root, at /llms.txt. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024. The idea is to hand a language model a clean, curated map of your site so it can find the pages that matter without wading through navigation, cookie banners and templated boilerplate.
The format is defined and short:
- An H1 with the site or project name. This is the only required line.
- A blockquote with a short summary of what the site is.
- Optional free markdown (paragraphs, lists) with more detail.
- One or more H2 sections, each a list of links in the form
[Title](url): optional note.
A section literally named ## Optional has special meaning: those links are secondary and can be skipped when a shorter context is needed. The spec is explicit that llms.txt is meant for inference time, that is, the moment a user is asking a model for help, rather than for training. It is a discovery and curation aid, not a data dump.
llms-full.txt: the whole thing in one file
llms-full.txt uses the same structure as llms.txt but does not stop at links. It inlines the full text of your pages into a single large markdown file, so a model can absorb the substance without fetching every URL one at a time.
A small naming note that trips people up: the reference tooling that ships with the spec (the llms_txt2ctx generator) produces files called llms-ctx.txt and llms-ctx-full.txt, where the "full" version also expands the links marked Optional. In practice the community settled on the plainer name llms-full.txt, and that is what you will see on most sites that publish one.
Use llms-full.txt only when your content is genuinely text heavy and worth reading in full: product documentation, references, a guide library. For a marketing site or a blog, the curated llms.txt index alone is usually enough, and a multi megabyte full dump is just weight nobody asked for.
AGENTS.md vs llms.txt vs llms-full.txt, side by side
| Dimension | AGENTS.md | llms.txt | llms-full.txt |
|---|---|---|---|
| Who reads it | Coding agents (Claude Code, Codex, Cursor) | Answer engines (ChatGPT, Claude, Perplexity) | Same, when they want full text |
| Where it lives | Code repo root (and nested) | Website root, /llms.txt | Website root, /llms-full.txt |
| What it contains | How to build, test, and follow conventions | Curated links to key pages | The full page text inlined |
| Layer | Execution: how to work in the code | Retrieval: what your content says | Retrieval, full context |
| Format | Free markdown, no schema | Defined markdown (H1, blockquote, H2 link lists) | Same, plus inlined body text |
| Fetched how | From a code checkout | Over HTTP by a model or tool | Over HTTP |
| Do you need it | If agents touch your code | Optional, low cost, unproven for SEO | Only if content is text heavy |
The honest part: does any of this move rankings
AGENTS.md and llms.txt sit at very different levels of proof, and it is worth being blunt about which is which.
AGENTS.md demonstrably works, because you can watch it work. Open a repo in a supporting tool, add an AGENTS.md, and the agent follows it. The benefit is verifiable on your own machine.
llms.txt is a different story. Adoption is real but modest: analyses in 2026 put it on roughly ten percent of surveyed domains, with a few thousand sites in tracked technology profiles, including names like Anthropic, Cloudflare and Hugging Face. What is missing is evidence that the big answer engines actually fetch it to decide what to cite. Google has said it does not use llms.txt, and John Mueller has publicly likened its value to the old keywords meta tag. No major AI search provider has confirmed that publishing one changes whether or how you get cited.
So the honest read is this. Publish llms.txt if you run documentation an agent might navigate, or because it costs a couple of hours and does no harm. Do not publish it expecting a ranking or citation lift, and do not let it distract from the things that are proven to matter: content quality, crawlability, rendering, structured data and internal links. If you want the evidence based version of that argument, see llms.txt Explained: What It Is and Whether You Need One.
What else is in this family, and what is actually missing
AGENTS.md and the llms files are two visible corners of a much larger set of machine readable signals. If the question is "what else should we be talking about," these are the pieces that matter, and most of them are better proven than llms.txt.
| Signal | Layer | What it does |
|---|---|---|
robots.txt | Access | The original crawler control file. Now also where you allow or block AI bots by user agent (GPTBot, ClaudeBot, Google-Extended, CCBot). |
| XML sitemaps | Discovery | The machine map of which URLs exist and when they changed. Proven, boring, essential. |
| Schema.org / JSON-LD | Meaning | Structured data that tells engines what an entity or page is. The most reliably consumed machine signal there is. |
| RSL | Licensing | Really Simple Licensing, an official standard since December 2025. Adds machine readable license terms to robots.txt: attribution, pay per crawl, pay per inference. Backed by the AP, Guardian, Yahoo and others. |
| MCP / WebMCP | Action | Lets agents do things on your site, not just read it. The tool calling layer of the agentic web. |
| ai.txt / TDMRep | Licensing | Earlier opt out signals for AI training and text and data mining. Narrower and less adopted than RSL, but still seen. |
| C2PA / Content Credentials | Provenance | Cryptographic "where did this come from" metadata for images and media. Growing as AI generated media spreads. |
.well-known/ | Discovery | The standard directory for machine files like security.txt. Where several of these signals are increasingly expected to live. |
The single biggest gap in most "AI readiness" checklists is the licensing layer. Everyone argues about llms.txt while RSL quietly became the standard that decides whether AI companies can use your content and whether you get paid for it. If you publish anything of value, that is the conversation worth having. For the reader side of the same coin, our complete robots.txt reference covers how the AI bots are actually named and controlled, and WebMCP, Explained covers the action layer.
Which file should you actually create
Skip the checklist theatre. Match the file to what you have.
| If you have | Create |
|---|---|
| A code repository agents work in | AGENTS.md at the repo root (nested files per package in a monorepo) |
| A documentation or knowledge site you want quoted accurately | llms.txt, and llms-full.txt only if it is text heavy |
| Content you want to license or protect from AI use | robots.txt rules plus RSL license terms |
| Actions you want agents to perform | An MCP server or WebMCP endpoint |
| A brand you want recognized as an entity | Schema.org markup and clean entity HTML |
For the wider picture of how these fit together, our overview of the machine readable web maps the whole stack, and how to become a cited source in AI answers covers the parts that actually influence citations.
FAQ
No, but it is the successor to them. Tool specific files like CLAUDE.md and .cursorrules each spoke to one agent. AGENTS.md is the cross tool standard that many of those same tools, including Claude Code, now read. Some tools still fall back to their own file if present, so you may keep both during the transition, but AGENTS.md is the one place they all agree to look.
There is no confirmed evidence that it does. Google has said it does not use llms.txt, and no major answer engine has confirmed fetching it to decide citations. Treat it as a low cost, low risk convenience for documentation and agent navigation, not as a ranking tactic.
Usually not. Publish llms.txt as a curated index first. Add llms-full.txt only when your content is text heavy documentation and you want a model to read the whole thing in one fetch. For a blog or marketing site, the full dump is unnecessary weight.
It goes in the root of a code repository, with optional nested files inside subprojects where the nearest file wins. It is read from a code checkout, not fetched by a search crawler, so putting one at your public website root serves no reader. Answer engines look for llms.txt there, not AGENTS.md.
llms.txt is a curated list of links to your key pages. llms-full.txt inlines the actual text of those pages into one file. The first is a map, the second is the territory. Same format, very different file size and purpose.
The proven signals: correct robots.txt and sitemaps, server rendered HTML that crawlers can read, Schema.org structured data, strong internal linking, and genuine content quality. If AI licensing matters to your business, look at RSL. Those move the needle in ways llms.txt has not been shown to.
Want your site read correctly by search engines and AI both
An advanced SEO audit checks the signals that are proven to matter, from rendering and structured data to how the AI crawlers actually see your pages.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
