# SEO ProCheck
> Technical SEO and AI-search consulting by Claude Vincent. Receipt-backed, no-hype guidance on how search engines and AI answer engines actually read, render, and cite websites. This file inlines the full text of the cornerstone AI-readability guides for models that want the substance in one fetch.
# AGENTS.md vs llms.txt vs llms-full.txt: Which Agent File Does What
Source: https://seoprocheck.com/research/ai-search/agents-md-vs-llms-txt/
These three files get lumped together because they are all plain text you drop somewhere and hope a machine reads. They are not interchangeable. One is for agents that write code, two are for models that read your website, and confusing them wastes effort on files nobody in your audience will ever fetch. Here is what each one actually does, how they differ, and what else in the same family is worth knowing about.
## AGENTS.md: a README for coding agents
AGENTS.md is a markdown file you place in the root of a code repository. Its job is simple: tell an AI coding agent how to work in this project. The official description calls it "a README for agents". Where README.md explains a project to a human, AGENTS.md explains it to the agent that is about to build, test, and commit code.
A typical AGENTS.md covers the things a new engineer would ask on day one: how to install and build, how to run tests, the code style to follow, commit and pull request conventions, and any security notes. There is no required schema. It is ordinary markdown with whatever headings you find useful, so an agent simply parses the text you give it.
The format matters because of who backs it. AGENTS.md started as an OpenAI initiative in August 2025, and by December 2025 it moved under the Linux Foundation's Agentic AI Foundation, alongside Anthropic's Model Context Protocol. Backers include OpenAI, Anthropic, Google and AWS. By mid 2026 it is read natively by more than thirty tools, including Claude Code, OpenAI Codex, Cursor, GitHub Copilot, Devin, Aider, Gemini CLI, Windsurf and Amazon Q, and it appears in tens of thousands of public repositories. That breadth is the point: before AGENTS.md, every tool wanted its own file (one for Cursor, one for Claude, one for the next tool), and teams drowned in near duplicate instruction files. AGENTS.md is the consolidation.
In a monorepo you can place nested AGENTS.md files inside individual packages. Agents read the nearest file in the directory tree, so the closest one wins and each subproject can ship tailored instructions. OpenAI's own repository reportedly carries dozens of them.
The key thing for a website owner: AGENTS.md is not a website file. It is read from a code checkout, not fetched over HTTP by a search crawler. If you do not ship code that agents operate on, you do not need one.
## llms.txt: a curated map for language models
llms.txt is a markdown file at your website root, at /llms.txt. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024. The idea is to hand a language model a clean, curated map of your site so it can find the pages that matter without wading through navigation, cookie banners and templated boilerplate.
The format is defined and short:
- An H1 with the site or project name. This is the only required line.
- A blockquote with a short summary of what the site is.
- Optional free markdown (paragraphs, lists) with more detail.
- One or more H2 sections, each a list of links in the form [Title](url): optional note.
A section literally named ## Optional has special meaning: those links are secondary and can be skipped when a shorter context is needed. The spec is explicit that llms.txt is meant for inference time, that is, the moment a user is asking a model for help, rather than for training. It is a discovery and curation aid, not a data dump.
## llms-full.txt: the whole thing in one file
llms-full.txt uses the same structure as llms.txt but does not stop at links. It inlines the full text of your pages into a single large markdown file, so a model can absorb the substance without fetching every URL one at a time.
A small naming note that trips people up: the reference tooling that ships with the spec (the llms_txt2ctx generator) produces files called llms-ctx.txt and llms-ctx-full.txt, where the "full" version also expands the links marked Optional. In practice the community settled on the plainer name llms-full.txt, and that is what you will see on most sites that publish one.
Use llms-full.txt only when your content is genuinely text heavy and worth reading in full: product documentation, references, a guide library. For a marketing site or a blog, the curated llms.txt index alone is usually enough, and a multi megabyte full dump is just weight nobody asked for.
## AGENTS.md vs llms.txt vs llms-full.txt, side by side
Dimension
AGENTS.md
llms.txt
llms-full.txt
Who reads it
Coding agents (Claude Code, Codex, Cursor)
Answer engines (ChatGPT, Claude, Perplexity)
Same, when they want full text
Where it lives
Code repo root (and nested)
Website root, /llms.txt
Website root, /llms-full.txt
What it contains
How to build, test, and follow conventions
Curated links to key pages
The full page text inlined
Layer
Execution: how to work in the code
Retrieval: what your content says
Retrieval, full context
Format
Free markdown, no schema
Defined markdown (H1, blockquote, H2 link lists)
Same, plus inlined body text
Fetched how
From a code checkout
Over HTTP by a model or tool
Over HTTP
Do you need it
If agents touch your code
Optional, low cost, unproven for SEO
Only if content is text heavy
## The honest part: does any of this move rankings
AGENTS.md and llms.txt sit at very different levels of proof, and it is worth being blunt about which is which.
AGENTS.md demonstrably works, because you can watch it work. Open a repo in a supporting tool, add an AGENTS.md, and the agent follows it. The benefit is verifiable on your own machine.
llms.txt is a different story. Adoption is real but modest: analyses in 2026 put it on roughly ten percent of surveyed domains, with a few thousand sites in tracked technology profiles, including names like Anthropic, Cloudflare and Hugging Face. What is missing is evidence that the big answer engines actually fetch it to decide what to cite. Google has said it does not use llms.txt, and John Mueller has publicly likened its value to the old keywords meta tag. No major AI search provider has confirmed that publishing one changes whether or how you get cited.
So the honest read is this. Publish llms.txt if you run documentation an agent might navigate, or because it costs a couple of hours and does no harm. Do not publish it expecting a ranking or citation lift, and do not let it distract from the things that are proven to matter: content quality, crawlability, rendering, structured data and internal links. If you want the evidence based version of that argument, see llms.txt Explained: What It Is and Whether You Need One.
## What else is in this family, and what is actually missing
AGENTS.md and the llms files are two visible corners of a much larger set of machine readable signals. If the question is "what else should we be talking about," these are the pieces that matter, and most of them are better proven than llms.txt.
Signal
Layer
What it does
robots.txt
Access
The original crawler control file. Now also where you allow or block AI bots by user agent (GPTBot, ClaudeBot, Google-Extended, CCBot).
XML sitemaps
Discovery
The machine map of which URLs exist and when they changed. Proven, boring, essential.
Schema.org / JSON-LD
Meaning
Structured data that tells engines what an entity or page is. The most reliably consumed machine signal there is.
RSL
Licensing
Really Simple Licensing, an official standard since December 2025. Adds machine readable license terms to robots.txt: attribution, pay per crawl, pay per inference. Backed by the AP, Guardian, Yahoo and others.
MCP / WebMCP
Action
Lets agents do things on your site, not just read it. The tool calling layer of the agentic web.
ai.txt / TDMRep
Licensing
Earlier opt out signals for AI training and text and data mining. Narrower and less adopted than RSL, but still seen.
C2PA / Content Credentials
Provenance
Cryptographic "where did this come from" metadata for images and media. Growing as AI generated media spreads.
.well-known/
Discovery
The standard directory for machine files like security.txt. Where several of these signals are increasingly expected to live.
The single biggest gap in most "AI readiness" checklists is the licensing layer. Everyone argues about llms.txt while RSL quietly became the standard that decides whether AI companies can use your content and whether you get paid for it. If you publish anything of value, that is the conversation worth having. For the reader side of the same coin, our complete robots.txt reference covers how the AI bots are actually named and controlled, and WebMCP, Explained covers the action layer.
## Which file should you actually create
Skip the checklist theatre. Match the file to what you have.
If you have
Create
A code repository agents work in
AGENTS.md at the repo root (nested files per package in a monorepo)
A documentation or knowledge site you want quoted accurately
llms.txt, and llms-full.txt only if it is text heavy
Content you want to license or protect from AI use
robots.txt rules plus RSL license terms
Actions you want agents to perform
An MCP server or WebMCP endpoint
A brand you want recognized as an entity
Schema.org markup and clean entity HTML
For the wider picture of how these fit together, our overview of the machine readable web maps the whole stack, and how to become a cited source in AI answers covers the parts that actually influence citations.
## FAQ
Is AGENTS.md the same as CLAUDE.md or a .cursorrules file?
No, but it is the successor to them. Tool specific files like CLAUDE.md and .cursorrules each spoke to one agent. AGENTS.md is the cross tool standard that many of those same tools, including Claude Code, now read. Some tools still fall back to their own file if present, so you may keep both during the transition, but AGENTS.md is the one place they all agree to look.
Does llms.txt help me rank or get cited in AI answers?
There is no confirmed evidence that it does. Google has said it does not use llms.txt, and no major answer engine has confirmed fetching it to decide citations. Treat it as a low cost, low risk convenience for documentation and agent navigation, not as a ranking tactic.
Do I need both llms.txt and llms-full.txt?
Usually not. Publish llms.txt as a curated index first. Add llms-full.txt only when your content is text heavy documentation and you want a model to read the whole thing in one fetch. For a blog or marketing site, the full dump is unnecessary weight.
Where does AGENTS.md go, and should I put one on my website root?
It goes in the root of a code repository, with optional nested files inside subprojects where the nearest file wins. It is read from a code checkout, not fetched by a search crawler, so putting one at your public website root serves no reader. Answer engines look for llms.txt there, not AGENTS.md.
What is the real difference between llms.txt and llms-full.txt?
llms.txt is a curated list of links to your key pages. llms-full.txt inlines the actual text of those pages into one file. The first is a map, the second is the territory. Same format, very different file size and purpose.
If llms.txt is unproven, what should I spend the time on instead?
The proven signals: correct robots.txt and sitemaps, server rendered HTML that crawlers can read, Schema.org structured data, strong internal linking, and genuine content quality. If AI licensing matters to your business, look at RSL. Those move the needle in ways llms.txt has not been shown to.
---
# The Machine-Readable Web: What robots.txt, llms.txt, Schema, WebMCP and Entity Signals Actually Do in 2026
Source: https://seoprocheck.com/research/ai-search/machine-readable-web-standards/
There has never been more advice about "files you must add so AI can read your site." robots.txt, llms.txt, llms-full.txt, Schema, WebMCP, AI opt-out tokens: every week someone sells one of them as the thing standing between you and visibility. Most of that is noise. This is the honest map of every standard that governs how crawlers, search engines, and AI agents read your site in 2026, what each one actually does, how widely it is really used, and which ones move the needle versus which ones are optional or still experimental. No hype, no fear-selling.
Use this as the index. Each standard gets a plain-English summary here and a link to the full reference when you want the depth.
Four layers of the machine-readable web
1. Access · who is allowed to crawl robots.txt, AI opt-out tokens
2. Discovery · what exists and where XML sitemaps, llms.txt
3. Meaning · what your content means and who you are Schema.org, entity signals
4. Action · what an agent can do on your site WebMCP
Most "must-have AI files" live in layers 1 and 2. The durable advantage lives in layer 3.
🗺️ TL;DR
The standards that reliably matter are the boring, well-supported ones: robots.txt for access, meta robots for indexing, Schema and entity signals for meaning. llms.txt is on roughly one in ten sites and the major AI crawlers largely ignore it for content, so treat it as optional and mostly agent-facing. WebMCP is genuinely interesting but in early browser preview as of 2026, so it is a watch-and-experiment item, not a requirement. Optimise the rendered HTML and your entity clarity first. Everything else is a layer on top.
~10%
of sites have llms.txt, and AI crawlers largely skip it for content (SE Ranking, 300k domains)
Confirmed
Microsoft stated at SMX 2025 that schema markup helps its LLMs understand content
Feb 2026
WebMCP entered early browser preview (Chrome 145), so it is emerging, not established
The complete library
Every guide in this series, in one place. Start anywhere.
Access: who is allowed to read you
- The Complete robots.txt Reference: Precedence, Wildcards, AI Bots & Real-World Receipts
- The AI Crawler Map: Every Bot Reading Your Site in 2026
- Which AI Bots Are You Actually Blocking? (GPTBot, ClaudeBot, Perplexity & More)
- Is Your Cloudflare or WAF Secretly Blocking GPTBot?
Discovery: telling AI what exists
- llms.txt Explained: What It Is and Whether You Need One
Meaning: what your content means and who you are
- The Forgotten HTML: What AI Crawlers Really See on Your Expensive Website
- SSR vs CSR: Why Rendering Decides Whether AI Can Read Your Site
- Structuring Content So AI Can Actually Extract It
- Entity SEO: Helping Search Engines and AI Understand Who You Are
- Entity HTML: How to Become a Machine-Readable Brand (Knowledge Graph, sameAs & the Entitymap)
- FAQ Schema: Why You Still Keep It After Google Retired the Rich Result
- Schema Markup for AI Search: FAQPage, Tables, and Structured Data
Action: what an agent can do on your site
- WebMCP, Explained: How Your Site Hands AI Agents a Menu of Actions
AI engines and getting cited
- Google AI Overviews: How to Optimize for AI Overview Citations
- How to Become a Cited Source in AI Answers
- ChatGPT SEO: How to Optimize for ChatGPT and SearchGPT Citations
- Claude SEO: How to Get Cited in Claude AI
- Perplexity SEO: How to Get Cited in Perplexity AI
- Brand Mentions vs Backlinks: What Matters More for AI Visibility
- How AI Search Tools Source Information from Organic Search Results
- JavaScript and AI Search: Why Server-Side Rendering Matters for GEO
More in this series
- SEO vs AEO vs GEO: the differences
- Is your site agent-ready? The Lighthouse agentic audit
## 📊 Every standard at a glance
Standard
What it does
Audience
Reliability
Adoption
Verdict
robots.txt
Controls which paths a crawler may request
All crawlers
Voluntary, widely honoured
Universal
Essential
Meta robots / X-Robots-Tag
Controls indexing (the real noindex)
Search engines
Authoritative
Universal
Essential
XML sitemap
Lists URLs for discovery
Search crawlers
Well supported
Universal
Useful
Schema.org structured data
States what content means
Search and AI
Used; some rich results restricted
Widespread
High value
Entity signals (sameAs, KG)
Defines who you are as a thing
Search and AI
Foundational, confidence-based
Growing
High value, rising
AI opt-out tokens (Google-Extended, etc.)
Opt out of AI model training
Specific AI vendors
Honoured by issuer
As needed
Use if opting out
llms.txt
Proposed index of key content for AI
AI agents, not search
Crawlers largely skip it; Google Search says not needed
~10%
Optional
llms-full.txt
Full content concatenated for LLMs
AI agents
Same caveats, rarer
Minimal
Optional
WebMCP
Exposes site actions as callable tools for agents
Browser AI agents
Early preview
Minimal
Emerging, watch
## 🔓 Layer 1: access (robots.txt and AI tokens)
What it does: robots.txt tells compliant crawlers which paths they may request. AI opt-out tokens like Google-Extended and Applebot-Extended let you decline model training without leaving search. Reliability: high among mainstream crawlers, though compliance is voluntary and some AI bots have been observed ignoring it. The honest take: this is genuinely essential, and the most common mistake is blocking the wrong thing, for example removing yourself from AI answers by accident. Full detail in the complete robots.txt reference and the per-bot breakdown in the AI Crawler Map.
## 🗂️ Layer 2: discovery (sitemaps and llms.txt)
XML sitemaps list your URLs so search engines can find them. Boring, universal, worth having. llms.txt is the controversial one: a proposed Markdown file that points AI systems at your key content. Reliability and adoption: here is the no-BS part. As of 2026 it sits on roughly one in ten sites, the major AI crawlers overwhelmingly fetch your HTML directly rather than the file, and Google Search states plainly that it is not needed for AI Overviews or AI Mode. At the same time, Chrome's Lighthouse now checks for it under an agentic-browsing audit, and Anthropic and OpenAI reference it for their agent tooling. So yes: one part of Google now checks for a file another part of Google says you do not need. That is a little WTF, and if you find it confusing, the confusion is on them, not you. The honest take: it is not "robots.txt for AI," it cannot block anything, and it will not move search visibility. It may help agent and developer-documentation use cases. Treat it as optional. The full, receipt-backed story is in llms.txt explained.
## 🧠 Layer 3: meaning (Schema and entity signals)
This is where the durable advantage lives, and where most sites underinvest. Schema.org structured data states what your content is in a vocabulary machines share. Microsoft confirmed at SMX 2025 that schema helps its LLMs understand content, and structured data underpins rich results and answer eligibility, even though Google restricted some rich results such as FAQ to government and health sites in 2023. Entity signals go further: they define your brand as a thing in the knowledge graph through Organization schema, sameAs corroboration, and a clear entity home. The honest take: being a confident, well-described entity is increasingly the prerequisite for being cited at all. Start here. Depth lives in Entity HTML, the rendering angle in The Forgotten HTML, and the schema-specific case in FAQ schema, why you still keep it.
## 🤝 Layer 4: action (WebMCP)
What it does: WebMCP, the Web Model Context Protocol, is a browser-native API built by Google and Microsoft engineers under the W3C. It lets a site publish a "tool contract," a structured list of actions an AI agent can call directly, so the agent books, searches, or filters by calling your functions instead of guessing at buttons. Reliability and adoption: it entered early preview in February 2026 in Chrome 145, so it is experimental and barely deployed. The honest take: this is the most forward-looking item on the list and worth understanding now, because the agentic web is where the browser vendors are clearly heading. It is not something you are behind on today. Watch it, prototype if you are technical, and do not let anyone tell you it is make-or-break in 2026. Read the full WebMCP explainer, with working code, here.
## ✅ The no-BS verdict: what to actually do
If you do nothing else, do these, in order:
- Get access right. A correct robots.txt that lets the engines you want in, including AI answer crawlers, and uses meta robots for real indexing control.
- Be readable. Make sure your content survives rendering, because every layer above is wasted if the crawler sees an empty shell.
- Be a clear entity. Organization schema, sameAs, a clean entity home. This is the highest-leverage, most under-done work.
- Add structured data where it is genuine. Schema for real content, never faked to chase a feature.
- Treat llms.txt and WebMCP as optional and experimental. Add llms.txt if agent or developer audiences matter to you. Watch WebMCP. Neither is a search-visibility lever today.
The pattern across all of it: the durable wins are the unglamorous, well-supported standards plus a clear identity. The new files get the headlines, but they sit on top of fundamentals that most sites still have not finished. If there is any real BS in this space, it is the constant pressure to adopt a shiny new file before the boring, proven basics are even in place.
One caveat worth stating plainly: this space moves fast, and the vendors themselves say the standards for the agentic web are still emerging. Nothing here is settled or "dead." Treat every verdict as current as of 2026 and re-check the primary sources before a major decision.
Not sure which of these your site actually needs?
An advanced audit cuts through the hype: it checks your access, rendering, schema, entity signals, and AI readiness, and hands you a prioritised list of what to fix, with the receipts.
Request an advanced SEO and AI-visibility audit →
📚 The deep dives (each standard in full)
- The complete robots.txt reference
- The AI Crawler Map: every bot reading your site
- llms.txt explained
- Entity HTML: becoming a machine-readable brand
- The Forgotten HTML: what AI crawlers really see
- FAQ schema: why you still keep it
- Schema markup for AI search
- Entity SEO basics
---
# The AI Crawler Map: Every Bot Reading Your Site in 2026
Source: https://seoprocheck.com/research/ai-search/ai-crawler-map/
Open your server logs for a single day and you will find a parade of robots you never invited: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bytespider, and a dozen more. Each one wants something different, each one reacts differently when you block it, and most site owners treat all of them as a single switch labelled "AI." That is exactly how good pages quietly vanish from AI answers while their owners congratulate themselves on "blocking the scrapers."
This is the map I wish every client had before they touched their robots.txt. It covers every major AI user-agent crawling the open web in 2026, what each one actually does, and the specific thing that breaks when you say no.
How a page becomes an AI citation, or doesn't
A user asks an AI engine
→
The engine sends a fetch bot
→
robots.txt + server respond
→
Page rendered, text extracted
✓ Allowed → you're cited in the answer
✕ Blocked / 403 → absent from the answer
Block the wrong user-agent and you remove yourself from the answer, the page still ranks in classic search.
🗺️ TL;DR
There is no single "AI bot." There are roughly twenty named user-agents doing three different jobs: training models, indexing pages to answer questions, and fetching a page live for one user. Blocking a training crawler costs you almost no traffic. Blocking a search/answer crawler removes you from AI citations entirely. Knowing which is which is the whole game.
~20
named AI user-agents crawl the open web today
3 jobs
every bot does exactly one: train, index-to-answer, or fetch-for-a-user
1 line
in robots.txt can erase you from AI answers while search stays untouched
## 💡 The only mental model you need: three jobs
Every AI crawler on the planet is doing one of three jobs. Sort each bot into the right bucket and the entire "should I block this?" question answers itself.
- Training crawlers harvest text to teach a model. They visit once in a while, take a copy, and leave. Blocking them protects your content from future model training but costs you essentially zero live visibility, the model was never going to link back to you anyway.
- Search & answer crawlers build the live index an assistant searches when a user asks a question. These are the bots that cite you. Block one and you remove yourself from that engine's answers, the AI equivalent of deleting yourself from Google's index.
- User-fetch agents retrieve a single page in real time because a specific human asked the assistant to read that URL. Block them and "summarise this link" requests to your pages simply fail.
The expensive mistake is treating all three as the same threat. People who paste a "block all AI" snippet to stop model training also quietly delete themselves from ChatGPT Search, Claude, and Perplexity, the surfaces actually sending qualified readers in 2026.
## 🤖 The map: every major AI bot in 2026
Grouped by operator. "Honors robots.txt" reflects each operator's published policy; the asterisks matter, and I unpack them in the limitations section.
User-agent
Operator
Job
robots.txt
What blocking costs you
GPTBot
OpenAI
Train
Yes
Your text won't train GPT models
OAI-SearchBot
OpenAI
Index → answer
Yes
You disappear from ChatGPT Search citations
ChatGPT-User
OpenAI
User fetch
Yes
"Open this link" requests fail
ClaudeBot
Anthropic
Train
Yes
Your text won't train Claude
Claude-SearchBot
Anthropic
Index → answer
Yes
You're absent from Claude's web citations
Claude-User
Anthropic
User fetch
Yes
User-initiated fetches fail
PerplexityBot
Perplexity
Index → answer
Yes*
You vanish from Perplexity citations
Perplexity-User
Perplexity
User fetch
No*
Usually still fetches (user-initiated)
Googlebot
Google
Index → Search + AI Overviews
Yes
You leave Google Search entirely, AI Overviews can't be blocked alone
Google-Extended
Google
Token: Gemini training
Token
Opt out of Gemini training/grounding; Search unaffected
Bingbot
Microsoft
Index → Search + Copilot
Yes
You leave Bing and Copilot answers
Applebot
Apple
Index → Siri/Spotlight
Yes
Absent from Siri and Spotlight results
Applebot-Extended
Apple
Token: Apple Intelligence training
Token
Opt out of Apple AI training; Search unaffected
Amazonbot
Amazon
Index → Alexa answers
Yes
Absent from Alexa answers
Meta-ExternalAgent
Meta
Train
Yes
Your text won't train Meta AI
CCBot
Common Crawl
Train (open dataset)
Yes
Less inclusion in a dataset many labs train on
Bytespider
ByteDance
Train
No*
May keep crawling regardless
DuckAssistBot
DuckDuckGo
Index → DuckAssist
Yes
Absent from DuckAssist answers
A "Token" entry is a directive in robots.txt, not a crawler with its own user-agent, Google-Extended and Applebot-Extended govern training only and never affect how you rank in their search products.
## 🚦 What actually breaks when you block each type
Here is the part the "block all AI" crowd never thinks through. The three jobs carry wildly different costs.
Block a training crawler (GPTBot, ClaudeBot, CCBot, Bytespider) → low cost
You keep your work out of future model training. You lose almost no live traffic, because training crawlers don't send readers, they don't even cite. This is the safe, defensible block if your concern is "don't train on my IP without asking." Pair it with the -Extended tokens to cover Gemini and Apple Intelligence.
Block a search/answer crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) → high cost
This is self-sabotage dressed as caution. These bots build the index the assistant searches at answer time. Disallow them and you are simply not a candidate to be cited, your competitor who left them allowed gets the mention and the click. If you want to appear in AI answers, these must stay open.
Block a user-fetch agent (ChatGPT-User, Claude-User) → friction cost
These fire when a real person pastes your URL and asks the assistant to read it. Block them and that person gets "I couldn't access that page." It rarely moves rankings, but it is a bad look at the exact moment someone is trying to engage with your content directly.
## 🔧 robots.txt recipes you can paste today
Multiple User-agent lines can share one Disallow block, that is valid syntax, not a hack.
Recipe 1, Stay in AI answers, opt out of model training. The configuration most publishers actually want:
# Let the answer engines cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Recipe 2, Block everything AI, accept the cost. Legitimate for paywalled or licensing-sensitive sites, as long as you know you're leaving every AI answer surface:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Notice what is not on that list: Googlebot and Bingbot. You cannot opt out of Google AI Overviews or Bing Copilot without also leaving classic search, those answer features are fed by the same index that ranks you. Google-Extended only governs Gemini training, not AI Overviews. Anyone who tells you otherwise is selling something.
## 🚧 What this map does not mean
Honesty is the difference between a field guide and a fairy tale. Five caveats you should hold onto:
- robots.txt is a request, not a firewall. Compliance is voluntary. Independent monitoring (notably by Cloudflare in 2025) reported that some operators, Perplexity and ByteDance's Bytespider among the most cited, fetched pages that disallowed them. The asterisks in the table flag exactly those disputed cases. If you need a hard block, enforce it at the edge with WAF rules and verified IP ranges, not robots.txt alone.
- User-agents can be spoofed. Anyone can send a request claiming to be ClaudeBot. The real operators publish IP ranges (and some support reverse-DNS verification) so you can confirm a bot is genuine before trusting, or rate-limiting, it.
- Allowing a crawler is necessary, not sufficient. Letting OAI-SearchBot in does not guarantee a citation. It makes you eligible. Whether you're actually quoted still depends on relevance, clarity, and whether your content survives rendering, which is a separate problem entirely.
- The list changes. These operators add and rename agents regularly (the -SearchBot variants are relatively new). Treat any AI-crawler list, including this one, as a snapshot, re-check the official docs before a migration.
- Blocking is reversible, lost ground is slower. If you disallow a search crawler, re-allowing it doesn't instantly restore your presence, the engine has to re-crawl and re-index on its own schedule.
## ✅ Audit your own site in ten minutes
Don't take my table on faith, go look at what's actually happening on your server. The replication steps:
- Read your live robots.txt. Visit yourdomain.com/robots.txt and find every User-agent line. Map each one to the table above and ask: am I blocking a training bot (fine) or an answer bot (probably a mistake)?
- Grep your access logs for one week. Filter for the user-agents in the table. You'll see who actually visits, how often, and which ones you're serving 200 versus 403.
- Check for accidental edge blocks. Many sites block AI bots at Cloudflare or a WAF without realising it, the "Block AI Scrapers" toggle is one click. Confirm your answer crawlers aren't being stopped before they reach robots.txt.
- Test a live fetch. Ask ChatGPT or Claude to open one of your URLs. If it returns "couldn't access," your user-fetch agents are blocked somewhere in the stack.
- Decide on purpose. Write down, per operator, whether you want to be trained on, indexed for answers, or fetched live, then make robots.txt say exactly that. Default-by-accident is how good pages disappear.
Not sure which bots your site is quietly turning away?
An AI-visibility audit maps your real crawler access, robots.txt, edge rules, and rendering, against the answer engines that matter, and tells you exactly what to change.
Request an advanced SEO & AI-visibility audit →
📡 AI Crawler Visibility series
- The Forgotten HTML (pillar)
- Which AI Bots Are You Actually Blocking?
- SSR vs CSR for AI Crawlers
- llms.txt vs llms-full.txt: What AI Actually Reads
---
# The Forgotten HTML: What AI Crawlers Really See on Your Expensive Website
Source: https://seoprocheck.com/research/ai-search/the-forgotten-html/
You can tell when a website cost a fortune. The type is set just right, the animations are buttery, every pixel is in its place. Then you fetch that same page the way an AI crawler does , raw HTML, no JavaScript, no negotiation , and half the time you get nothing. An empty shell. A loading spinner. Or a door slammed in your face with a 403.
This is the forgotten HTML: the version of your site that machines actually receive, which almost nobody on the team has ever looked at. The polish is for human eyes. The bots get something else entirely , and the bots are increasingly who decide whether you exist in an AI answer.
## The uncomfortable truth
Most AI crawlers do not run JavaScript. Googlebot renders (slowly, expensively); almost nothing else does. In its analysis of over 500 million GPTBot fetches, Vercel found zero evidence of JavaScript execution (Vercel: The rise of the AI crawler). GPTBot, ClaudeBot and PerplexityBot grab the raw HTML and move on. So if your content only appears after JavaScript runs in a browser , or if your server never lets the bot in at all , you are invisible to the systems answering billions of questions a month.
## Think I'm exaggerating? Here are the receipts.
I am not asking you to trust me. Open a terminal and run this against any site:
curl -sI -A "GPTBot/1.0" https://example.com/ | head -1
# and to count what the bot actually receives:
curl -s -A "GPTBot/1.0" https://example.com/ | sed 's/<[^>]*>/ /g' | wc -w
I ran exactly that against a stack of polished, well-funded sites in June 2026. Five of them never even let the crawler through the door , they answered GPTBot with an HTTP 403:
Site
What GPTBot got
Airbnb
403 , blocked
Coinbase
403 , blocked
Medium
403 , blocked
Product Hunt
403 , blocked
Udemy
403 , blocked
Crunchbase
3,347 words ✅
Robinhood
1,677 words ✅
Linear
1,572 words ✅
Wikipedia
6,657 words ✅
Boom. Run the command yourself , you will get the same thing. Now, some of those blocks are deliberate (plenty of publishers block AI crawlers on purpose, and that is a perfectly valid choice). The problem is that most teams have no idea which camp they are in. A Cloudflare bot rule or an over-zealous WAF will quietly 403 GPTBot, ClaudeBot and PerplexityBot while everyone congratulates themselves on the new design. You meant to win AI visibility; you accidentally bricked it.
## What's actually happening
Expensive sites go invisible to AI in three ways, and you cannot see any of them from the front-end:
- The empty shell. A client-side-rendered build ships near-zero content in the raw HTML; everything is assembled by JavaScript the bot never runs.
- The missing money content. The shell renders fine, the word count looks healthy , but prices, reviews, specs, and anything behind a tab or a "load more" button are injected later. The bot sees your product page but not its price.
- The locked door. The server returns a 403 (or a CAPTCHA, or a JS challenge) to the crawler's user agent. Content is irrelevant when the bot never gets in.
## I've spent a lot of time in this exact mess
I will be honest: a good chunk of what I know here came from cleaning it up. Across several projects I have sat between development, content, and UX teams untangling the same gap , content the CMS swore was published, that looked perfect in the browser, that Search Console even showed rendered, and that the AI crawlers simply never received. It is almost never one person's fault. The dev team optimized for a slick client-side experience, the content team wrote great copy, the UX team made it beautiful , and nobody owned the question of what the raw HTML contained. That question is the whole game now.
## Check your own site in 30 seconds
### The bookmarklet (drag it to your bookmarks bar)
New bookmark, paste this as the URL, click it on any of your pages. It compares the raw HTML a bot receives to what you see and tells you the percentage of content the bots actually get:
javascript:(async()=>{try{const r=await fetch(location.href,{cache:'no-store',credentials:'omit'});const raw=await r.text();const d=new DOMParser().parseFromString(raw,'text/html');d.querySelectorAll('script,style,noscript,template').forEach(e=>e.remove());const rawT=(d.body?d.body.textContent:'').replace(/s+/g,' ').trim();const renT=document.body.innerText.replace(/s+/g,' ').trim();const rw=rawT?rawT.split(' ').length:0;const rn=renT?renT.split(' ').length:0;const p=rn?Math.round(100*rw/rn):0;alert('AI-CRAWLER VISIBILITY CHECKn'+location.href+'nnWords in RAW HTML (what GPTBot/ClaudeBot/PerplexityBot get): '+rw+'nWords after JS renders (what you see): '+rn+'nnNon-rendering AI bots see ~'+p+'% of your content.n'+(p<70?'WARNING: a large share of your content is injected by JavaScript and is INVISIBLE to AI crawlers.':'OK: most of your content is in the raw HTML.'));}catch(e){alert('Could not fetch the raw HTML (cross-origin or CSP). Run it on your own site.n'+e);}})();
### The rest of the toolkit
- View Source vs Inspect: Ctrl/Cmd+U (what the bot gets) vs DevTools → Elements (what you see). Content in one and not the other is your answer.
- View Rendered Source , Raw / Rendered / Difference, line by line.
- Quick Javascript Switcher , turn JS off, reload; blank page, blank bot.
- Wappalyzer , identifies the framework so you know your risk.
- The curl receipts above , check each bot you care about: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended.
## How to fix it
- Render on the server , SSR (Next.js, Nuxt, SvelteKit) or static generation (SSG). Real HTML, immediately, for bots and humans.
- Put the money content in the HTML , prices, reviews, specs and key copy in the initial payload. Let JavaScript enhance, not create.
- Unlock the door on purpose. Audit robots.txt and your WAF/CDN rules so the assistants you want citing you are not silently 403'd. If you block AI crawlers, make it a decision, not an accident.
- Re-check with the curl commands and the bookmarklet until the raw HTML contains your content and the bots get a 200.
## The takeaway
A site can look like a million dollars and hand a machine a blank page or a closed door , and the prettier the build, the easier it is to never notice. Looks are for humans. The forgotten HTML is for machines. Go read yours.
## Related reading
- Google AI Overviews: How to Optimize for AI Overview Citations
- AI Crawler Comparison: GPTBot, ClaudeBot, PerplexityBot
- ChatGPT SEO · Perplexity SEO · Claude SEO · GEO FAQ
## Want to know what AI crawlers actually see on your site?
I run advanced technical audits that check rendering, retrievability, and bot access , the things that decide whether you show up in search and AI answers at all. See how an advanced SEO audit works →
📡 The AI Crawler Visibility series
- The Forgotten HTML, what AI crawlers really see
- Which AI bots are you actually blocking? (GPTBot, ClaudeBot, Perplexity & more)
- SSR vs CSR, why rendering decides if AI can read you
- AI Crawler Comparison · Google AI Overviews · GEO FAQ
---
# SSR vs CSR: Why Rendering Decides Whether AI Can Read Your Site
Source: https://seoprocheck.com/research/ai-search/ssr-vs-csr-ai-crawlers/
Two sites can look identical in a browser and be worlds apart to a machine. The difference is where the HTML gets built , on the server before it is sent, or in the browser after JavaScript runs. For AI crawlers, which mostly do not run JavaScript, that single choice decides whether your content exists at all.
## The four words you need
- CSR (client-side rendering): the server sends a near-empty shell plus JavaScript; the browser builds the page. Great for app-like interactivity, terrible for non-rendering bots.
- SSR (server-side rendering): the server builds the full HTML per request and sends it ready to read. Bots and humans both get content immediately.
- SSG (static generation): pages are pre-built into HTML at deploy time. Fastest and most crawler-proof.
- Hydration: ship real HTML, then let JavaScript "wake it up" with interactivity. The content is in the payload; the JS only enhances it.
## Why AI crawlers make this non-negotiable
Googlebot renders JavaScript (slowly, at a cost). Almost nothing else does. Vercel's analysis of over 500 million GPTBot fetches found zero JavaScript execution (Vercel); roughly 69% of AI crawlers can't run JS at all. So with CSR, GPTBot, ClaudeBot and PerplexityBot receive your empty shell and leave. With SSR or SSG, they get the whole page.
## The receipts
I fetched a set of well-known sites with a GPTBot user agent and counted the body words in the raw HTML , before any JavaScript. The ones serving content server-side hand the bot real text:
Site
Words in raw HTML (bot's view)
Wikipedia (reference)
6,657
Crunchbase
3,347
Robinhood
1,677
Linear
1,572
react.dev
1,439
Vercel
1,102
Notion
702
That is what "done right" looks like: even sophisticated single-page-app companies render their marketing and content pages on the server so machines can read them. A pure-CSR build, by contrast, would show a near-empty raw HTML and a full rendered page , the gap is exactly the content the bots never see.
## Which one do you have? Check in 30 seconds
- View Source (Ctrl/Cmd+U): if your headline and body copy are not in there but appear in DevTools → Elements, you are client-rendering it.
- The bookmarklet from The Forgotten HTML reports the exact percentage of your content that reaches a non-rendering bot.
- Wappalyzer tells you the framework; Quick Javascript Switcher lets you disable JS and watch what survives.
## How to fix it
- Render on the server. Next.js, Nuxt, SvelteKit and Angular Universal all do SSR; frameworks like Astro and 11ty do SSG. Modern frameworks make this the default , you often just have to not opt out of it.
- Static-generate anything that doesn't change per request (blogs, docs, marketing).
- Keep the money content in the HTML. Prices, reviews, specs and key copy belong in the initial payload , not injected after a tab click or "load more."
- Prerender as a stopgap if a full migration isn't feasible: serve bots a fully-rendered snapshot.
- Re-test with View Source and the bookmarklet until the raw HTML contains your content.
And once the content is renderable, make sure the bots are actually allowed in , see which AI bots you might be blocking.
📡 The AI Crawler Visibility series
- The Forgotten HTML, what AI crawlers really see
- Which AI bots are you actually blocking? (GPTBot, ClaudeBot, Perplexity & more)
- SSR vs CSR, why rendering decides if AI can read you
- AI Crawler Comparison · Google AI Overviews · GEO FAQ
## Not sure which bots can see (or are blocked from) your site?
An advanced technical audit checks rendering, retrievability, and bot access end to end. See how an advanced SEO audit works →
---
# Entity HTML: How to Become a Machine-Readable Brand (Knowledge Graph, sameAs & the Entitymap)
Source: https://seoprocheck.com/research/ai-search/entity-html-machine-readable-brand/
The web spent twenty years optimising strings, keywords, phrases, exact-match text. Search engines and AI assistants now think in things: people, companies, products, places, and the relationships between them. If a machine can't resolve your brand to a confident, well-connected entity, you are a string it has to guess about, and in an AI answer, guessing means citing someone else.
This is the complete reference for entity HTML: how to make your brand a machine-readable thing, where those signals must physically live in your markup, and how to build the kind of entity confidence that earns Knowledge Panels and AI citations. It's the structural companion to The Forgotten HTML, because none of this matters if it isn't in the HTML the crawler actually renders.
🗺️ TL;DR
An entity is a thing a machine can identify and connect, not a keyword. Google's Knowledge Graph is the map of those things, and modern AI answers lean on it to decide who is credible enough to cite. You become an entity by giving machines an unambiguous identity (Organization/Person schema), corroborating it across independent sources that agree (sameAs → Wikidata, Wikipedia, LinkedIn, Crunchbase), and anchoring it to one canonical entity home. Schema alone doesn't create an entity, corroboration does.
500B+
facts on 5B+ entities in Google's Knowledge Graph, and Gemini is trained on it
0.66 vs 0.22
one 2026 analysis: brand-entity mentions correlate with AI-Overview visibility far more than backlinks (correlation, not proof)
Wikidata
the single highest-leverage sameAs target, a primary feed into the Knowledge Graph
## 🧩 Strings vs things: what an entity actually is
Google announced the shift in 2012 with the Knowledge Graph and the phrase "things, not strings." A string is text, the characters a-p-p-l-e. A thing is a uniquely identified concept, Apple the company, distinct from apple the fruit, with attributes (founded 1976, HQ Cupertino) and relationships (founded by Steve Jobs). Each entity has a stable internal ID, and the Knowledge Graph stores what's true about it and how it connects to everything else.
Why this matters more every quarter: being a confident entity is now the prerequisite for the surfaces that are eating clicks. Knowledge Panels, AI Overviews, AI Mode, and Gemini answers all draw on entity understanding to choose who to feature. If Google isn't sure who you are, you are not a candidate, no amount of keyword optimisation fixes an identity problem.
## 🗺️ The entitymap: a useful way to picture it
Waikay's EntityMap case study (April 2026) frames this cleanly. In their words, "an entitymap tells AI models what your brand means, what it does, and how its parts relate to each other." It's the same idea as a Knowledge-Graph node for your brand, made deliberate: a structured description of your entity and its relationships, published where machines read it.
Their reported result is a useful illustration of the mechanism, credited, with the caveat that it's a single vendor case study, not independent research: after deploying an entitymap, they observed "AI Knowledge Scores improved by up to 26 points in 48 hours," and the entitymap was "cited 2.2, 3.0× more often than the site's own About page." Treat the exact numbers as directional; the direction, clearer entity, more citations, is what's worth internalising.
## ⚙️ How entity confidence is actually built
Machines don't trust a single self-declaration. Entity confidence comes from independent sources agreeing about who you are. The mechanism:
How a brand becomes a confident entity
DeclareEntity home + Organization/Person schema state who you are
→
CorroboratesameAs links to Wikidata, Wikipedia, LinkedIn, Crunchbase
→
AgreeIndependent sources state the same facts; confidence rises
→
SurfaceKnowledge Panel + eligibility to be cited in AI answers
The more independent sources agree on your details, the higher your entity confidence, and confidence is what unlocks citation.
## 🔩 The building blocks (where the signals live in your HTML)
- The entity home. One canonical URL that is your entity, usually your homepage or /about. Everything else points back to it. Pick one and be consistent; a split identity confuses the graph.
- Organization (or Person) schema. JSON-LD that states your name, logo, URL, and, critically, sameAs. This is the machine-readable declaration of identity.
- sameAs corroboration. An array of URLs to other authoritative profiles of the same entity: Wikidata, Wikipedia, LinkedIn, Crunchbase, official social. Wikidata is the highest-leverage target because it feeds the Knowledge Graph directly. More agreeing identifiers = higher confidence.
- Consistent naming (NAP). Identical name, address, and phone everywhere. Inconsistency reads as two different entities.
- Entity linking in content. When you mention your brand, people, or products, link them to their entity home; reference well-known entities (and their Wikipedia pages) to place yourself in the right neighbourhood of the graph.
- Author entities. Real Person schema for authors, with their own sameAs, the backbone of E-E-A-T and a signal AI Mode uses when picking expert sources.
## 🧱 A correct Organization + sameAs block
Every URL in sameAs must genuinely describe the same entity and should, ideally, link back to you, corroboration is mutual.
## 🗺️ What an entitymap actually looks like
This stays abstract until you see one, so here is a worked example for a fictional company, "Acme Analytics." An entitymap is just a structured statement of the entity, its core attributes, the relationships it has to other entities, and the independent sources that confirm all of it.
Acme Analytics · Organization
Founded 2017
HQ Austin, TX
Industry: SaaS analytics
Relationships: Founder: Jane Doe (Person) · Flagship product: AcmeInsights (SoftwareApplication)
sameAs (corroboration): Wikidata Q123456 · Wikipedia · LinkedIn · Crunchbase
That same map, expressed as the JSON-LD you publish on your entity home, with the founder modelled as a linked Person entity:
### How to create your entitymap, step by step
- Name the entity precisely. The exact, consistent name you want machines to recognise. One name, used everywhere.
- List the stable attributes. Founding date, location, industry, and a one-line statement of what you actually do. Facts, not adjectives.
- Map the relationships. Founders, key people, parent or subsidiary companies, and flagship products. Each important one should ideally be its own entity with its own sameAs.
- Gather corroborating sources. Create or correct your Wikidata item, claim your LinkedIn and Crunchbase profiles, and make sure they all state the same facts. Agreement is what builds confidence.
- Express it as JSON-LD on your entity home. Organization (or Person) with sameAs and linked entities, published on the one canonical URL that defines you.
- Keep it consistent everywhere. Same name, same facts, in your footer, your About page, your profiles, and your markup. Contradictions lower confidence.
### What to add, and what to leave out
✓ Add
- Real, verifiable attributes (dates, location, industry)
- sameAs to profiles that are genuinely you and link back
- Founders and key people as linked Person entities
- One canonical entity home that everything points to
- Facts that independent, trusted sources also state
✕ Leave out
- Aspirational or unverifiable claims
- sameAs to profiles that are not really you, or that contradict your data
- Marketing adjectives (schema describes facts, not vibes)
- Duplicate or conflicting entity homes
- Padding the map with entities you have no real relationship to
## 🎯 The priority order most brands should follow
Entity work has a sequence. Doing it out of order wastes effort:
- Entity home, pick and harden the one canonical URL that defines you.
- Wikidata entry, create or correct it; it's the most direct Knowledge-Graph input and is openly editable (with sources).
- sameAs schema, wire the corroboration links from your entity home.
- Entity linking, connect mentions across your content to entity homes.
- Mentions & PR, earn references on sources Google already trusts. This is where the brand-mention signal compounds.
## 🚧 What entity HTML does not do
Measured expectations, because this is a confidence game, not a switch:
- Schema alone won't manufacture an entity. Declaring Organization with no corroboration is a claim with no witnesses. Without independent agreement, confidence stays low.
- It won't conjure a Knowledge Panel on demand. Panels appear when Google is confident enough, markup helps, but notability and corroboration drive it, and it takes time.
- Faking sameAs backfires. Linking to profiles that aren't really you, or that contradict your other data, lowers confidence instead of raising it.
- It's not instant, and it's not a ranking cheat. Entity confidence accrues as sources align over weeks and months. Treat it as foundational infrastructure, not a campaign.
## 🔎 How to check your entity
- Search your brand and see whether a Knowledge Panel appears, that's Google showing you it has a confident entity.
- Look yourself up on Wikidata. No entry, or a thin/incorrect one, is a direct gap in the Knowledge Graph's primary feed.
- Validate your schema with the Schema.org validator and Google's Rich Results Test, confirm Organization/Person and sameAs parse.
- Ask an AI assistant who you are. If it's vague, contradictory, or wrong, your entity isn't resolving cleanly, exactly the gap this work closes.
## ❓ Entity SEO FAQ
What is an entity home?
The one canonical URL that defines your entity, usually your homepage or About page. Everything else (schema, links, profiles) should point back to it.
Do I need a Wikipedia page to be an entity?
No. Wikipedia helps but requires notability. Wikidata is more accessible and is a more direct Knowledge-Graph feed, start there.
What does sameAs actually do?
It tells machines "these other profiles are the same entity as me," letting them cross-check and merge facts. Agreement across those sources raises your entity confidence.
Will Organization schema give me a Knowledge Panel?
Not by itself. It supports the entity, but panels depend on Google's overall confidence and your notability across the web.
Wikidata or Wikipedia first?
Wikidata, it's openly editable with sources and feeds the Knowledge Graph directly. Pursue Wikipedia later if you meet notability guidelines.
Are entities only for big brands?
No. Any organisation or author can build a clear, corroborated entity. You don't need fame, you need consistency and independent sources that agree.
Want to know how clearly AI engines actually understand your brand?
An entity audit maps your Knowledge-Graph presence, sameAs corroboration, and schema, and gives you the priority fixes that move you from "string" to confidently-cited "thing."
Request an entity & AI-visibility audit →
🗺️ Related, entities & AI search
- The Forgotten HTML: what AI crawlers really see
- Entity SEO: helping AI understand who you are
- Brand Mentions vs Backlinks for AI visibility
- Structuring content so AI can extract it
- FAQ schema: why you still keep it
---
# WebMCP, Explained: How Your Site Hands AI Agents a Menu of Actions
Source: https://seoprocheck.com/research/ai-search/webmcp-explained/
For years, AI agents have used websites the way a person would with the lights off: take a screenshot, guess which pixels are the button, click, and hope. It is slow, it breaks the moment you change your layout, and it is the reason "agentic browsing" has felt more like a demo than a tool. WebMCP is the proposed fix. Instead of making agents guess, your site hands them a clean, structured menu of the things it can actually do.
This is the plain-English explanation: what WebMCP is, how it works with real code, how it differs from the MCP you may have heard of, and an honest read on whether it matters yet. Because it is very new, and pretending otherwise would be its own kind of BS.
🗺️ TL;DR
WebMCP (Web Model Context Protocol) is a browser-native API, navigator.modelContext, that lets a website publish a "tool contract": a structured list of actions an AI agent can call directly, like addToCart or searchFlights, complete with input schemas. The agent calls your function instead of clicking around blindly. It was proposed by Google and Microsoft engineers and is being standardised at the W3C. It is in early browser preview as of 2026, so it is a thing to understand and prototype now, not a box to tick.
Feb 2026
entered early preview in Chrome 145, so it is experimental, not shipped to everyone
2 APIs
a Declarative one (HTML form attributes) and an Imperative one (JavaScript)
W3C
proposed by Google and Microsoft engineers, standardised in the open at the W3C
## 🤔 What WebMCP actually is
WebMCP stands for Web Model Context Protocol. It is a browser API exposed as navigator.modelContext that lets your site declare its capabilities as callable "tools." Each tool has a name, a human and agent readable description, and an input schema describing exactly what it accepts. An AI agent running in the browser reads that contract and calls the tools directly, rather than interpreting your interface visually.
The mental model is simple: today your site exposes a user interface for humans. WebMCP lets it also expose an action interface for agents. Same underlying functions, a second door built for machines.
## 🆚 WebMCP vs MCP: not the same thing
This trips people up, so it is worth being precise. Anthropic's MCP (Model Context Protocol) is a server-side protocol that connects AI models to data sources and tools on the backend. (Anthropic introduced MCP in late 2024, and it has quickly become a widely adopted standard for connecting AI tools to data.) WebMCP is a client-side browser API that lets a website expose actions to an agent running in the user's browser. They share a name and a philosophy, structured tools instead of guesswork, but they operate in different places. WebMCP is the web-page-facing cousin.
## 🏛️ Who built it, and why that matters for your decision
When you are deciding whether to bet time on a new standard, who is behind it tells you a lot, and this is where WebMCP looks sturdier than most of the "AI files" doing the rounds. WebMCP was proposed by engineers at Google and Microsoft and is being developed in the open at the W3C, inside its Web Machine Learning Community Group. Two major browser makers plus a standards body is a very different signal from a single proponent.
Contrast it with llms.txt, the file it often gets lumped in with. llms.txt was proposed by one person, Jeremy Howard of Answer.AI, in September 2024, and he said plainly at the time that it "isn't an official standard." That does not make it bad. robots.txt also began as a grassroots idea back in 1994. But it does change the risk profile. A proposal with Google, Microsoft, and a W3C group behind it is more likely to ship across browsers and stick around. The honest caveat: backing improves the odds, it does not guarantee them, and plenty of well-funded web proposals have quietly stalled. So treat strong backing as a reason to pay attention, not proof that it has already won.
## 🩹 The problem it solves
Agentic browsing today mostly works by visual interpretation: the agent screenshots the page, runs a model to locate elements, and simulates clicks and typing. That approach is brittle. A redesign breaks it, dynamic content confuses it, and every action is a guess that can go wrong. Honestly, asking a model to play "find the button" on every site was always going to be the fragile way to do this.
WebMCP replaces guessing with declaration. The agent gets:
- Discovery: a list of exactly what actions exist.
- Schemas: the precise inputs and outputs each action expects.
- State: which tools are available right now, given the current page or login state.
Without WebMCP
The agent screenshots the page, guesses which element is the "Add to cart" button, simulates a click, and hopes the layout did not change since last time.
With WebMCP
The agent reads your tool contract and calls addToCart(sku, quantity) directly, with a defined schema and a predictable result.
## ⚙️ How it works: the two APIs
WebMCP gives you two ways to define tools. Use the simple one when your actions map to forms, and the powerful one when they do not.
### 1. The Declarative API (HTML)
For actions that already exist as forms, you add a few attributes and the browser generates the tool schema from your existing markup. No JavaScript required.
### 2. The Imperative API (JavaScript)
For dynamic or complex actions, register tools in code with navigator.modelContext.registerTool(). Always feature-detect first, because the API is not yet everywhere.
if ("modelContext" in navigator) {
navigator.modelContext.registerTool({
name: "addToCart",
description: "Add a product to the cart by SKU and quantity.",
inputSchema: {
type: "object",
properties: {
sku: { type: "string" },
quantity: { type: "number" }
},
required: ["sku"]
},
readOnly: false,
async execute({ sku, quantity = 1 }) {
await cart.add(sku, quantity);
return { content: [{ type: "text",
text: `Added ${quantity} x ${sku} to the cart` }] };
}
});
}
When the available actions change, for example after a user signs in, navigator.modelContext.provideContext() replaces the whole tool set in one call. Set readOnly: true on tools that only read data, so agents can call them without a confirmation step.
One thing to hold onto: this is a contract, and the quality of the contract is the whole job. A clear name, an honest description, and a precise schema mean agents call your tool correctly every time. A sloppy one means they do not.
## 💰 What it costs to implement
Good news first: there is no license, no fee, and no server requirement. WebMCP is an open standard built into the browser, so the real cost is engineering time, and that swings a lot by approach.
- Low cost, the declarative path: if you already have clean, well-labelled HTML forms, exposing them means adding toolname and tooldescription attributes. That is hours, not weeks, and it doubles as an excuse to finally tidy up your forms.
- Higher cost, the imperative path: dynamic actions registered with registerTool() need real work: schema design, an execute handler, state management, and testing. Budget days per meaningful tool, more if the action is complex.
- Ongoing cost, maintenance: the tool contract has to stay in sync with your site. Change a checkout flow and you change the tool. This is a living interface, not a set-and-forget file.
- Hidden cost, security: every exposed tool is a surface to review and protect. Budget for that, not just the build.
- Risk cost, the spec is a draft: the API can still change before it stabilises, so expect some rework if you adopt during the preview.
## 🛠️ How to implement it, step by step
- Inventory your valuable actions. List what users actually come to do: search, filter, add to cart, book, submit. Those are your candidate tools.
- Start declarative. For actions that are already forms, add toolname and tooldescription and label the fields. Lowest effort, immediate coverage.
- Add imperative tools for the dynamic stuff. Use registerTool() with a clear name, an honest description, and a precise inputSchema. Mark read-only tools readOnly: true.
- Feature-detect, always. Wrap everything in if ("modelContext" in navigator) so browsers without support are unaffected.
- Secure the sensitive ones. Gate anything that spends money or changes data behind authentication and a confirmation step.
- Test in Chrome Canary with the experimental API enabled, and watch how an agent actually calls your tools. Tune the descriptions and schemas until it calls them correctly every time.
- Maintain the contract. Treat it like an API you own: update it whenever the site changes.
## 📏 How reliable and adopted is it, honestly
Here is the no-spin status. WebMCP entered early preview in Chrome 145 in February 2026 and runs in Chrome Canary behind the experimental API. It is not in stable browsers for everyone, the specification is a W3C draft that can still change, and real-world adoption is minimal. Chrome's Lighthouse has started checking whether sites expose it, under an agentic-browsing audit, which is a signal of where the browser team is heading rather than proof of mass use.
So the measured take: this is the most forward-looking item in the AI-readability stack. It is worth understanding now and worth prototyping if you have clear transactional actions, but you are not behind for lacking it in 2026, and nobody can honestly tell you it is a ranking or visibility factor today.
## 🔎 What WebMCP means for SEO and visibility
This is the part that matters most for marketers, and it is easy to get wrong in both directions. The cleanest way to hold it: SEO and AI search help agents find you. WebMCP helps them finish the job. They are different layers. WebMCP is not a ranking signal and not an AI-citation signal, so adding it will not move your positions or get you quoted more often.
What it changes is what happens after an agent arrives. As people start handing tasks to AI assistants ("find blue running shoes in size 10 and add them to a cart"), the sites that let an agent complete that task cleanly are the ones that capture the action, and the ones that force the agent to fumble through a visual interface get skipped. One analysis frames the gap bluntly: screen-based agent actions take several seconds each with a meaningful error rate, while a called tool resolves in a second or two with near-zero errors.
So the honest framing is: WebMCP is transaction readiness, not a discovery trick. It sits on top of the fundamentals, it does not replace them. You still need to be crawlable, rendered cleanly, well structured, and a clear entity first. Anyone selling WebMCP as a shortcut to rankings has it backwards.
## ⚖️ WebMCP pros and cons
Pros
- Reliable agent actions: a called tool beats a guessed click
- Captures agentic traffic: you stay usable when an agent arrives to act
- You control the contract: you decide what is exposed, and how
- Low effort if you already have clean HTML forms (the declarative API)
- First-mover position on a standard most sites have not noticed yet
Cons
- Very early: Chrome preview only, and the spec can still change
- No SEO or ranking benefit today
- Adds a new attack surface you have to secure
- Real ROI depends on agents going mainstream, and that timing is not guaranteed
- Needs engineering time that informational sites may be better off spending elsewhere
## 🛍️ What is it good for, and who benefits
WebMCP earns its keep wherever user intent is clear and the valuable actions are structured. The clearest winners:
- Ecommerce: search the catalog, filter, check stock, add to cart, start checkout. This is the headline use case, and where agentic commerce is heading first.
- Travel and booking: search availability, book, change, or cancel a reservation.
- SaaS: create or update records, run a report, or trigger a workflow, each exposed as a callable tool.
- Local services and support: book an appointment, submit a request, check an order, or open a ticket.
Who benefits most: transactional sites where an agent completing an action equals revenue or a qualified lead. If your customers search, book, buy, or submit, this can matter to you sooner than it looks.
Who benefits least, for now: purely informational or content sites. Your prize is being read and cited, which lives in the other layers (rendering, schema, entities), not in exposing actions. For you, WebMCP can wait.
## 🧭 Should you implement it yet?
- Experiment now if you run an ecommerce, booking, SaaS, or search-driven site with clear, valuable actions (add to cart, book, filter, search) and you like being early on standards your competitors have not noticed.
- Wait and watch if you are a mostly informational site, or you have no engineering time to spare. The content-readability layers (rendering, schema, entities) will do more for you today.
- Either way, read the spec, prototype in Canary, and keep an eye on which browsers move it toward stable.
## 🔐 A word on security
Exposing actions to agents is powerful, so treat it with the same care as a public API. Gate sensitive tools behind authentication, use readOnly honestly, require confirmation for anything that spends money or changes data, and never assume the caller is friendly. A tool contract is an attack surface as well as a feature.
## ❓ WebMCP FAQ
Is WebMCP live and usable today?
It is in early preview in Chrome 145 (Canary) as of February 2026, behind an experimental flag. It is not yet in stable browsers for general use.
Is WebMCP the same as Anthropic's MCP?
No. MCP is a server-side protocol connecting models to backend tools. WebMCP is a client-side browser API for exposing a website's actions to an agent in the browser.
Does WebMCP replace my REST or GraphQL API?
No. It sits in front of your existing logic to expose actions to in-browser agents. Your APIs still do the work underneath.
Will WebMCP help my SEO or AI visibility?
Not directly, and not today. It is about agents taking actions, not about being found or cited. Treat it as future capability, not a visibility lever.
Is it a Google product or an open standard?
It was proposed by Google and Microsoft engineers and is being developed openly at the W3C Web Machine Learning Community Group, so it is a standards effort, not a single vendor's product.
Want to be early and correct on the agentic web?
An advanced audit covers where WebMCP fits alongside your rendering, schema, and entity work, so you prototype the right things in the right order, without the hype.
Request an advanced SEO and AI-readiness audit →
📚 Related, the machine-readable web
- The Machine-Readable Web (the field guide)
- The AI Crawler Map: every bot reading your site
- Entity HTML: becoming a machine-readable brand
- llms.txt explained
- The Forgotten HTML: what AI crawlers really see
---
# Is Your Site Agent-Ready? The Lighthouse Agentic-Browsing Audit and How to Pass It
Source: https://seoprocheck.com/research/ai-search/agent-ready-lighthouse-agentic-browsing/
AI agents are starting to do things on websites, not just read them: search a catalog, fill a form, book a slot, add to a cart. And as of May 2026, Google ships a way to measure whether your site is ready for them. It is a new category in Lighthouse called Agentic Browsing, and it quietly grades how usable your pages are to an automated agent. If you have wondered "is my site agent-ready, and how would I even know," this is the answer: what the audit checks, how to pass each part, and how much of it actually matters yet.
🗺️ TL;DR
Lighthouse's Agentic Browsing audit (May 2026) checks four things: an llms.txt file, WebMCP tool declarations on your forms, a clean accessibility tree, and layout stability (CLS). It is experimental and reports pass or fail per check rather than a 0 to 100 score, because the standards are still forming. None of it is a Google ranking factor, Google Search has said so plainly. It is about whether an agent can use your site, which matters most for transactional sites. The single highest-leverage item is the boring one: a clean, semantic accessibility tree.
4 checks
llms.txt, WebMCP, the accessibility tree, and layout stability
Pass / fail
no 0 to 100 score; the category is experimental, a pass fraction, because the standards are still emerging
Not a ranking factor
Google Search says none of this affects your positions; it is about agents, not SERPs
## 🤖 What "agent-ready" actually means
An agent-ready page is one an AI agent can understand and operate without guessing. A human looks at your page and infers what the "Buy" button does and which field is the email. An agent cannot rely on that visual intuition reliably. It needs the meaning to be declared: clear labels, declared actions, a stable layout it can target. Lighthouse's Agentic Browsing category is Google's first attempt to score exactly that, and it landed in Lighthouse 13.3 in May 2026.
Worth being honest up front: this is a Chrome and web-platform initiative, not a Search ranking system. That is why you get the slightly confusing situation where Google Search says you do not need llms.txt, while Google's own Lighthouse checks for it. Different teams, different goals. It is a little WTF when you first see it, and the confusion is on them, not you. The resolution: the audit is about agent usability, not rankings.
## ✅ The four checks, and how to pass each one
1. llms.txtCan an agent find a machine-readable summary of your site?
2. WebMCPAre your forms and actions declared as callable tools?
3. Accessibility treeIs every interactive element programmatically named?
4. Layout stabilityDoes the page stay still so an agent can click reliably?
### 1. llms.txt: discoverability
Lighthouse checks whether your domain root serves an llms.txt, a Markdown summary that points an agent at your key content. How to pass: publish a valid /llms.txt. The honest caveat: Google Search's own May 2026 AI optimization guide says llms.txt is not needed for AI Overviews or AI Mode, and the major AI crawlers largely ignore it for content. Lighthouse measures it anyway, for agent discovery. So pass the check if you like, but do not expect search visibility from it. The full story is in our llms.txt explainer.
### 2. WebMCP: declared actions
The audit looks for forms that are not declared through WebMCP and flags them, because an undeclared form is a black box to an agent: it has to guess what the fields mean. If you do declare tools, Lighthouse validates the input schema and fails it if the schema is invalid. How to pass: expose your key forms and actions with WebMCP, starting with the lightweight declarative form attributes, and make sure any input schemas are valid. This is the newest and highest-effort check. Details and code are in our WebMCP explainer.
### 3. Accessibility tree: the machine-eye view
This is the one to care about most, and it is not new. Agents use the accessibility tree as their primary model of your page, the same structure screen readers use. Lighthouse checks that interactive elements have programmatic names and labels, that the tree is valid, and that interactive content is not hidden from assistive systems. How to pass: use semantic HTML (real button, label, nav elements), give every control an accessible name, and add ARIA only where semantics fall short. The bonus: this is plain good accessibility, so it helps humans and screen readers too. It is the same idea as our Forgotten HTML thesis: machines act on the structured layer, not the pixels.
### 4. Layout stability (CLS): stay still
Agents need stable DOM positions to click reliably. If your layout shifts while content loads, an agent can click the wrong thing, and it fails silently. Lighthouse reuses Cumulative Layout Shift here. How to pass: reserve space for images and embeds, avoid inserting content above existing content, and keep CLS low. If you have already done Core Web Vitals work, you are most of the way there.
Check
What it measures
How to pass
Effort
llms.txt
Agent discoverability
Publish a valid /llms.txt
Low
WebMCP
Declared, callable actions
Annotate forms, valid input schemas
High
Accessibility tree
Machine-readable structure
Semantic HTML, names on every control
Medium
Layout stability
Reliable click targets
Keep CLS low
Low to medium
## 🏛️ Who is behind it, and what that signals
The Agentic Browsing category comes from Google's Chrome and Lighthouse team, and it builds on standards being developed in the open: WebMCP at the W3C, the accessibility tree as a long-standing web standard, and llms.txt as a grassroots proposal. That mix matters. The accessibility and CLS parts rest on mature, stable foundations, so investing there is safe regardless of how the agentic web shakes out. The WebMCP and llms.txt parts are newer and still moving, so treat them as forward bets, not settled requirements. Google itself underlines this by refusing to give the category a single score and labeling it experimental, with the note that the standards for the agentic web are still emerging. That is a healthy signal: even the vendor is telling you not to over-commit yet.
## 🔎 Does passing this help your SEO?
Directly, no. Google Search has stated that none of these signals affect your rankings, and the Lighthouse category produces no ranking-relevant score. So if someone is selling "pass the agentic audit to rank higher," that is the BS to walk past. What it genuinely affects is whether an AI agent can complete a task on your site, which is a conversion and future-readiness question, not a discovery one. The honest framing is the same one that applies across this whole space: search and AI answers help agents find you, agent-readiness helps them finish the job once they arrive.
## ⚖️ Should you invest now? Pros and cons
Reasons to do it now
- The accessibility and CLS work pays off immediately for humans and SEO, audit or not
- You get ahead on agent-readiness before competitors notice the category exists
- Most of it is hygiene you should be doing anyway
Reasons to wait on parts of it
- WebMCP is in preview and the spec can change, so heavy investment may need rework
- None of it moves rankings today
- The category is experimental and may evolve before it stabilises
## 🧭 Who benefits most
Most: transactional sites where an agent completing a task equals revenue, ecommerce, travel, booking, SaaS, local services. For them, agent-readiness is a real competitive edge as agentic shopping grows. Least, for now: purely informational and content sites. Your value is being read and cited, which lives in the content and entity layers, so focus the agentic work on the accessibility tree and CLS and skip the heavy WebMCP build until you have a reason.
## 🛠️ How to make your site agent-ready, step by step
- Run the audit. Open Lighthouse in Chrome (the Agentic Browsing category is in 13.3 and later) and see which of the four checks you fail.
- Fix the accessibility tree first. Highest leverage, helps everyone. Semantic HTML, an accessible name on every control, ARIA only where needed.
- Tame layout shift. Reserve space for media, avoid late-injected content. Reuse any Core Web Vitals work.
- Add llms.txt if agents or developer docs matter to you. Low effort, useful for agent discovery, not for rankings.
- Pilot WebMCP on your top one or two actions. Start with the declarative form attributes, validate the schemas, and expand only if it earns its keep.
- Re-run and re-check. Treat it as a living target, since the category is still evolving.
## ❓ Agent-readiness FAQ
Is the Lighthouse Agentic Browsing audit a ranking factor?
No. Google Search has stated these signals do not affect rankings. The category is about agent usability, not SERP position.
Do I need llms.txt to pass?
To pass that one check, yes, you publish a valid /llms.txt. But it is optional overall and not a search-visibility signal, so weigh it as agent-discovery hygiene.
How do I run the audit?
Use Lighthouse in Chrome DevTools, version 13.3 or later, and select the Agentic Browsing category. It returns pass or fail per check rather than a single score.
What is the single most important check?
The accessibility tree. It is mature, it helps humans and SEO, and it is the structure agents actually read.
Is agent-readiness worth it if I run a blog?
Do the accessibility and CLS parts, they help regardless. Skip the heavy WebMCP work until agents completing actions on your site is something you actually want.
Want to know exactly where your site stands on agent-readiness?
An advanced audit runs the agentic checks alongside your crawlability, rendering, schema, and entity signals, and hands you a prioritised list of what to fix first, with the receipts.
Request an advanced SEO and agent-readiness audit →
📚 Related, the agentic web
- WebMCP, explained
- llms.txt explained
- The Forgotten HTML: what AI crawlers really see
- The Machine-Readable Web (the field guide)
---
# The Complete robots.txt Reference: Precedence, Wildcards, AI Bots & Real-World Receipts
Source: https://seoprocheck.com/indexing-crawl/robots-txt-complete-reference/
Most robots.txt files are copied from a template, pasted once, and never understood. That is how a single line silently removes a site from half the web's crawlers, or, just as often, fails to block the thing the owner thought it blocked. This is the complete reference: what every directive does, the precedence rules almost nobody reads, real configurations pulled from live sites in June 2026, and the mistakes that quietly cost traffic.
🗺️ TL;DR
robots.txt controls crawling, not indexing, a disallowed URL can still rank if it's linked. Crawlers obey only their single most-specific user-agent group, which is why User-agent: * / Disallow: / doesn't block Google when Googlebot has its own group. Within a group, Google picks the most-specific (longest) rule, not the first, but legacy crawlers use first-match, so order still matters if you care about more than Google. Never use robots.txt to hide a page you also want de-indexed: if Google can't crawl it, it can't see your noindex.
500 KB
Google's robots.txt size cap, content past it is ignored
1 group
a crawler obeys only its single most-specific user-agent group, and ignores all the others
≠ noindex
robots.txt blocks crawling, never indexing, these are different jobs
## 📍 What robots.txt is, and the three things it is not
robots.txt is a plain-text file at the root of a host (https://example.com/robots.txt) that tells compliant crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. That is all it does. It is not:
- Not a security control. The file is public and advisory. Listing Disallow: /admin/ tells the world where your admin lives while stopping no determined actor. Protect with authentication, not robots.txt.
- Not a way to de-index. Disallowing a URL stops crawling, but Google can still index a disallowed URL it finds via links, showing it with no snippet. To remove a page from search, use a noindex meta tag or header (and leave it crawlable so Google can see it).
- Not a guarantee. Compliance is voluntary. Mainstream crawlers honour it; scrapers and some AI bots do not (covered below).
## 🔧 The anatomy: every directive that matters
A robots.txt file is one or more groups. Each group starts with one or more User-agent lines and contains Allow/Disallow rules. Sitemap is independent of groups.
User-agent: Googlebot # this group applies to Googlebot only
Disallow: /cart/ # block a folder
Allow: /cart/promo.html # …but allow one file inside it
Disallow: /*?sort= # wildcard: any URL containing ?sort=
Disallow: /*.pdf$ # $ anchors the end: only URLs ending .pdf
User-agent: * # everyone else
Disallow: /private/
Sitemap: https://example.com/sitemap.xml # absolute URL, group-independent
- User-agent, names the crawler the group applies to. * matches any bot that doesn't have its own group.
- Disallow, a path prefix the crawler must not request. Empty Disallow: means "allow everything."
- Allow, carves an exception out of a broader Disallow. Supported by Google and most majors.
- * and $, * matches any sequence of characters; $ anchors the end of the URL. Google supports both; some legacy crawlers ignore them entirely.
- Sitemap, an absolute URL to your XML sitemap. It is global, not tied to any user-agent group, and you can list several.
## ⚖️ The precedence rules almost nobody reads
This is where well-meaning files go wrong. Two separate precedence questions decide what a crawler actually does, and they resolve differently across crawlers.
How a crawler decides what it may fetch
1. Pick ONE groupThe crawler selects the single most-specific user-agent group matching its name, and ignores every other group entirely.
→
2. Match the pathWithin that one group, find the rules whose path matches the URL.
→
3. Resolve the winnerGoogle: the most-specific (longest) rule wins; a tie goes to Allow.Legacy/first-match: the first matching rule wins, so order matters.
### Rule 1, User-agent groups: most specific wins, and it's winner-take-all
A crawler reads only the one group whose user-agent token is the longest match for its name. It does not merge groups. So a Googlebot group makes Googlebot ignore the * group completely, even the rules you assumed were universal.
Receipt (live, June 2026): LinkedIn, Yelp and Reuters all publish User-agent: * → Disallow: /. It looks like "block the entire internet." It isn't: each of them also defines dedicated Googlebot and Bingbot groups, so those engines crawl freely while everyone else is turned away. The * block never applies to Google because Google matched a more specific group first. LinkedIn even appends a note: apply for whitelisting by email.
### Rule 2, Within a group: Google uses specificity, legacy crawlers use order
Once a crawler is inside its group and a URL matches several rules, who wins? Here the standards diverge, and this is the part that surprises people:
- Googlebot (and most modern majors): the most-specific rule wins, specificity measured by the number of characters in the path. Order in the file is irrelevant. If an Allow and a Disallow are exactly equally specific, Google breaks the tie toward the least restrictive rule (the Allow).
- Legacy / strict-RFC crawlers: many older or simpler parsers use first-match-wins. With those, the order of your Allow/Disallow lines changes the outcome.
So the honest answer to "does the order of rules in robots.txt matter?" is: not for Google, but yes for a meaningful slice of other crawlers. Write your file so it produces the correct result under both interpretations, put the specific Allow before the broad Disallow, and don't rely on Google's longest-match cleverness if you care who else obeys you.
# Correct under BOTH longest-match and first-match crawlers:
User-agent: *
Allow: /downloads/whitepaper.pdf # specific exception first
Disallow: /downloads/ # broad block second
## 🌐 Real configurations in the wild (June 2026)
Pulled live from production robots.txt files, these are observations, not value judgements. The point is what each does.
Site(s)
Configuration
Effect
LinkedIn, Yelp, Reuters
User-agent: * → Disallow: / + dedicated Googlebot/Bingbot groups
Default-deny allowlist: only named engines crawl; everyone else is blocked
NYT
Standalone Disallow: / for OAI-SearchBot, PerplexityBot, Claude-SearchBot
Deliberately absent from ChatGPT, Perplexity and Claude citations
CNN, BBC, The Verge, The Guardian, Amazon, eBay
Block one or more AI search crawlers
Reduced or zero eligibility for those engines' answer citations
GitHub, Walmart
Crawl-delay: 1 / Crawl-delay: 5
Honoured by Bing/Yandex; ignored by Google, which has no crawl-delay support
The AI-bot blocks above are usually deliberate (licensing posture). The trap is doing the same thing by accident, see the AI section below.
## 🚫 The mistakes that quietly cost you
Blocking a page you also want de-indexed
The most common own-goal. You add noindex to a page and Disallow it in robots.txt. Google can no longer crawl the page, so it never sees the noindex, and the URL can linger in the index. Pick one: to de-index, allow crawling and use noindex; to block crawling, accept the URL may still appear.
Using unsupported directives
Noindex: in robots.txt has not been supported by Google since September 2019. Crawl-delay is ignored by Google. Both are silently skipped, you think you've set a rule that does nothing.
Blocking render-critical CSS/JS
If Googlebot can't fetch the CSS and JavaScript needed to render a page, it evaluates a broken version of it. Never disallow the assets your layout and content depend on. (Blocking a single tracking or JSONP endpoint is fine, blocking /assets/ or /wp-includes/ wholesale is not.)
Path and scope slips
Paths are case-sensitive (/Folder/ ≠ /folder/); directive names are not. robots.txt is scoped to one origin, protocol, host and port, so https:// and http://, and each subdomain, need their own file. And it must sit at the root; a file at /blog/robots.txt is ignored.
## 🤖 robots.txt and AI crawlers
The fastest-growing reason to touch robots.txt in 2026 is AI. The critical distinction, covered in depth in The AI Crawler Map, is that AI bots do different jobs: some train models, some index pages to answer questions, some fetch a page live for a user. Blocking a training crawler costs almost no traffic; blocking a search/answer crawler removes you from that engine's citations. The publisher blocks in the table above are deliberate licensing decisions. The danger is replicating them by pasting a "block all AI" snippet you found online and silently deleting yourself from ChatGPT Search, Perplexity and Claude. The trap is reusing a broad "block all AI" snippet without checking which bots it actually covers, so a training block quietly takes out the search and answer crawlers too.
And remember: robots.txt is a request. Independent monitoring through 2025 reported that some AI operators fetched disallowed pages anyway. If you need a hard block, enforce it at the edge (WAF, verified IP ranges), not in a text file.
## 🧪 How to test it
- Read the live file. Open yourdomain.com/robots.txt in a browser, that's exactly what crawlers see.
- Inspect a URL in Search Console. The URL Inspection tool reports whether a specific URL is blocked by robots.txt for Googlebot.
- Use a validator. Google's open-source robots.txt parser (the basis for its real behaviour) and several third-party testers will tell you whether a given URL/user-agent pair is allowed.
- Confirm assets render. Use the URL Inspection "view rendered page" to ensure CSS/JS aren't blocked.
## ❓ robots.txt FAQ
Does robots.txt stop a page from being indexed?
No. It stops crawling. A disallowed URL can still be indexed (without a snippet) if other pages link to it. Use noindex to keep a page out of search.
Does the order of rules matter?
Not for Google, which uses the most-specific rule regardless of order. Yes for first-match crawlers. Write your file to be correct under both.
Is robots.txt case-sensitive?
The paths are (/Page ≠ /page); the directive keywords (User-agent, Disallow) are not.
Can I block a folder but allow one file inside it?
Yes, add an Allow for the specific file. It's more specific than the folder Disallow, so it wins for Google; place it first for legacy crawlers too.
Does Google support crawl-delay?
No. Google ignores it; control Googlebot's rate via Search Console settings. Bing and Yandex do honour it.
How large can robots.txt be?
Google reads up to 500 KB; anything beyond is ignored. Keep it lean.
Does one robots.txt cover my subdomains?
No. It's per-origin. blog.example.com and shop.example.com each need their own, as do http and https.
What happens if I have no robots.txt at all?
Crawlers assume everything is allowed. A missing file (returning 404) is treated as "crawl freely", which is often fine.
Want your robots.txt audited against Google, Bing, and every AI engine that matters?
An advanced technical audit checks your crawl directives, rendering, and AI-bot access, and tells you exactly what to change, with the receipts.
Request an advanced technical SEO audit →
🔧 Related, crawling & indexing
- robots.txt: common mistakes
- The AI Crawler Map: every bot reading your site
- Crawl budget, explained
- SSR vs CSR: why rendering decides what AI can read
---
# llms.txt Explained: What It Is and Whether You Need One
Source: https://seoprocheck.com/research/ai-search/llms-txt-explained/
Few things in AI search are as misunderstood as llms.txt. Half the internet treats it as a magic ticket to AI citations; the other half confuses it with robots.txt. Neither is right. There are actually two files, they do different jobs, and the evidence on whether AI systems use them at all is… not flattering. Let me clear it up , with receipts.
## Two files, two jobs
- llms.txt, the index. A short Markdown file (kept under ~10KB) with a curated list of links to your most important pages, organized by topic. Think of it as a table of contents or sitemap written for an LLM, so an agent can navigate to the right page.
- llms-full.txt, the whole thing. A single large Markdown file containing your entire documentation/content concatenated, so a model can ingest everything in one fetch. Many sites that adopt the idea publish both.
Proposed by Jeremy Howard of Answer.AI in September 2024 (spec at llmstxt.org), the format solves a real problem: model context windows cannot swallow a whole HTML site full of nav, ads and JavaScript, so a clean Markdown map/dump is easier to consume. Crucially, it is aimed at inference time , an agent you point at your docs , not at the big training or search crawlers.
## Does anything actually use it? The receipts.
This is where the hype collides with reality:
- Google says no. In June 2025, Google's John Mueller stated plainly that "no AI system currently uses llms.txt," adding: "It's super-obvious if you look at your server logs… none of them fetch the llms.txt file." He compared it to the old keywords meta tag , something a site owner claims, that engines ignore (Search Engine Roundtable). Google has separately said normal SEO, not llms.txt, is what wins AI Overviews (Search Engine Land).
- The server logs agree. Multiple log analyses through 2025 found zero requests for llms.txt from GPTBot, ClaudeBot, PerplexityBot or Google-Extended , the very crawlers it is meant for.
- No citation lift. A large 2025 study of hundreds of thousands of domains found no correlation between having an llms.txt and being cited by AI systems (PPC Land).
## The nuance everyone misses: published ≠ consumed
Here is the twist that fuels the confusion. The file is widely published , Anthropic's own developer docs serve one, and documentation platforms like Mintlify auto-generate it for thousands of sites. So you see llms.txt everywhere and assume it must matter. But publishing a file is not the same as a major AI provider reading it. No major LLM provider , OpenAI, Anthropic, or Google , has confirmed consuming it in production search/answers, and the objective measures (logs, citation studies) show negligible use.
The one place it genuinely helps today: developer tools and agents you explicitly point at your docs (Cursor, Claude Code, Copilot and similar) , they can ingest a clean llms-full.txt far more easily than crawling your HTML.
## So should you add one?
- If you publish documentation a clean llms.txt / llms-full.txt is a reasonable, low-cost nicety for agent ingestion.
- Do not expect AI search citations from it. As of 2026 the evidence says it does not drive AI visibility.
- Spend your effort where it counts first: being crawlable and in the raw HTML (not client-rendered), not blocking the AI bots you want, and publishing genuinely useful, well-structured content. Those are what actually get you cited.
Bottom line: llms.txt is a tidy idea with real utility for agents and dev docs , and, as of now, no demonstrated impact on AI search visibility. Add it if it is cheap; never let it distract from the fundamentals.
🤖 More on AI search & GEO
- The Forgotten HTML · Which AI bots are you blocking? · How to get cited in AI
## Want to show up in AI answers (for real)?
I help companies earn AI citations through the things that actually move the needle. See how an advanced SEO audit works →
---
# FAQ Schema: Why You Still Keep It After Google Retired the Rich Result
Source: https://seoprocheck.com/schema-guides/faq-schema-still-worth-it/
Here is the argument that keeps surfacing in technical-SEO reviews: "Google retired FAQ rich results, so rip out the FAQPage schema." It is half right, and the wrong half is the one that costs you. Google did retire the FAQ rich result for almost every site. That is true, and you should say so out loud. But the schema itself is still a supported, useful, near-zero-cost asset, and removing it gains you nothing.
This is the full, receipt-backed picture, so you can make the call on purpose instead of on a rumour: what Google actually changed, what it did not change, and exactly when keeping FAQPage markup is worth it.
What actually happened to FAQ rich results
Aug 2023Google limits FAQ rich results to "well-known, authoritative government and health" sites
→
Late 2023HowTo rich result restricted, then removed entirely; FAQ stays gov/health-only
→
TodayRich result still gov/health-only, but FAQPage remains a valid, supported Schema.org type
Google changed the display (the rich result). It never removed the markup or told anyone to delete it. Google introduced FAQ rich results, encouraged the markup, then a few years later restricted the feature to a narrow set of sites. That kind of reversal is a big part of why teams keep asking whether the schema is still worth keeping.
🗺️ TL;DR
In August 2023 Google restricted the FAQ rich result to authoritative government and health sites, so for nearly everyone, marked-up FAQs stopped showing those expandable Q&As in search. That part of the call-out is correct. But FAQPage is still a valid Schema.org type Google supports; it costs a few kilobytes, it makes your Q&A content explicitly machine-readable for AI answer engines and assistants, and it forfeits nothing. Keep it where you have genuine, visible FAQs, just don't expect the old rich result, and never fake questions to earn markup.
Aug 2023
Google limited FAQ rich results to authoritative government & health sites
Still valid
FAQPage was never removed from Schema.org or Google's supported types, only the rich result changed
Don't fake it
Only mark up FAQs genuinely visible on the page, the rule that keeps schema an asset, not a liability
## 📌 The receipt: what Google actually announced
On 8 August 2023, Google published "Changes to FAQ and HowTo rich results" on Search Central. The two concrete changes:
- FAQ rich results would only be shown for "well-known, authoritative government and health websites." For every other site, the expandable FAQ treatment in search results was switched off, globally, rolling out over the following weeks.
- HowTo rich results were limited to desktop, and then removed entirely; Google later deprecated HowTo structured data reporting altogether.
Google's stated reason was a cleaner, less cluttered results page. Crucially, the announcement was about when the rich result appears. Google did not deprecate the FAQPage type, did not ask anyone to remove existing markup, and kept the structured-data documentation live (updated to note the new eligibility limit). So the honest framing for any deck or audit is: "Google retired the FAQ rich result for non-gov/health sites in 2023", not "FAQ schema is dead."
## 🔍 What changed vs. what didn't
✕ What you lost
- The expandable FAQ rich result in Google for non-gov/health sites
- The extra SERP real estate and CTR those accordions used to capture
- FAQ impressions/clicks reporting in the Search Console enhancement report
✓ What you kept
- FAQPage as a valid, supported Schema.org type
- Explicit, machine-readable Question → acceptedAnswer pairs
- Eligibility if you are a gov/health site, the rich result still fires
- Clean Q&A structure that feeds featured snippets, People Also Ask, and voice answers
## 🤖 Why keep it: the case for FAQPage in an AI-search world
The strongest reason to keep FAQPage isn't the old rich result, it's that structured data is the cheapest way to make your content unambiguous to machines. A Q&A buried in prose has to be inferred. A FAQPage block states it: this is the question, this is the answer, full stop.
- It's a shared vocabulary, not a Google feature. Schema.org is consumed by many systems, search engines, voice assistants, and AI answer engines that parse JSON-LD to extract entities and relationships. Marking up Q&A pairs removes guesswork for any of them. (Be precise: no major AI engine has publicly confirmed FAQPage as a ranking signal, see the limitations below, but the parsing benefit is mechanical and real.)
- It forces good content architecture. Writing genuine, scannable FAQs that map to real user questions is exactly what wins featured snippets and PAA placements, with or without the rich result.
- The cost is a rounding error. A FAQPage block is a few kilobytes of JSON-LD in the head. There is no rendering cost, no CrUX impact, no maintenance burden beyond keeping it accurate.
- It's forward-compatible optionality. Eligibility rules have changed before and can change again. If Google re-expands the rich result, marked-up sites benefit immediately; unmarked sites scramble.
Put bluntly: removing valid FAQPage markup gives you nothing and quietly removes optionality. That is a bad trade.
## 🧭 The bigger reason: don't architect only for Google
There is a strategic point underneath the tactical one. Google retired the FAQ rich result on its own timetable, with little notice, the same way it introduced FAQ results and then took them away. If your structured-data strategy is built purely around Google's current display decisions, you are signing up to rebuild it every time Google changes its mind. That is not a stable foundation; it is a dependency on a single vendor's roadmap.
And Google is no longer the only audience that matters. Search is fragmenting. AI assistants and answer engines increasingly sit between people and the open web, and they read the same structured signals to decide what to surface and cite. A markup choice that looks pointless through a Google-only lens looks very different once you count every engine that parses your pages. Optimising for the durable, shared layer, clean, machine-readable structure that any consumer can use, ages far better than optimising for one company's rich-result policy. Keep the schema for the web that is coming, not just the SERP that exists today.
## 🚦 When to keep it, add it, or skip it
Situation
Verdict
Why
Gov / health site with real FAQs
Keep / add
You still qualify for the live rich result
Any site with genuine, visible FAQ content
Keep
Machine-readable Q&A; near-zero cost; future optionality
Product/support pages with real Q&As
Keep
Helps assistants and snippets parse answers cleanly
Page with no real FAQs (markup only)
Skip
Marking up invisible/fake FAQs breaks Google's guidelines
You're keeping it only for the rich result
Reset expectations
Non-gov/health sites won't get it, keep it for the other reasons
## 🔧 Implementation: a correct FAQPage block
JSON-LD in the , one Question per real on-page FAQ, with the answer text matching what users actually see:
Validate with the Schema.org validator or Google's Rich Results Test. The Rich Results Test will tell you the markup is valid even though it won't preview a rich result for a non-eligible site, that is expected, not an error.
## 🚧 What FAQ schema will not do
Balance is the point of a real analysis, so here are the honest limits, the same caveats that make the "keep it" case credible:
- It will not restore the rich result for non-gov/health sites. The deck call-out is correct on this; don't promise stakeholders the accordions are coming back.
- It is not a confirmed AI ranking or citation signal. The parsing benefit is mechanical and plausible, but no major AI engine has publicly documented FAQPage as a weighting factor. Sell it as content hygiene and optionality, not a magic lever.
- It will not rescue thin content. Schema describes a page; it doesn't improve one. Fake or hidden FAQs marked up to chase features violate Google's structured-data guidelines and can earn a "spammy structured markup" manual action.
- It will not move traffic on its own. Treat it as an enabler that compounds with genuinely good Q&A content, not a standalone tactic.
## ✅ The one-line recommendation
Keep FAQPage schema wherever you have real, visible FAQ content. State plainly that Google retired the rich result for non-gov/health sites in 2023, that's accurate and builds trust, and keep the markup anyway, because it is machine-readable, future-proof, and costs essentially nothing. The only place to remove it is where the FAQs aren't genuinely on the page. That's the entire decision.
Want your structured-data strategy audited against what actually earns visibility in 2026?
An advanced SEO audit maps your schema, rich-result eligibility, and AI-answer readiness, and tells you what to keep, add, or cut, with the receipts.
Request an advanced SEO & schema audit →
🔧 Related, Schema & AI search
- Structured Data: the fundamentals
- LocalBusiness Schema: a complete property guide
- Google AI Overviews: how to get cited
---
# Google AI Overviews: How to Optimize for AI Overview Citations
Source: https://seoprocheck.com/research/ai-search/google-ai-overviews-seo-optimize-citations/
Most "AI Overview optimization" advice is hand-waving. The data isn't. Google AI Overviews cite your page overwhelmingly when you already rank in the top 10, so the fastest route to AI citations is ruthlessly good classic SEO, plus a handful of structural choices that make your content easy to lift. Below is what the numbers actually say, and the same playbook I implement for clients.
Google AI Overviews show the strongest correlation between organic rankings and AI citations of any platform. According to Ahrefs research, 76% of AI Overview citations come from pages ranking in Google's top 10, with position #1 providing a 33% citation probability.
## Key Statistics: AI Overview Citation Patterns
Metric
Value
Source
Citations from Google top 10
76.10%
Ahrefs
Median ranking position for citations
Position 3
Ahrefs
Position #1 citation probability
33.07%
Ahrefs
Position #10 citation probability
13.04%
Ahrefs
Sources per AI Overview response
5-28
Ahrefs
Maximum documented links
95
Ahrefs
## Citation Probability by Ranking Position
Ahrefs' analysis of 1.9 million citations from 1 million AI Overviews found a clear correlation between ranking position and citation likelihood:
Ranking Position
Citation Probability
Relative to Position 1
Position 1
33.07%
100% (baseline)
Position 2
21.54%
65%
Position 3
17.82%
54%
Position 5
15.21%
46%
Position 10
13.04%
39%
Key insight: Position #1 provides 2.5x the citation probability of position #10.
## How to Optimize for AI Overview Citations
### 1. Focus on Traditional Google SEO
Unlike other AI platforms, Google AI Overviews directly reward organic rankings. Standard Google SEO remains the primary strategy:
- Target high-volume keywords where you can rank top 10
- Build topical authority in your domain
- Create comprehensive, well-structured content
- Earn quality backlinks from authoritative sources
### 2. Build Brand Authority
Ahrefs found that brand mentions correlate 3x stronger with AI visibility than backlinks (0.664 vs 0.218). Top 50 brands account for 28.9% of all AI Overview citations.
- Invest in PR and earned media
- Build brand mentions across the web
- Establish E-E-A-T signals (experience, expertise, authority, trust)
- Get cited by authoritative publications in your industry
### 3. Use Structured Data
While Google states structured data isn't required for AI features, research shows significant impact:
Schema Type
Impact on AI Overview Appearance
FAQPage schema
3.2x higher likelihood
Proper heading hierarchy
40% more likely to be cited
Tables in content
2.5x more citations
List formatting
50% of top citations use lists
### 4. Optimize for Featured Snippets
AI Overviews often pull from the same content that wins featured snippets:
- Answer questions directly in the first paragraph
- Use the question as an H2 heading
- Provide concise, factual answers (40-60 words)
- Follow with supporting detail
## Google-Extended: Controlling AI Training
According to Google's documentation, Google-Extended is a robots.txt control token (not a crawler) that lets you block AI training while maintaining search visibility:
User-agent: Google-Extended
Disallow: /
This prevents your content from being used to train Google's AI models but does not affect:
- Regular Google search rankings
- AI Overview citations (these use existing search index)
- Google Discover or News visibility
## AI Overview Trigger Queries
Not all queries trigger AI Overviews. They appear most frequently for:
- Informational queries ("how to," "what is")
- Comparison queries ("X vs Y")
- Research queries ("best," "top")
- Complex questions requiring synthesis
Transactional and navigational queries rarely trigger AI Overviews.
## Frequently Asked Questions
### Do I need to do anything special for AI Overviews?
Focus on ranking well in traditional Google search. 76% of AI Overview citations come from top 10 results. Good SEO is good AI Overview optimization.
### Does blocking Google-Extended affect AI Overview citations?
No. Google-Extended controls AI training, not AI Overview citations. Your content can still appear in AI Overviews even if you block Google-Extended.
### How many sources do AI Overviews cite?
Typically 5-28 sources per response, though one documented response contained 95 links.
### Why does position #1 matter so much?
Position #1 provides 33% citation probability versus 13% at position #10. That's 2.5x the visibility from a single position improvement.
## Sources
- Ahrefs: 76% of AI Overview Citations Pull From Top 10 Pages
- Ahrefs: An Analysis of AI Overview Brand Visibility Factors
- Google: About AI Overviews
- Google: Common Crawlers Overview
## Related Research
- How AI Search Tools Source Information from Organic Search Results
- Brand Mentions vs Backlinks: What Matters for AI Visibility
- Schema Markup for AI Search: FAQPage and Structured Data
## Related Guides on SEO ProCheck
- Generative Engine Optimization (GEO) FAQ, the broader framework for ranking across AI search.
- E-E-A-T FAQ, the trust signals that drive both rankings and AI citations.
- Schema Markup FAQ, structured data that makes your content machine-liftable.
## Want your pages cited in AI Overviews?
I run technical SEO audits and content optimization that target exactly this, earning organic top-10 positions and the AI citations that follow. See how an advanced SEO audit works →
📡 The AI Crawler Visibility series
- The Forgotten HTML, what AI crawlers really see
- Which AI bots are you actually blocking? (GPTBot, ClaudeBot, Perplexity & more)
- SSR vs CSR, why rendering decides if AI can read you
- AI Crawler Comparison · Google AI Overviews · GEO FAQ
---
# How to Become a Cited Source in AI Answers
Source: https://seoprocheck.com/research/ai-search/how-to-get-cited-in-ai/
TL;DR
AI answer engines like ChatGPT, Claude, Perplexity, and Google AI Overviews cite sources that are reachable by their crawlers, that state a fact clearly enough to lift, that present content in a structured and extractable way, and that other reputable places corroborate. The practical work: allow AI crawlers in robots.txt, server-render your content, write self-contained answer passages under clear headings, publish original data with named authors, and earn mentions elsewhere. It rhymes with classic SEO but rewards clarity and citability over ranking tricks. Measure it through referral traffic from AI tools and manual prompt checks, and treat every citation as probabilistic rather than promised.
Getting a blue link to rank is a problem the SEO field has studied for two decades. Getting an AI system to name you as the source behind a sentence it generates is a younger problem, and it behaves differently. The good news is that most of the groundwork overlaps with technical SEO you may already do well. The work that follows assumes you want citations from the four engines people actually use in 2026: ChatGPT, Claude, Perplexity, and Google AI Overviews.
## How AI engines pick sources
These systems do not all work the same way, but the citation behavior they share is easier to reason about than it first appears. An answer engine either retrieves live pages at query time (Perplexity, ChatGPT with search, AI Overviews) or draws on what it learned during training, then attaches references to back up specific claims. To become one of those references, your page generally needs to clear four bars.
First, the system has to be able to reach your content. If its crawler cannot fetch the page, or fetches only an empty shell because the text loads later in the browser, you are invisible to it regardless of quality. Second, your page has to be the clear, attributable source of a fact. Engines prefer to cite the passage that states something plainly and looks like where the claim originated, not a page that hints at it. Third, the content has to be structured so a machine can lift a clean, self-contained chunk without dragging in noise. Fourth, the source has to look trustworthy and corroborated: other reputable sites reference it, the author and publisher are identifiable, and the claim is consistent with what the model sees elsewhere. Clarity and corroboration are doing most of the work here.
## The playbook
Four pillars, in the order that tends to unblock the most ground.
### Be reachable
- Allow AI crawlers in robots.txt. Named agents include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended for Google's AI uses. If your robots file blocks them, or your CDN or firewall silently drops them, you have opted out of being cited. Confirm the rules say what you intend.
- Serve real status codes and stable URLs. Soft 404s, redirect chains, and pages gated behind logins or aggressive bot challenges all cost you. A crawler that gets a clean 200 with the content present is one that can quote you.
- Do not assume blocking helps. Some publishers block AI crawlers on principle. That is a legitimate choice, but it removes you from the live-retrieval citation pool. Decide deliberately rather than by default.
### Be the source of a fact
- State claims plainly and own them. A sentence like "The figure was X as of [date], according to [our study]" is far more citable than the same fact buried in a paragraph of hedging.
- Publish things only you can publish. Original data, first-party benchmarks, definitions, and methodology are what engines reach for when a user asks a specific question. Aggregating other people's numbers puts you behind the people you aggregated.
- Date and attribute everything time-sensitive. Engines favor sources that show when a fact was true and who stands behind it.
### Be structured and extractable
- Write self-contained answer passages. Each section should answer one question in a way that stands alone if lifted out of the page. Lead with the answer, then support it.
- Use clear headings and lists. Descriptive H2s and H3s phrased like real questions, short paragraphs, and tables for comparisons all make extraction cleaner. Our note on content structure for AI goes deeper on passage design.
- Keep the meaning in the markup. Server-render the content and use semantic HTML and relevant schema so machines parse it the same way readers do. The broader case is in our piece on machine-readable web standards.
### Be trusted and corroborated
- Make authorship and entity signals explicit. Named authors with real credentials, a clear publisher identity, and consistent details across your site and the wider web help an engine decide you are a safe thing to cite. Our guide to entity SEO covers how to build that identity.
- Earn corroborating mentions. When other reputable sites reference your data or name your organization, you become the claim's apparent origin rather than one voice among many. This is the part you cannot fully control, and it is also the most durable.
- Stay internally consistent. A page that contradicts your other pages, or itself, gives an engine a reason to pick someone steadier.
## How it differs from classic SEO
Classic SEO optimizes for a ranked position on a results page, where a user then chooses to click. AI citation optimizes for being the quotable source behind a synthesized answer, where the engine chooses on the user's behalf and may show your name without sending a click at all. The overlap is real: crawlability, fast clean rendering, and authority help in both worlds. The divergence matters too.
Helps citation
- One claim answered cleanly per passage
- Original, attributable facts and data
- Plain statements an engine can quote verbatim
- Named authors and a clear publisher entity
Optimized for clicks, not citation
- Keyword-stuffed copy that reads for a ranking
- Answers withheld to push a click or scroll
- Long preambles before the actual point
- Recycled facts with no original source
The shift in mindset: stop writing to win a position and start writing to be the thing worth quoting. A page can earn citations without ranking first, and a page can rank first without ever being cited.
## How to measure it
Measurement here is coarser than classic rank tracking, so combine two honest signals rather than chasing one precise number.
- Referral traffic from AI tools. In your analytics, segment visits whose referrer is an AI surface (chatgpt.com, perplexity.ai, and similar). This undercounts, because many cited answers never produce a click, but a rising trend tells you citations are happening and converting to visits.
- Manual prompt checks. Periodically ask the four engines the questions your page answers, and record whether you are cited, which passage they pulled, and who they cite instead. This is tedious and worth automating into a small recurring routine. It is the only way to see citations that generate no traffic.
- Watch the gap. If competitors get cited for claims you also make, compare their passage structure and corroboration against yours. The difference is usually clarity or authorship, not topic coverage.
## Honest limits
Citation is probabilistic, not guaranteed. The same prompt can cite you today and someone else tomorrow, because these systems sample their output and refresh their retrieval. Doing everything in this playbook raises your odds materially; it does not buy you a slot. Engines also change their crawler names, their robots conventions, and their citation behavior without notice, so treat any specific tactic as current rather than permanent, and revisit your robots rules and structure on a schedule. Finally, more AI visibility can mean fewer clicks even as your name spreads, which is a strategic tradeoff worth naming out loud before you optimize for it.
## FAQ
Do I need to allow every AI crawler to get cited?
For live-retrieval engines like Perplexity and AI Overviews, yes: if their crawler is blocked, they cannot fetch and quote you. Allowing crawlers is a deliberate choice with tradeoffs, but blocking them removes you from that citation pool.
Will schema markup get me cited?
Schema helps machines parse your meaning and confirm authorship and dates, which supports citation. It is a clarity aid, not a switch. Clear prose, original facts, and corroboration carry more weight.
Does client-side rendering hurt my chances?
It can. If your text only appears after JavaScript runs in a browser, a crawler that fetches the raw HTML may see an empty page and have nothing to quote. Server-render the content that matters.
How fast will I see results?
Crawl and retrieval changes can show up within days for live engines, while corroboration and authorship signals build over weeks or months. Manual prompt checks will show movement before referral traffic does.
Want to know why AI engines skip your pages?
We audit crawlability, rendering, structure, and entity signals against how ChatGPT, Claude, Perplexity, and AI Overviews actually pick sources, then hand you a prioritized fix list.
Request an advanced SEO audit
---
# Meta Robots and X-Robots-Tag: The Complete Reference
Source: https://seoprocheck.com/indexing-crawl/meta-robots-x-robots-tag-reference/
TL;DR
The robots meta tag and the X-Robots-Tag HTTP header give the same instructions to search and AI crawlers, but they live in different places. The meta tag goes in the HTML of a page; the X-Robots-Tag rides in the HTTP response, so it can control non-HTML files like PDFs and images. Both control what crawlers do after they fetch a page: whether to index it, follow its links, show a snippet, cap a preview size, or drop it on a date. This is different from robots.txt, which controls whether crawlers fetch the page at all. The two most consequential directives in 2026 are noindex (keep a page out of the index) and nosnippet / max-snippet (which now also gate whether your text feeds AI Overviews and AI Mode). One trap dominates all others: if robots.txt blocks a URL, the crawler never reads the noindex on it, so the page can still get indexed. Block in robots.txt or noindex in the page, not both.
Robots meta directives are the most precise indexing controls you have. A single line tells Google or Bing exactly how to treat one page or one file: index it or not, follow its links or not, how large a preview to show, and how that content may appear in AI answers. This reference covers every directive that the major engines support in 2026, the difference between the meta tag and the HTTP header, the ordering and conflict rules that decide which directive wins, and the mistakes that quietly cost rankings.
If you are looking for the file that controls crawling rather than indexing, see the companion complete robots.txt reference. The distinction between those two systems is the single most misunderstood thing in technical SEO, so we start there.
## Meta robots vs X-Robots-Tag vs robots.txt
These three mechanisms are constantly confused because all three involve the word "robots." They do different jobs and operate at different stages of the crawl. The table below is the mental model worth memorizing.
Mechanism
Where it lives
Controls
Works on non-HTML?
Crawler must fetch the page?
robots.txt
One file at the domain root
Whether the crawler fetches the URL at all
Yes (any path)
No, it reads the rule before fetching
Robots meta tag
HTML (or body) of one page
Indexing, link following, snippets, previews
No, HTML only
Yes, it must read the HTML
X-Robots-Tag
HTTP response header for one resource
Same directives as the meta tag
Yes (PDF, image, video, any file)
Yes, it must read the response
The load-bearing row is the last column. Both the meta tag and X-Robots-Tag are read only after the crawler fetches the resource. robots.txt is read before. That is why blocking a URL in robots.txt and also putting noindex in its meta tag is self-defeating: the crawler obeys robots.txt, never fetches the page, never sees the noindex, and the URL can still appear in results as a bare link discovered from external sources.
## Every robots meta directive, explained
The robots meta tag is a single line in the head. The default behavior, if no tag is present, is identical to index, follow.
Directive
What it does
index / follow / all
The defaults. Stating them has no effect; they exist mainly to be explicit or to override an inherited directive.
noindex
Do not show this page in search results. The single most important directive. Removes the page from the index on next crawl.
nofollow
Do not follow any links on this page for crawl discovery or link equity.
none
Shorthand for noindex, nofollow.
nosnippet
No text snippet or video preview in results. In 2026 this also stops the content from being used in Google AI Overviews and AI Mode. A static image thumbnail may still appear.
max-snippet:[n]
Cap the text snippet at n characters. 0 behaves like nosnippet; -1 means no limit. The cap applies to AI features too unless a separate content licensing agreement exists.
max-image-preview:[setting]
Largest image preview allowed: none, standard, or large.
max-video-preview:[n]
Maximum seconds of video preview. 0 means a static image only; -1 means no limit.
noimageindex
Do not index images hosted on this page.
notranslate
Do not offer a translation of this page in results.
indexifembedded
Allow indexing of content embedded via iframe even when the page also carries noindex. Only works alongside noindex.
unavailable_after:[date]
Drop the page from results after the given date and time (RFC 822, RFC 850, or ISO 8601). Useful for time-limited offers and event pages.
noarchive / nocache
Google has retired noarchive along with its cached-page feature, so it has no effect there. Bing still honors both noarchive and its synonym nocache to suppress a cached copy.
## Directive support matrix: Googlebot vs Bingbot
Not every directive is honored by every engine, and the differences are exactly where audits go wrong. "Not documented" below means the engine's official documentation does not list the directive; in practice it is ignored. When a directive is unsupported, the engine treats it as unknown text and falls back to defaults, so an unsupported directive never breaks the supported ones next to it in the same tag.
Directive
Googlebot
Bingbot
noindex
Supported
Supported
nofollow
Supported
Supported
none
Supported (equals noindex, nofollow)
Not documented; use noindex, nofollow explicitly
nosnippet
Supported; also removes content from AI Overviews / AI Mode
Supported
max-snippet:[n]
Supported
Supported (adopted the max-* family in 2020)
max-image-preview
Supported
Supported
max-video-preview
Supported
Supported
noimageindex
Supported
Not documented
notranslate
Supported
Not documented
indexifembedded
Supported (Google-only directive, introduced 2022)
Not supported
unavailable_after:[date]
Supported
Not documented
noarchive / nocache
No effect; Google retired the cached-page feature
Supported (both spellings)
nositelinkssearchbox
Supported
Not applicable
## The nosnippet, max-snippet, and data-nosnippet controls
Snippet controls used to be cosmetic. They now decide whether your words feed AI answers. Google has confirmed that nosnippet and max-snippet govern not only the classic blue-link snippet but also whether content can appear in AI Overviews and AI Mode. If you set nosnippet, you remove the page from those AI surfaces entirely, which is a real trade-off: less exposure in AI answers, but also no chance of being summarized without a click. This is the lever to think hardest about in 2026.
There is a finer instrument than the page-wide tag. The data-nosnippet HTML attribute excludes a specific block of text from snippets while leaving the rest of the page eligible. Apply it to a span, div, or section:
This sentence can appear in a snippet.
This phrase will never appear in a snippet.
Two caveats. The element must be valid HTML with a proper closing tag, and the attribute should be present when the element is first created in the DOM. Adding data-nosnippet with JavaScript after render is unreliable.
## X-Robots-Tag for non-HTML files
The meta tag only works in HTML, because there is no in a PDF, a JPEG, or a video file. To keep those out of the index you set the same directives in the HTTP response with the X-Robots-Tag header. Every robots meta value works identically here.
The simplest case, a single directive on one response:
X-Robots-Tag: noindex
To keep every PDF on a site out of search, set the header on all PDF responses. On Apache, target the file type in your configuration or .htaccess:
Header set X-Robots-Tag "noindex, nofollow"
On Nginx, add the header inside a location block matched to the extension:
location ~* .pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
You can also target a specific crawler by prefixing its name, and you can stack multiple headers in one response:
X-Robots-Tag: googlebot: noindex, nofollow
X-Robots-Tag: otherbot: noindex
X-Robots-Tag: unavailable_after: 25 Dec 2026 15:00:00 PST
One detail that saves hours of debugging: header names, crawler names, and directive values are all case-insensitive, but the header must actually appear on the response. Use a header-inspection tool or your browser network panel to confirm the directive is being sent before you assume it is working.
## Targeting specific crawlers
Replace robots with a named user agent to scope a rule to that crawler. A generic robots tag applies to all of them; a named tag overrides the generic one for that crawler. Use separate tags for separate crawlers:
For the full catalog of which bots read which signals, including the AI crawlers, see the AI crawler map.
## AI-specific directives: noai, noimageai, and Google-Extended
A separate family of directives addresses AI training rather than search indexing. noai asks AI crawlers not to use a page's content for model training; noimageai does the same for images. They are placed in the same robots meta tag or X-Robots-Tag header:
Be clear-eyed about their status. noai and noimageai originated as a community proposal, not a formal standard from any search engine. There is no specification and no enforcement; compliance is voluntary. Well-behaved crawlers from major AI companies increasingly respect them, but a scraper that ignores them faces no consequence. Treat them as a polite request, not a wall.
One common mix-up to settle: Google-Extended is not a meta directive. It is a robots.txt user-agent token that controls whether your content trains Gemini and related Google AI products. You cannot set Google-Extended in a meta tag; it belongs in robots.txt. This is exactly the kind of cross-system confusion the comparison table above is meant to prevent.
## Ordering and conflict rules
When directives disagree, the rule is simple: the most restrictive directive wins. A few worked examples:
- max-snippet:50 together with nosnippet resolves to nosnippet, because no snippet is more restrictive than a capped one.
- A generic robots tag and a named googlebot tag both present: Googlebot obeys its named tag for any directive it specifies, and falls back to the generic tag for directives the named tag omits.
- A meta tag and an X-Robots-Tag header on the same HTML page: the engine combines them, again taking the most restrictive value for each directive.
The exception worth remembering is indexifembedded, which is not a conflict but a deliberate pairing: it only does anything when noindex is also present, relaxing it for embedded content.
## How to verify what the crawler actually sees
Most robots-directive incidents are not exotic; they are directives that were never actually served, or were served to crawlers but not to the browser you tested with. Verify at three layers before declaring a directive live.
1. Check the HTTP header from the command line. Browser dev tools work, but curl is faster and shows exactly what a bot receives, with no extension or cache interference:
curl -sI https://example.com/whitepaper.pdf | grep -i x-robots-tag
If nothing comes back, the header is not being sent, whatever your server config claims. Re-test with a Googlebot user-agent string (curl -sI -A "Googlebot") as well: CDNs and security layers sometimes serve different headers to bots.
2. Check the rendered HTML, not just view-source. Google reads the robots meta tag from the rendered DOM. An SEO plugin, a tag manager, or a framework hydration step can inject or remove a meta robots tag after initial HTML load. In Google Search Console, run the URL through URL Inspection, then open View crawled page → HTML and search the source for name="robots". The Indexing allowed? row in the same report states plainly whether Google detected a blocking directive on its last crawl.
3. Find every affected page in bulk. The report path in Search Console is Indexing → Pages → "Excluded by 'noindex' tag". Read that list on every audit; it is where accidental site-section noindexes surface. If a page you care about appears there, you have found the incident. For a crawler-side sweep, Screaming Frog reports both the meta tag and the X-Robots-Tag header in the Directives tab, which catches the header-based noindexes that HTML-only greps miss.
## Common mistakes
- Blocking and noindexing the same URL. robots.txt stops the fetch, so the noindex is never read. Pick one. To remove an indexed page, allow crawling and serve noindex until it drops out, then block if you wish.
- Site-wide noindex left over from staging. A noindex applied during development and never removed at launch is a classic traffic-zeroing bug. Audit the live header and meta tag on day one.
- Expecting noindex to free crawl budget. A noindexed page is still crawled to read the directive. It saves index space, not crawl budget.
- Using noindex where a 410 is correct. For permanently removed content, a status code is cleaner than a noindexed placeholder. See 404 vs 410 status codes for when to use each.
- Assuming noarchive still works on Google. It does not; the cached-page feature is gone. The directive is only meaningful on Bing now.
- Setting nosnippet without intending to leave AI surfaces. Because nosnippet now also removes content from AI Overviews and AI Mode, applying it broadly can cut AI visibility you wanted to keep.
## FAQ
Does noindex remove a page immediately?
No. The engine must re-crawl the page to see the directive, then process the removal. It usually takes days to a few weeks. To force a faster review, request indexing of the URL so the crawler returns sooner.
Should I use the meta tag or the X-Robots-Tag header?
Use the meta tag for HTML pages because it is easy to set per page. Use X-Robots-Tag for anything that is not HTML, such as PDFs, images, and videos, or when you want to apply a rule across many files at the server level.
Do noai and noimageai actually stop AI training?
Only for crawlers that choose to honor them. They are a voluntary convention, not an enforced standard. Reputable AI companies increasingly respect them, but they offer no technical protection against scrapers that ignore them.
Is Google-Extended a meta robots directive?
No. Google-Extended is a robots.txt user-agent token that governs AI training for Google products. It cannot be placed in a meta tag or X-Robots-Tag header.
What happens if a meta tag and the HTTP header conflict?
The engine combines them and applies the most restrictive value for each directive. There is no concept of one overriding the other wholesale.
Does a long-standing noindex eventually stop links from being followed?
Yes. Google has stated that a page kept in noindex for a long time is eventually treated as noindex, nofollow, because pages dropped from the index are crawled less and their links stop being processed. If you rely on a noindexed hub page purely to pass internal links, that arrangement decays; link to the targets from indexable pages instead.
How do I add a noindex tag in WordPress?
Per page, use your SEO plugin: in Yoast it is the Advanced section of the post sidebar ("Allow search engines to show this content?"), in Rank Math the Advanced tab's robots meta checkboxes. Site-wide, Settings → Reading → "Discourage search engines from indexing this site" outputs a global noindex, which is also the setting to check first when a WordPress site vanishes from Google after a relaunch.
Not sure your indexing directives are doing what you think?
A single stray noindex or a robots.txt block over a noindexed page can quietly remove pages from search. An audit catches it.
Request an Advanced SEO Audit
---