The AI Crawler Map: Every Bot Reading Your Site in 2026

Open your server logs for a single day and you will find a parade of robots you never invited: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bytespider, and a dozen more. Each one wants something different, each one reacts differently when you block it, and most site owners treat all of them as a single switch labelled "AI." That is exactly how good pages quietly vanish from AI answers while their owners congratulate themselves on "blocking the scrapers."

This is the map I wish every client had before they touched their robots.txt. It covers every major AI user-agent crawling the open web in 2026, what each one actually does, and the specific thing that breaks when you say no.

💡 The only mental model you need: three jobs

Every AI crawler on the planet is doing one of three jobs. Sort each bot into the right bucket and the entire "should I block this?" question answers itself.

• Training crawlers harvest text to teach a model. They visit once in a while, take a copy, and leave. Blocking them protects your content from future model training but costs you essentially zero live visibility, the model was never going to link back to you anyway.
• Search & answer crawlers build the live index an assistant searches when a user asks a question. These are the bots that cite you. Block one and you remove yourself from that engine's answers, the AI equivalent of deleting yourself from Google's index.
• User-fetch agents retrieve a single page in real time because a specific human asked the assistant to read that URL. Block them and "summarise this link" requests to your pages simply fail.

The expensive mistake is treating all three as the same threat. People who paste a "block all AI" snippet to stop model training also quietly delete themselves from ChatGPT Search, Claude, and Perplexity, the surfaces actually sending qualified readers in 2026.

🤖 The map: every major AI bot in 2026

Grouped by operator. "Honors robots.txt" reflects each operator's published policy; the asterisks matter, and I unpack them in the limitations section.

A "Token" entry is a directive in robots.txt, not a crawler with its own user-agent, Google-Extended and Applebot-Extended govern training only and never affect how you rank in their search products.

🚦 What actually breaks when you block each type

Here is the part the "block all AI" crowd never thinks through. The three jobs carry wildly different costs.

🔧 robots.txt recipes you can paste today

Multiple User-agent lines can share one Disallow block, that is valid syntax, not a hack.

Recipe 1, Stay in AI answers, opt out of model training. The configuration most publishers actually want:

# Let the answer engines cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Recipe 2, Block everything AI, accept the cost. Legitimate for paywalled or licensing-sensitive sites, as long as you know you're leaving every AI answer surface:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Notice what is not on that list: Googlebot and Bingbot. You cannot opt out of Google AI Overviews or Bing Copilot without also leaving classic search, those answer features are fed by the same index that ranks you. Google-Extended only governs Gemini training, not AI Overviews. Anyone who tells you otherwise is selling something.

🚧 What this map does not mean

Honesty is the difference between a field guide and a fairy tale. Five caveats you should hold onto:

• robots.txt is a request, not a firewall. Compliance is voluntary. Independent monitoring (notably by Cloudflare in 2025) reported that some operators, Perplexity and ByteDance's Bytespider among the most cited, fetched pages that disallowed them. The asterisks in the table flag exactly those disputed cases. If you need a hard block, enforce it at the edge with WAF rules and verified IP ranges, not robots.txt alone.
• User-agents can be spoofed. Anyone can send a request claiming to be ClaudeBot. The real operators publish IP ranges (and some support reverse-DNS verification) so you can confirm a bot is genuine before trusting, or rate-limiting, it.
• Allowing a crawler is necessary, not sufficient. Letting OAI-SearchBot in does not guarantee a citation. It makes you eligible. Whether you're actually quoted still depends on relevance, clarity, and whether your content survives rendering, which is a separate problem entirely.
• The list changes. These operators add and rename agents regularly (the -SearchBot variants are relatively new). Treat any AI-crawler list, including this one, as a snapshot, re-check the official docs before a migration.
• Blocking is reversible, lost ground is slower. If you disallow a search crawler, re-allowing it doesn't instantly restore your presence, the engine has to re-crawl and re-index on its own schedule.

✅ Audit your own site in ten minutes

Don't take my table on faith, go look at what's actually happening on your server. The replication steps:

1. Read your live robots.txt. Visit yourdomain.com/robots.txt and find every User-agent line. Map each one to the table above and ask: am I blocking a training bot (fine) or an answer bot (probably a mistake)?
2. Grep your access logs for one week. Filter for the user-agents in the table. You'll see who actually visits, how often, and which ones you're serving 200 versus 403.
3. Check for accidental edge blocks. Many sites block AI bots at Cloudflare or a WAF without realising it, the "Block AI Scrapers" toggle is one click. Confirm your answer crawlers aren't being stopped before they reach robots.txt.
4. Test a live fetch. Ask ChatGPT or Claude to open one of your URLs. If it returns "couldn't access," your user-fetch agents are blocked somewhere in the stack.
5. Decide on purpose. Write down, per operator, whether you want to be trained on, indexed for answers, or fetched live, then make robots.txt say exactly that. Default-by-accident is how good pages disappear.