Which AI Bots Are You Actually Blocking? (GPTBot, ClaudeBot, Perplexity & More)

"Block AI bots" sounds like one switch. It is not. Every AI company runs several crawlers with different jobs, and blocking the wrong one is how sites accidentally disappear from ChatGPT, Claude and Perplexity answers while believing they only opted out of model training. If you are going to make a decision about AI crawlers — and you should — make it the right one.

Every AI bot has one of three jobs

• Training — collects content that may train future models. Blocking it affects whether you contribute to the model, not whether you show up in answers today.
• Search / indexing — builds the live index the assistant retrieves from. Block this and you vanish from that product's cited answers.
• User-fetch — grabs a specific URL when a human asks the assistant to read it. Block this and the assistant cannot open your page on request.

That distinction is the whole game. Here is who's who.

The 2026 AI crawler field guide

Sources: OpenAI's crawler docs and Anthropic's documented agents.

The mistake almost everyone makes

Someone reads a scary headline about AI scraping, drops a blanket Disallow for "AI bots," or a security plugin/WAF starts 403'ing anything that looks like a bot. The result: you block OAI-SearchBot, Claude-SearchBot and PerplexityBot too — the exact crawlers that decide whether you get cited — while thinking you only said no to training. (For real, reproducible examples of big sites doing this, see The Forgotten HTML.)

The two robots.txt recipes that actually make sense

1. "I want AI citations, just not to train the models." Allow the search and user bots; block the training ones:

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /

# leave these ALLOWED (default):
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot

2. "Block everything AI." Disallow all of the above — but understand you are opting out of AI-answer visibility entirely, and accept the traffic trade-off.

One caveat: robots.txt is a request, not a wall. Some user-fetch bots ignore it, and a firewall can override it in either direction. If it must be enforced, do it at the WAF/CDN.

How to check what you're actually blocking

Test each user agent directly — a 403 means you're blocking it, content means you're not:

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot PerplexityBot; do
  echo -n "$ua: "; curl -s -o /dev/null -w "%{http_code}n" -A "$ua" https://yoursite.com/
done

Then read your robots.txt, and check your CDN/WAF bot-management rules — that is where accidental blocks usually live.