ClaudeBot

What ClaudeBot is

ClaudeBot is Anthropic's primary web crawler — the agent that collects publicly available pages that may end up in the training corpus for future Claude models. Whether you allow it decides whether your content can shape what Claude "knows" natively, which is a different question from whether Claude can cite you in a live answer.

The user agent in your logs

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

That contact email in the UA string is real — Anthropic invites site owners to write in about crawl behavior. Few bot operators do that; it's a useful tell when you're separating the genuine article from spoofers.

One vendor, three agents: know which one you're blocking

Anthropic documents three distinct agents, and lumping them together is the single most expensive mistake site owners make with this vendor. Here's the matrix:

You'll still find legacy tokens like anthropic-ai and claude-web in old robots.txt files around the web. Anthropic's current documentation centers on the three agents above — if your robots.txt only names the legacy tokens, you may not be expressing the policy you think you are.

Crawl pressure: a real history lesson

ClaudeBot earned a reputation in 2024 when several site operators — iFixit most loudly — went public about aggressive crawl volumes, with reports of hundreds of thousands of hits in a day. Anthropic responded and behavior settled down, but the episode left two durable lessons. First, robots.txt is your policy layer, not your capacity layer: if any bot's request rate hurts you, rate-limit at the server or CDN and don't wait for the operator to notice. Second, keep the receipts — a dated log extract showing request volume is what turns a complaint email to that UA-string contact address into a fixed problem.

How to check ClaudeBot activity

1. Search your access logs for all three tokens, not just one: grep -iE "ClaudeBot|Claude-User|Claude-SearchBot" access.log.
2. Break the hits down by status code. 200s mean it's reading you; 403s mean something upstream (WAF, CDN bot rules) is blocking it regardless of robots.txt.
3. Look at which paths it requests. Heavy crawling of parameter junk or faceted URLs is a crawl-hygiene problem worth fixing for every bot, not just this one.
4. Fetch your robots.txt and confirm which group each Anthropic agent actually matches — a specific ClaudeBot group overrides your * group for that agent.
5. Recheck after any CDN or security-plugin update. Bot-management defaults change quietly, and AI crawlers are a favorite target.

Common mistakes and fixes

• Blocking ClaudeBot and expecting to vanish from Claude's answers. Training and retrieval are separate pipes. Live citations come via Claude-SearchBot and Claude-User; the training crawler has nothing to do with this week's answers. Fix: block the agent that maps to your actual objection.
• Only using the legacy anthropic-ai token. Fix: name the three current agents explicitly and keep the legacy lines if you want belt-and-suspenders.
• Silent WAF blocks. Your robots.txt says welcome, your firewall says 403, and you never notice because nothing errors on your side. Fix: log-audit status codes served to each AI agent quarterly — the AI bot user-agents guide shows what a healthy pattern looks like.
• Assuming ClaudeBot renders JavaScript. It doesn't render like Googlebot does; content that only exists after client-side hydration is invisible to it. Fix: server-side render anything you want AI systems to read — details in SSR vs CSR for AI crawlers.
• Setting policy once and never revisiting. Agent names, defaults, and your own licensing position all drift. Fix: calendar a twice-yearly review against Anthropic's docs.

FAQ

Will blocking ClaudeBot remove my content from Claude today?

No. Anything already absorbed into a trained model stays there until that model is retired. Blocking is forward-looking only — it keeps future crawls out of future training runs.

What's the practical difference between ClaudeBot and Claude-User?

Scale and intent. ClaudeBot crawls broadly on Anthropic's initiative; Claude-User fetches one URL because a human asked Claude to look at it. Blocking the second mostly punishes your own readers.

Does ClaudeBot obey robots.txt?

Anthropic says yes for its crawlers, and log studies generally bear that out. Verify on your own logs anyway — that's the only compliance report that matters for your site.

How does ClaudeBot compare to PerplexityBot and GPTBot?

Same family, different purposes per agent — Perplexity's crawler feeds an answer-engine index rather than model training, which changes the tradeoff. Start with the PerplexityBot entry, then the full three-way comparison.

Should a small publisher allow it?

If AI visibility is worth anything to you, allow Claude-SearchBot and Claude-User at minimum. ClaudeBot itself is a judgment call between future-model presence and content-control principles — just make it deliberately.