ClaudeBot Blocked

What This Check Actually Looks At

This check reads your site's robots.txt file and looks for a rule that disallows the ClaudeBot user agent. If it finds a block, either a blanket Disallow: / under User-agent: ClaudeBot or a narrower path block, it flags this issue. ClaudeBot is the crawler Anthropic operates to gather web content, primarily for training Claude models. Anthropic also runs Claude-User, which fetches pages in response to a live user request inside Claude (for example when someone pastes a URL or asks Claude to look something up), and Claude-SearchBot, which supports Claude's search features. These are separate user agents and each one respects its own robots.txt rule independently, so blocking ClaudeBot does not automatically block Claude-User or vice versa. If you only see "ClaudeBot Blocked" here, check whether you meant to block all three or just one.

Why This Matters

I will say the quiet part: blocking ClaudeBot is not an SEO penalty in the Google-rankings sense. Nobody is going to lose position 3 on a head term because ClaudeBot got a 403. This check exists because AI visibility is becoming its own channel, separate from classic organic search, and site owners are increasingly asked "does our content show up when someone asks Claude about this topic." If ClaudeBot cannot crawl you, your pages cannot be used in training data going forward, and if Claude-User or Claude-SearchBot are also blocked, your pages cannot be fetched live either, which matters more immediately for anything Claude does at answer time. For publishers who monetize traffic or want brand presence in AI answers, this is a real tradeoff. For sites that explicitly do not want their content used for model training, the block is doing exactly what it is supposed to do. There is no universal right answer here, only an informed one.

How To Detect It

You do not need a paid tool for this one, though several help confirm the full picture.

• Manual check: visit https://yourdomain.com/robots.txt in a browser and search the file for "ClaudeBot", "Claude-User", "Claude-SearchBot", and "anthropic-ai". Look at what directive sits under each.
• Screaming Frog: the built in robots.txt tester (Configuration > robots.txt > Custom) lets you paste your file and test specific user agents and URLs to see exactly what is allowed or blocked.
• Server logs: grep your access logs for the ClaudeBot user agent string to see whether it was actually trying to hit you and getting turned away, versus never showing up at all. A block that is working shows repeated attempts with no successful crawls after the rule went live.
• Anthropic's own documentation: the Claude support site publishes the current list of user agent names and recommended robots.txt syntax, worth checking periodically since bot names and behavior get updated.

How To Fix It

1. Open your robots.txt file, usually at the site root and editable through your CMS, a robots.txt plugin, or directly on the server.
2. Find the block targeting ClaudeBot. It will look something like User-agent: ClaudeBot followed by Disallow: /.
3. Decide what you actually want. If you want to allow crawling entirely, delete the block or change Disallow: / to Disallow: (empty value means allow everything).
4. If you want fine grained control, keep some paths blocked (login pages, admin areas, thin filter pages) and allow the rest, the same way you would for Googlebot.
5. Repeat the review for Claude-User and Claude-SearchBot separately since they are independent user agents with independent rules.
6. Save, deploy, and re-fetch /robots.txt to confirm the change is live. There is no submission or resubmission process, ClaudeBot simply reads the current file on its next visit.
7. Watch your server logs over the following days to confirm ClaudeBot requests are now returning 200s instead of being skipped.

What Good Looks Like

A deliberate, documented choice. Either your robots.txt clearly allows ClaudeBot because you have decided AI visibility matters to your business, or it clearly blocks ClaudeBot because you have decided training-data use is not something you consent to, and that decision was made by someone with the authority to make it, not left over from a boilerplate robots.txt template nobody reviewed. What you don't want is an accidental block from a WordPress security plugin's "block AI bots" toggle that got flipped on without anyone realizing it also cuts off Claude-User's ability to fetch your page when a customer asks Claude about you directly.

Frequently Asked Questions