Claude

What Claude is (for site owners)

Claude is Anthropic’s AI assistant, and with web search enabled it reads live pages mid-conversation and answers with source links. For a site owner that makes Claude two separate things to manage: a training pipeline that may absorb your content into future models, and a real-time reading channel where your page gets pulled in front of a user the moment they ask the right question.

The stakes are the same asymmetry as every AI platform: the training side gives you nothing measurable, while the live side produces citations and referral visits, and one careless robots.txt line can kill both.

How Claude gets web content

Anthropic operates distinct agents with separate robots.txt tokens:

• ClaudeBot: the scheduled crawler collecting public pages for training future Claude models.
• Claude-User: the fetcher that retrieves a page while a human is actively talking to Claude, either through web search or a pasted URL.
• Claude-SearchBot: Anthropic’s indexing agent, improving how Claude’s search finds and ranks pages.

Same-named hits in your logs therefore mean very different things, and the ClaudeBot/Claude-User distinction is the one worth internalizing:

ClaudeBot vs Claude-User: what each hit means

Put bluntly: ClaudeBot hits are Anthropic’s interest in your content, Claude-User hits are users’ interest. Only one of those pays you back this quarter.

Example: reading a week of logs

Here’s the pattern from a mid-size B2B blog, one week, filtered to Anthropic agents:

  412  ClaudeBot          (spread across ~400 URLs, sitewide sweep)
   38  Claude-User        (31 of them on /integrations/ and /pricing/)
    9  Claude-SearchBot

The 412 ClaudeBot hits are corpus collection, interesting, worth nothing today. The 38 Claude-User hits concentrated on two commercial pages are the signal: dozens of real conversations this week needed those pages. That’s a topic-demand insight most keyword tools can’t give you, sitting in a log file nobody reads.

Robots.txt for Claude

The setup for a site that wants visibility but not training inclusion:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

And if you simply want everything open:

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

How to check it

1. Count each agent separately: the ratio is the diagnosis:

   grep -ioE "claudebot|claude-searchbot|claude-user" access.log | sort | uniq -c

   Lots of ClaudeBot and zero Claude-User means Anthropic reads you but users’ questions never surface you.

2. Find which pages Claude-User reads:

   grep -i "claude-user" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

   This is a free demand report: these topics are being asked about right now.

3. Audit robots.txt for all three tokens, including inheritance from User-agent: * rules.
4. Check response codes for these UAs: 403s or challenge pages mean your WAF is making the decision, not your robots file.
5. Filter analytics for referrer claude.ai and trend it monthly.

Common mistakes

• One wildcard rule for anything containing “Claude”. A CDN pattern meant to stop training crawls also kills Claude-User, and with it every citation. Match exact tokens, and verify with the Claude visibility check.
• Reading a ClaudeBot hit as an impression. It isn’t. It’s corpus collection. Impressions look like Claude-User.
• Guessing token names. Claude-Bot, ClaudeAI and Anthropic-AI in robots.txt match nothing current; typos silently allow what you meant to block.
• Judging Claude’s view of you by its training cutoff. With search on, Claude reads today’s page. Stale model knowledge doesn’t matter if the live fetch path is open.
• Serving JavaScript-dependent content. Claude’s fetchers read delivered HTML; content that only exists post-hydration is unreadable to them.

FAQ

Does Claude send referral traffic?

Yes: answers produced with web search include source links, and visits show up under the claude.ai referrer. Volume is smaller than ChatGPT’s but comes from the same high-intent research behavior.

If I block ClaudeBot, does Claude stop citing me?

No. Citations come from live retrieval by Claude-User and search indexing by Claude-SearchBot. Blocking ClaudeBot only opts your content out of future training collection.

How do I get cited by Claude more often?

Open access for Claude-User and Claude-SearchBot, then answer-first structure with verifiable specifics, the same qualities that win other engines, detailed in Claude SEO: getting cited in Claude. Track the outcome as part of overall AI visibility.

Is ClaudeBot heavier on servers than other AI crawlers?

It has a reputation for aggressive crawl bursts on large sites. If load is the concern, rate-limit at the server level rather than banning it, behavior across bots is compared in the AI crawler comparison guide.

Does Anthropic honor robots.txt?

Anthropic documents that its crawlers respect robots.txt directives for their respective tokens. As with all AI bots, enforcement of impostors faking the UA is your job: verify suspicious hits by IP before drawing conclusions.