AI Summary
Claude cites pages it can retrieve through Brave Search, so Claude visibility is won at three gates: be crawlable by ClaudeBot, be present in Brave's independent index, and be structured so a passage can be lifted and attributed. Google rankings do not carry over, because Brave builds its own index from scratch.
- ClaudeBot does not execute JavaScript. If your content is client-rendered, Claude sees an empty shell.
- Allow
ClaudeBotin robots.txt, and useCrawl-delayto manage its rate: it is the only major AI crawler that honors the directive. - Check Brave, not Google. A page absent from search.brave.com cannot be retrieved no matter where it ranks elsewhere.
- Write literal question headings with a complete answer underneath, put comparison data in real HTML tables, and keep every section self-contained.
- A WAF or CDN rule quietly serving 403s to ClaudeBot is a common invisible blocker that no console will report.
Quick answer: To get cited by Claude, make your pages visible in Brave Search (Claude's search provider), allow ClaudeBot in robots.txt, and serve your content as server-rendered HTML, because ClaudeBot does not execute JavaScript. Then structure pages for extraction: direct answers under literal question headings, data in real HTML tables, and specific attributable facts rather than generalities. Google rankings do not transfer: Brave's independent index is the visibility layer that matters here.
Claude uses Brave Search as its search infrastructure, giving it access to Brave's independent index of 30+ billion pages. According to Anthropic's announcement, this makes Claude one of the few AI assistants with access to a truly independent search index not derived from Google or Bing.
Key Facts: Claude Search Infrastructure
| Aspect | Detail | Source |
|---|---|---|
| Search provider | Brave Search | Anthropic |
| Index size | 30+ billion pages | Brave Search |
| Crawler name | ClaudeBot | Anthropic |
| Robots.txt compliance | Respects directives | Anthropic |
| Crawl-delay support | Yes (unique among AI crawlers) | Anthropic |
| IP ranges published | No | Anthropic |
Why Brave Search Matters
According to Brave's documentation, Brave Search operates independently from Google and Bing. It builds its own index from scratch, which means:
- Sites that rank well in Google may not rank in Brave
- Brave may surface different results than Google or Bing
- Independent index means independent visibility opportunities
- Less dominated by the same large publishers
How to Optimize for Claude Citations
1. Verify Brave Search Indexing
Check if Brave has indexed your important pages:
- Search your domain in Brave: search.brave.com
- Check individual page URLs
- Compare Brave results to Google results for your target keywords
- If pages are missing, ensure your site is crawlable and has proper technical SEO
2. Allow ClaudeBot in Robots.txt
According to Anthropic's documentation, ClaudeBot is unique among AI crawlers in supporting the Crawl-delay directive. Example configuration:
User-agent: ClaudeBot Allow: / Crawl-delay: 10
Legacy directives for ANTHROPIC-AI and CLAUDE-WEB are also honored.
3. Ensure Server-Side Rendering
According to Vercel's research, ClaudeBot cannot execute JavaScript. If your site relies on client-side rendering (React, Vue, Angular without SSR), Claude will see a blank or incomplete page.
- Implement server-side rendering (SSR)
- Use static site generation (SSG) for content pages
- Pre-render critical pages for crawlers
- Test with JavaScript disabled to see what crawlers see
4. Structure Content for Extraction
Claude synthesizes information from retrieved pages. Content optimized for extraction includes:
- Clear heading hierarchy (H1, H2, H3)
- Direct answers to questions near headings
- Tables with structured data
- FAQ sections with question-answer format
- Specific facts and statistics
5. Citation-Friendly Content Properties
Claude cites sources when it synthesizes an answer from retrieved pages, and the pages that get quoted share observable properties. This table maps each property to why it earns the citation and how to implement it:
| Content property | Why Claude cites it | How to implement |
|---|---|---|
| Direct answer under a literal question heading | The retrieved passage answers the user's question without reconstruction, making it the easiest source to quote and attribute | Phrase the H2/H3 as the question users ask; put a complete 2-3 sentence answer immediately below it |
| Specific, attributable facts | An assistant citing sources needs concrete claims to anchor the citation; vague prose gives nothing to attribute | Name numbers, dates, tools, and sources inline ("supports Crawl-delay, per Anthropic's documentation") instead of "many experts say" |
| Data in real HTML tables | Tabular facts survive extraction intact and answer comparison questions in one lift | Use <table> with header rows, not styled <div> grids or images of tables |
| Content present in initial HTML | ClaudeBot does not execute JavaScript; anything client-rendered simply does not exist to it | SSR or SSG for content pages; verify with JavaScript disabled or a text-mode fetch |
| Self-contained sections | Retrieval pulls passages, not whole pages; a section that depends on context elsewhere loses its meaning | Make each H2 block understandable standing alone, restating the subject instead of writing "as mentioned above" |
| Dated, maintained pages | For time-sensitive queries, a visibly current page is a safer synthesis source than an undated one | Show a visible updated date and keep facts current; stale claims get corrected around you, not cited |
| Indexed and ranking in Brave | Claude can only retrieve what its search layer returns; Brave visibility is the gate before any content quality matters | Verify site and page presence at search.brave.com; fix crawlability before polishing prose |
Testing Whether Claude Can See and Cite You
A practical verification loop, in the order failures actually occur:
- Fetch your page the way ClaudeBot does.
curl -sA "ClaudeBot" https://yoursite.com/page/ | grep -i "distinctive phrase". If your key content is not in that raw HTML response, no amount of on-page optimization matters, because the crawler never runs your JavaScript. - Check Brave indexing per page. Search the exact URL and your target queries at search.brave.com. A page absent from Brave cannot be retrieved by Claude's web search regardless of its Google position.
- Confirm ClaudeBot is reaching you.
grep -c ClaudeBot access.login your server logs, and check the status codes those hits receive: a firewall or bot-protection rule silently serving 403s to ClaudeBot is a common invisible blocker. If raw logs are new territory, the log file analysis primer covers field layout and bot verification. - Ask Claude directly. With web search enabled, ask the questions your page answers and note which domains get cited. Repeat across sessions before drawing conclusions, since retrieved sources vary per query phrasing.
- Iterate on the losing comparison. When a competitor gets the citation, fetch their page the same way and compare structure: usually the difference is a tighter direct answer or a table where you have prose.
Diagnosing the three gates when you are not being cited
"Claude never cites us" is not one problem, it is three, and they need different fixes. Work them in order, because a failure at gate one makes everything downstream untestable:
| Gate | Test | Failure looks like | Fix |
|---|---|---|---|
| 1. Fetchable | curl -sA "ClaudeBot" URL and grep for your key sentence | Empty shell, a JS bundle, a 403, or a challenge page | SSR or SSG the content; allowlist ClaudeBot in the WAF or CDN bot rules; confirm robots.txt does not disallow it |
| 2. Indexed | Search the exact URL, then your target query, at search.brave.com | URL returns nothing, or the site appears but the page does not | Standard technical SEO: internal links to the page, a clean sitemap, no noindex, no canonical pointing elsewhere |
| 3. Extractable | Read your own H2 block cold, with no surrounding page | The section only makes sense in context, or has no quotable claim in it | Rewrite the opening of each section as a standalone answer; convert comparison prose into a table; add named, dated facts |
Gate one is where most sites actually fail, and it is the cheapest to check. Note that gates one and two involve different controls: robots.txt decides whether ClaudeBot may fetch a URL at all, while noindex in the meta robots tag or X-Robots-Tag header decides what a search index does with the page afterwards. Blocking a URL in robots.txt also hides any noindex you placed on it, since the crawler never gets far enough to read the directive.
ClaudeBot Crawling Behavior
ClaudeBot has been noted for aggressive crawling. 404 Media reported that iFixit saw ClaudeBot hit their site nearly 1 million times in 24 hours in 2024.
To manage ClaudeBot's crawl rate, use the Crawl-delay directive:
User-agent: ClaudeBot Crawl-delay: 30
This tells ClaudeBot to wait 30 seconds between requests. ClaudeBot is the only major AI crawler that respects this directive.
Brave Search vs Google: Key Differences
| Aspect | Brave Search | |
|---|---|---|
| Index source | Independent | Proprietary |
| Index size | 30+ billion pages | Hundreds of billions |
| Ranking factors | Less documented | Extensively documented |
| Large publisher dominance | Less pronounced | Significant |
| Real-time indexing | Slower | Fast |
Frequently Asked Questions
Not directly. Claude uses Brave Search, which has its own independent index. Your Brave rankings matter more than Google rankings for Claude visibility.
Search your target keywords at search.brave.com and note where your pages appear. Compare to your Google rankings to identify gaps.
If you want Claude visibility, yes. You can use Crawl-delay to manage crawl rate. Blocking ClaudeBot may reduce your visibility in Claude's responses.
According to Anthropic's documentation, they use service provider public IPs rather than dedicated ranges. This makes verification harder but doesn't change robots.txt compliance.
There is no documented confirmation from Anthropic that structured data influences Claude's citations. What demonstrably matters is that the content exists in the initial server-rendered HTML with clear heading structure, since ClaudeBot does not execute JavaScript. Schema is worth keeping for Google, and it does not hurt here, but do not expect it to substitute for extractable on-page content.
Grep your access logs for ClaudeBot and look at the status codes those requests receive. A stream of 403 or 429 responses means your WAF, CDN bot-management rules, or rate limiter is turning the crawler away even though your robots.txt allows it, a silent visibility killer, because nothing in any console will report it. Remember the legacy ANTHROPIC-AI and CLAUDE-WEB tokens are also honored in robots.txt.
Sources
- Anthropic: Claude Now Has Web Search
- Anthropic: ClaudeBot Documentation
- Brave: Search Independence
- Vercel: The Rise of the AI Crawler
- 404 Media: ClaudeBot Crawling Behavior
Related Research
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
