AI Crawler Comparison: GPTBot, ClaudeBot, PerplexityBot Complete Guide
- December 18, 2024
- AI Search
AI Summary
AI companies run several crawlers with different jobs, so "blocking AI" is never one decision. The practical split is to block the training bots (GPTBot, Google-Extended) while allowing the answer-engine bots (ChatGPT-User, OAI-SearchBot, ClaudeBot) that can actually cite you and send readers back.
- OpenAI runs three separate agents:
GPTBot(training),ChatGPT-User(live browsing),OAI-SearchBot(search indexing). Each takes its own robots.txt rule. ClaudeBotis the only major AI crawler that honorsCrawl-delay, which matters because it can crawl very aggressively.Google-Extendedis a control token, not a crawler. Blocking it stops Gemini training and grounding but does not affect Search or AI Overviews.- PerplexityBot's robots.txt compliance is disputed by independent researchers, and blocking it is not total: Perplexity can still reach your content through Google and Bing APIs.
Quick answer: The main AI crawlers are OpenAI's GPTBot (training), ChatGPT-User (live browsing) and OAI-SearchBot (search indexing), Anthropic's ClaudeBot (training and retrieval), and PerplexityBot (indexing), plus Google's Google-Extended control token for AI training. GPTBot, ClaudeBot, and OAI-SearchBot respect robots.txt as documented by their operators; PerplexityBot's compliance is disputed by independent research. Block training bots and keep answer-engine bots if you want AI visibility without donating training data.
AI companies deploy multiple crawlers for different purposes, each with different behaviors regarding robots.txt compliance, JavaScript rendering, and crawling patterns. Understanding these differences is essential for controlling AI access to your content.
AI Crawler Overview
| Crawler | Company | Purpose | Robots.txt | Crawl-delay | JS Rendering |
|---|---|---|---|---|---|
| GPTBot | OpenAI | AI model training | Respects | No | No |
| ChatGPT-User | OpenAI | Real-time browsing | Respects | No | No |
| OAI-SearchBot | OpenAI | Search indexing | Respects | No | No |
| ClaudeBot | Anthropic | Training + retrieval | Respects | Yes | No |
| PerplexityBot | Perplexity | Indexing | Controversial | No | No |
| Google-Extended | AI training control | Control token | N/A | N/A | |
| Googlebot | Search + AI features | Respects | No | Yes | |
| AppleBot | Apple | Siri/Spotlight | Respects | No | Yes |
Source: Vercel: The Rise of the AI Crawler
OpenAI Crawlers (GPTBot, ChatGPT-User, OAI-SearchBot)
According to OpenAI's documentation, OpenAI operates three distinct crawlers:
GPTBot
- Purpose: Training data collection for AI models
- User agent: GPTBot
- Robots.txt: Respects directives
- IP ranges: Published by OpenAI
- Block this if: You don't want content used for AI training
ChatGPT-User
- Purpose: Real-time browsing triggered by user queries
- User agent: ChatGPT-User
- Robots.txt: Respects directives
- Block this if: You don't want to appear in ChatGPT's live responses
OAI-SearchBot
- Purpose: SearchGPT indexing
- User agent: OAI-SearchBot
- Robots.txt: Respects directives
- Block this if: You don't want to appear in SearchGPT results
Robots.txt Example (OpenAI)
# Block AI training, allow search features User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: /
Anthropic Crawler (ClaudeBot)
According to Anthropic's documentation, ClaudeBot is unique among AI crawlers in supporting the Crawl-delay directive.
- Purpose: Training data and real-time retrieval
- User agent: ClaudeBot
- Robots.txt: Respects directives
- Crawl-delay: Supported (unique among major AI crawlers)
- IP ranges: Not published (uses service provider IPs)
- Legacy agents: Also honors ANTHROPIC-AI and CLAUDE-WEB
Robots.txt Example (Anthropic)
User-agent: ClaudeBot Allow: / Crawl-delay: 10 # Legacy support User-agent: ANTHROPIC-AI Allow: /
Note: 404 Media reported that iFixit saw ClaudeBot hit their site nearly 1 million times in 24 hours. Use Crawl-delay if you allow ClaudeBot.
Perplexity Crawler (PerplexityBot)
PerplexityBot has been controversial due to documented stealth crawling behavior.
- Purpose: Supplementary indexing
- User agent: PerplexityBot
- Robots.txt: Controversial compliance
- IP ranges: Not published
Documented Issues
Independent researchers have documented concerns about PerplexityBot:
- Stealth crawling with modified user agents disguised as Chrome browsers
- Ignoring robots.txt through undeclared crawlers
- Rotating IPs and ASNs to evade blocks
- Using headless browsers with generic Chrome user agents
Important: Even if you block PerplexityBot, Perplexity can still access your content through Google and Bing APIs.
Google AI Crawlers
Google-Extended (Not Actually a Crawler)
According to Google's documentation, Google-Extended is a robots.txt control token, not a crawler. It uses existing Googlebot infrastructure.
- Purpose: Control AI training and Gemini grounding
- What it blocks: AI model training, Gemini grounding
- What it doesn't block: Search indexing, AI Overviews
User-agent: Google-Extended Disallow: /
Googlebot
Standard Googlebot handles both traditional search and AI Overview features.
- JavaScript rendering: Yes (one of few AI-related crawlers with JS support)
- AI Overviews: Uses existing Googlebot index
- Cannot be blocked: If you want search visibility
Crawl-to-Refer Ratios
According to Cloudflare's research, traditional crawlers send traffic back to sites. AI crawlers extract content with minimal referrals:
| Platform | Ratio | Meaning |
|---|---|---|
| Googlebot (traditional) | 3:1 | 1 referral per 3 crawls |
| OpenAI crawlers | 3,700:1 | Massive extraction, minimal traffic |
| Anthropic crawlers | 25,000-100,000:1 | Highest extraction ratio |
| Perplexity | 200:1 | Most favorable among AI platforms |
Identifying Each Crawler in Your Server Logs
Every bot above announces itself with its token inside the user-agent string, so a substring match against raw access logs is the practical identification method. This table maps each crawler to the robots.txt token, the log pattern to match, its documented robots.txt behavior (as covered in the sections above, as of the last check of each operator's documentation), and what your content feeds when it is allowed:
| Crawler | Robots.txt token | Match in logs (UA contains) | Respects robots.txt? | What it feeds if allowed |
|---|---|---|---|---|
| GPTBot | GPTBot | GPTBot | Yes (per OpenAI docs) | Training data for future OpenAI models |
| ChatGPT-User | ChatGPT-User | ChatGPT-User | Yes (per OpenAI docs) | Live answers when ChatGPT browses for a user |
| OAI-SearchBot | OAI-SearchBot | OAI-SearchBot | Yes (per OpenAI docs) | ChatGPT search results and citations |
| ClaudeBot | ClaudeBot | ClaudeBot | Yes, incl. Crawl-delay (per Anthropic docs) | Anthropic training and Claude retrieval |
| Legacy Anthropic | ANTHROPIC-AI, CLAUDE-WEB | anthropic-ai, claude-web | Honored as legacy tokens | Same as ClaudeBot (legacy) |
| PerplexityBot | PerplexityBot | PerplexityBot | Disputed (stealth crawling documented) | Perplexity's answer index and citations |
| Google-Extended | Google-Extended | Never appears in logs (control token only) | N/A: read by Googlebot | Gemini training and grounding |
| Googlebot | Googlebot | Googlebot | Yes | Google Search and AI Overviews |
Two practical caveats. First, a matching user-agent is a claim, not proof: OpenAI publishes IP ranges you can verify against, while ClaudeBot and PerplexityBot do not publish ranges, so treat their log lines as self-reported. Second, Google-Extended will never show up in a log file, because it exists only as a robots.txt token that Googlebot reads, which is why "blocking" it changes nothing about crawl volume.
Quick log audit for AI crawler activity
# Hits per AI crawler over the whole log
grep -oE 'GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot' access.log | sort | uniq -c | sort -rn
# What ClaudeBot is actually reading
grep ClaudeBot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# Are AI bots getting errors? (status distribution per bot)
grep GPTBot access.log | awk '{print $9}' | sort | uniq -c
Run the first command before deciding any robots.txt policy: sites are routinely surprised by which bots are already crawling them, and at what volume, and a policy written against real hit counts beats one written from headlines. If you have never worked with raw server logs before, the primer on log file analysis covers the field layout, bot verification and the one-liners in more depth.
Choosing a policy: four positions that actually make sense
Most robots.txt files end up incoherent because they were edited three times by three people with different goals. It helps to pick one of four defensible positions and write the file to match, rather than blocking bots one headline at a time:
| Position | Who it suits | Block | Allow | Trade-off you are accepting |
|---|---|---|---|---|
| Maximum reach | Publishers and B2B sites whose goal is citation and mindshare | Nothing | Everything, with Crawl-delay on ClaudeBot | Your content trains models that may answer without sending a click |
| Answers yes, training no (most common) | Most commercial sites | GPTBot, Google-Extended | ChatGPT-User, OAI-SearchBot, ClaudeBot, Googlebot | Slightly fewer surfaces, and the split relies on operators honoring their own token separation |
| Search only | Sites whose value is proprietary data or paid content | All AI agents plus Google-Extended | Googlebot, Bingbot | Little to no presence in AI answers, including ones your competitors will appear in |
| Locked down | Membership, legal and licensing-sensitive archives | Everything except search | Googlebot, Bingbot, behind auth where it matters | robots.txt is a request, not enforcement: only auth and WAF rules actually stop a determined crawler |
Whichever you pick, remember the ordering rule that catches people out: robots.txt controls whether a bot may fetch a URL at all, while meta robots and the X-Robots-Tag header control what happens after the fetch. A URL blocked in robots.txt can never have its noindex read, because the crawler never gets far enough to see it.
Complete Robots.txt Template
# OpenAI Crawlers User-agent: GPTBot Disallow: / # Block training User-agent: ChatGPT-User Allow: / # Allow live browsing User-agent: OAI-SearchBot Allow: / # Allow search indexing # Anthropic Crawler User-agent: ClaudeBot Allow: / Crawl-delay: 10 # Perplexity Crawler User-agent: PerplexityBot Disallow: / # Block due to controversial behavior # Google AI Training User-agent: Google-Extended Disallow: / # Block AI training, keep search
Frequently Asked Questions
Partially. You can block AI training crawlers (GPTBot, ClaudeBot, Google-Extended) while allowing search crawlers. But blocking all AI crawlers may reduce your visibility in AI-powered search features.
Only ClaudeBot supports Crawl-delay. Other AI crawlers (GPTBot, PerplexityBot) ignore this directive.
Consider it due to documented compliance issues. However, Perplexity can still access your content through Google and Bing APIs even if you block the crawler.
No. Google-Extended only affects AI training and Gemini grounding. Regular search indexing and AI Overviews are unaffected.
Not immediately, and not from live answers. Blocking GPTBot stops your content from being collected for training future OpenAI models; it does nothing about content already in trained models, and ChatGPT's browsing and search answers are fetched by ChatGPT-User and OAI-SearchBot, which are separate user-agents with their own robots.txt rules.
Server logs are the only reliable source, because none of these bots report to a webmaster console. Grep your access log for the tokens in the identification table above (GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User, OAI-SearchBot) and count hits per bot. If you are behind a CDN that serves cached copies, check the CDN's logs instead, since those requests may never reach your origin.
Sources
- Vercel: The Rise of the AI Crawler
- OpenAI: Bot Documentation
- Anthropic: ClaudeBot Documentation
- Google: Common Crawlers Overview
- Cloudflare: The Crawl-to-Click Gap
- 404 Media: ClaudeBot Crawling Behavior
Related Research
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
