ClaudeBot

Claudebot

ClaudeBot

ClaudeBot is the web crawler operated by Anthropic, the company behind the Claude AI assistant. It…
Learn More
Diagram showing the perplexitybot index crawler honoring a robots. Txt disallow while the perplexity-user live fetcher does not, with a note to verify by ip.

PerplexityBot

PerplexityBot is the web crawler operated by Perplexity, the AI answer engine. It discovers and indexes…
Learn More
Gptbot

GPTBot

GPTBot is the web crawler operated by OpenAI. It fetches publicly available web pages that may…
Learn More

Get new blog posts by email: