Crawling
Complete comparison of AI crawlers including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Covers robots.txt compliance, Crawl-delay support,…
Comprehensive guide to robots.txt covering implementation, best practices, troubleshooting, and advanced techniques.
A case study by Eoghan Henn on how he fixed Indexed, though blocked by robots.txt file.
ClaudeBot is the web crawler operated by Anthropic, the company behind the Claude AI assistant. It…
Crawling is the process by which search engines discover and download pages on the web using…
PerplexityBot is the web crawler operated by Perplexity, the AI answer engine. It discovers and indexes…
Get new blog posts by email:
Technical SEO consulting and GEO strategy for teams who need real implementation support. 20 years of enterprise experience across major media, technology, recruitment, and ecommerce brands, and a major enterprise brand. Case studies, research, and resources for search and AI visibility.