consulting services

AI Training Data

Diagram showing two separate pipes for ai: a training pipe where crawlers copy your published pages into model weights, and a retrieval pipe where search agents fetch live pages and cite them.

AI Training Data

AI training data is the large collection of text, images, code, and other content used to…
Learn More
Diagram of the claude citation pipeline: a server-rendered page is crawled by claudebot, indexed by brave search, retrieved as passages by claude and cited in the answer, with the failure mode at each stage and the six content properties that earn citations.

Claude SEO: How to Get Cited in Claude AI

Claude uses Brave Search with 30+ billion pages indexed independently from Google and Bing. Learn how…
Learn More
Comparison chart of ai crawlers by operator: openai gptbot blocked while chatgpt-user and oai-searchbot are allowed, anthropic claudebot and its legacy tokens allowed with crawl-delay support, perplexitybot blocked over documented stealth crawling, and google-extended blocked while googlebot stays allowed, plus crawl-to-refer ratios from cloudflare research.

AI Crawler Comparison: GPTBot, ClaudeBot, PerplexityBot Complete Guide

Complete comparison of AI crawlers including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Covers robots.txt compliance, Crawl-delay support,…
Learn More

Get new blog posts by email: