consulting services

Local Landing Pages

Diagram showing two separate pipes for ai: a training pipe where crawlers copy your published pages into model weights, and a retrieval pipe where search agents fetch live pages and cite them.

AI Training Data

AI training data is the large collection of text, images, code, and other content used to…
Learn More
Comparison chart of ai crawlers by operator: openai gptbot blocked while chatgpt-user and oai-searchbot are allowed, anthropic claudebot and its legacy tokens allowed with crawl-delay support, perplexitybot blocked over documented stealth crawling, and google-extended blocked while googlebot stays allowed, plus crawl-to-refer ratios from cloudflare research.

AI Crawler Comparison: GPTBot, ClaudeBot, PerplexityBot Complete Guide

Complete comparison of AI crawlers including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Covers robots.txt compliance, Crawl-delay support,…
Learn More
Annotated robots. Txt file showing the user-agent, disallow, allow, sitemap, and crawl-delay directives, with a callout that crawling is not the same as indexing.

Robots.txt FAQ: Complete Guide to Crawler Directives

Complete robots.txt FAQ with 35 questions covering syntax, directives, crawler control, testing, and common mistakes. Quick,…
Learn More

Get new blog posts by email: