Author Archives: Claude Vincent

A two by two matrix of 126 top websites. Where robots.txt does not block GPTBot, 87 sites serve the page and 20 deny it at the edge. Where robots.txt does block GPTBot, 14 sites serve the page anyway and 5 deny it.

Can AI Crawlers Actually Read Your Site? I Measured 400 of the Biggest

September 5, 2026

I requested the homepage of 400 of the world's most visited sites as a browser and as GPTBot, ClaudeBot and PerplexityBot, then checked…

Flowchart of the five check pre-publish quality gate for AI-assisted content: the anyone test, the no-AI test, says something new, the first screen, and the recommend test. All five passing leads to ship it; any failure leads back to editing.

The Pre-Publish Quality Gate for AI-Assisted Content

By Claude Vincent

August 6, 2026

The five check gate we run on every AI-assisted page, built after watching scaled-content suppression cut a similar site's impressions by roughly 90…

Diagram of the agent-readable file stack showing AGENTS.md in the code repository read by coding agents, llms.txt and llms-full.txt at the website root read by answer engines, and robots.txt plus RSL as the access and licensing layer beneath both.

AGENTS.md vs llms.txt vs llms-full.txt: Which Agent File Does What

By Claude Vincent

July 18, 2026

AGENTS.md, llms.txt and llms-full.txt look alike but serve different readers. What each file does, how they differ, and the licensing layer most checklists…

Profound vs Semrush and Ahrefs: What an AI-Search Tool Actually Replaces (and What It Doesn't)

By Claude Vincent

June 19, 2026

Profound vs Semrush and Ahrefs don't measure the same thing. A practitioner's honest take on what AI-search visibility tools actually replace in your…