Author: Claude Vincent (2209 articles)

Bar chart of structured data on 180 of the web's most visited homepages. 71 carry any json-ld, 60 declare an organization, 46 a website node, 26 a searchaction, 16 a breadcrumblist and 13 use microdata.

Most Top Sites Never Say Who They Are in Structured Data

I read the homepage of 180 of the most visited sites on the web. 109 publish…
Learn More
Histogram of homepage title pixel widths across 173 of the web's most visited sites. The distribution is spread widely below the 580 pixel cutoff with a large cluster under 300 pixels, and a thin tail of 30 titles beyond it.

Homepage Titles Are Mostly Too Short, Not Too Long

I measured the rendered width of the title on 173 of the most visited sites on…
Learn More
Bar chart of how many of 184 of the web's most visited sites serve each agent-facing text file. Llms. Txt 31 sites, llms-full. Txt 6, agents. Md 1, and neither well-known llms. Txt nor ai. Txt on any site.

Who Actually Serves an llms.txt? 184 Top Sites Measured

I asked 184 of the most visited sites on the web for five agent-facing files. llms.txt…
Learn More
Stacked bar of 184 of the web's most visited sites showing where an xml sitemap can be found. 78 sites by guessing the usual path, 38 more only by reading the sitemap line in robots. Txt, and 68 by neither route.

Finding a Competitor's Sitemap, and Turning It Into a Change Log

A competitor's XML sitemap is a change log they publish without meaning to. The hard part…
Learn More
Bar chart of the twelve most blocked ai crawler names across 148 top websites. Bytespider and ccbot lead with 21 sites each, then gptbot 18 and google-extended 15. Three names in the top ten, anthropic-ai, omgili and cohere-ai, are marked as names their vendor no longer publishes.

AI Crawler User Agents: The Current Names, and Which Ones Sites Actually Block

I read the robots.txt of 148 of the most visited sites on the web and checked…
Learn More
A two by two matrix of 126 top websites. Where robots. Txt does not block gptbot, 87 sites serve the page and 20 deny it at the edge. Where robots. Txt does block gptbot, 14 sites serve the page anyway and 5 deny it.

Can AI Crawlers Actually Read Your Site? I Measured 400 of the Biggest

I requested the homepage of 400 of the world's most visited sites as a browser and…
Learn More

Get new blog posts by email: