Robots.txt (19 articles)

Diagram of the indexed though blocked by robots. Txt warning in google search console and the steps that resolve it.

Fixing “Valid with Warnings” in Google Search Console

A case study that showed how that showed a fix for “Valid with warnings” warnings in…
Learn More
Diagram of common robots. Txt mistakes including blocked assets, a staging file shipped live, and skipping validation

Robots.txt Study: Common Mistakes and Optimization

Research on robots.txt optimization reveals key patterns and actionable insights for SEO practitioners.
Learn More
Annotated robots. Txt file showing the user-agent, disallow, allow, sitemap, and crawl-delay directives, with a callout that crawling is not the same as indexing.

Robots.txt FAQ: Complete Guide to Crawler Directives

Complete robots.txt FAQ with 35 questions covering syntax, directives, crawler control, testing, and common mistakes. Quick,…
Learn More
Gptbot

GPTBot

GPTBot is the web crawler operated by OpenAI. It fetches publicly available web pages that may…
Learn More
Before and after bar chart of googlebot request share on a large e-commerce site: faceted urls, parameters, and redirects dominate before optimization, while products and categories rise to the majority after canonical, robots, and redirect fixes.

Technical SEO: an experiment to optimize Crawl Budget in big ecommerce sites

A case study showing how blocking requests to an API helped improved crawl budget
Learn More
Diagram of six lesser known seo levers including log file mining, internal link equity, entity consistency, and crawl efficiency.

Esoteric SEO Tips I Hope You Don't Already Know

A talk by Oliver Mason identifies SEO tricks to do some very edge-case SEO things.
Learn More

Get new blog posts by email: