Crawling (63 articles)

Diagram of a robots. Txt file with grouped user agent rules beside four facts: one file per host, most specific group wins, longest path wins, and it controls crawling not indexing.

Robots.txt

Robots.txt is a plain text file at the root of a site that tells crawlers which…
Learn More
Diagram showing a large site moving from millions of thin duplicate urls through noindex and consolidation to a focused set of high value indexed pages.

You deleted how many pages? 130M and heres why

A case study about how Trainline deleted 130 million low-value pages.
Learn More
How to use gsc’s crawl stats reporting to analyze and troubleshoot site moves

How To Use GSC’s Crawl Stats Reporting To Analyze and Troubleshoot Site Moves

Glenn Gabe makes a case for why SEOs need to make Crawl Stats reports using a…
Learn More
Crawl traps: identification and prevention

Crawl Traps: Identification and Prevention

A crawl trap is any structure that produces an infinite or near-infinite set of URLs without…
Learn More
Chart showing search visibility staying stable then dropping sharply after a javascript relaunch, with common causes and fixes.

Hulu.com JavaScript Fail - 56% Visibility Drop and Counting

A case study showing the impact of Hulu migrating to JavaScript framework.
Learn More
Image migrations and lost signals – how long before images lose signals after a flawed url migration?

Image Migrations and Lost Signals – How long before images lose signals after a flawed url migration?

A case study showing the impact of not redirecting traffic-driving images.
Learn More

Get new blog posts by email: