Crawling (63 articles)

Diagram showing why a url blocked in robots. Txt can still be indexed by google, and the ordered fix of unblocking the url then adding a meta robots noindex tag.

Case study: Fixing “Indexed, though blocked by robots.txt”

A case study by Eoghan Henn on how he fixed Indexed, though blocked by robots.txt file.
Learn More
Diagram of cloaking: a server checks the user-agent and ip, serves googlebot a clean keyword-rich page, serves humans a different page, and google detects the mismatch as a spam-policy violation.

Disallow Humans (Bad Cloaking)

A cloaking case study seeing if Google penalizes the website if it were intentionally decieved.
Learn More
Line chart showing the twitter sistrix visibility index falling 32 percent within 24 hours after crawlers met a login wall, with before and after crawl states.

Twitter down 32% in 24 hours - SISTRIX

A case study from Sistrix on the impact of Twitter blocking Google from crawling and indexing.
Learn More
Two lane flow diagram of google two wave indexing, showing html indexed immediately while javascript waits in the render queue before rendering and indexing, with a bar comparison of about 9 times more time for javascript.

Rendering Queue: Google Needs 9X More Time To Crawl JS Than HTML

An experiment to find out how long it would take Googlebot to crawl JavaScript vs HTML…
Learn More
Diagram showing crawl budget as crawl capacity limit plus crawl demand, with google's thresholds for large sites and the main actions to manage url inventory.

Crawl Budget Management For Large Sites | Google Search Central | Documentation | Google Developers

Learn what crawl budget is and how you can optimize Google's crawling of large and frequently…
Learn More
Diagram comparing crawl budget and crawl efficacy using server log files, prioritisation and reduced crawl waste

Crawl efficacy: How to level up crawl optimization

Crawl budget is a vanity metric. Your goal should be to guide Googlebot in crawling important…
Learn More

Get new blog posts by email: