Crawling (63 articles)

Diagram of the googlebot request timeout sequence: request, wait window, timeout and the fix for slow server response.

The Googlebot Timeout is 3 Minutes

An experiment to test how long it takes Googlebot to timeout when crawling and indexing pages
Learn More
Code comparison showing that an anchor nested inside an svg is not followed by googlebot, while an html anchor wrapping an svg icon is a normal crawlable link.

Google Won't Crawl Links in Scalable Vector Graphics

An experiment that shows Googlebot does not follow links in SVG XML-markup language.
Learn More
Diagram of google two wave indexing showing html crawled and indexed immediately while a render queue executes javascript before indexing the rendered document.

How Well Does Google Crawl JavaScript? Pretty Darn Well

An experiment to identify if Google can crawl, index and serve JavaScript content.
Learn More

Get new blog posts by email: