How we optimized our Crawl Budget

Diagram showing a web worker returning data by postMessage to the main thread, which writes it into the DOM so the rendering service indexes the text.

Web Worker Content - Will It Index?

An experiment to see if Web Workers can be used to add content to a page.

Diagram showing a large site moving from millions of thin duplicate URLs through noindex and consolidation to a focused set of high value indexed pages.

You deleted how many pages? 130M and heres why

A case study about how Trainline deleted 130 million low-value pages.

Before and after bar chart of Googlebot request share on a large e-commerce site: faceted URLs, parameters, and redirects dominate before optimization, while products and categories rise to the majority after canonical, robots, and redirect fixes.

Technical SEO: an experiment to optimize Crawl Budget in big ecommerce sites

A case study showing how blocking requests to an API helped improved crawl budget

Diagram comparing the rel canonical HTTP header read during the initial fetch against the HTML tag that needs parsing, and noting the header works for PDFs.

Why the rel=”canonical” HTTP Header is faster than the rel=”canonical” HTML tag

"An experiment to see which would be faster: HTML canonical tag or rel=""canonical"" HTTP header."

Diagram showing that a website on shared hosting is crawled and ranked by Googlebot on its own page quality, speed and links, not by the other sites sharing its IP address.

Long Term Shared Hosting Experiment

An experiment to see if shared hosting with low-quality websites impacts rankings.

Diagram of one XML sitemap split into segmented sitemaps by section, each feeding the Google Search Console Sitemaps report so a low indexed to submitted ratio reveals the weak section.

An Alternative Approach to XML Sitemaps

A case study showing splitting XML Sitemaps into smaller 10K files improves crawling and indexing.

Load More

Get new blog posts by email: