Crawling (63 articles)

Comparison diagram of two ways to cut an seo backlog: layer by layer, where research, spec, build and measure each finish across every template before anything ships, against vertical slices where each slice runs through all four layers and ships on its own.

SEO and Agile Iterations - by Adam Gent - The SEO Sprint

A series of mini case studies that show what happen when you break SEO actions into…
Learn More
Crawl budget

Crawl Budget

Crawl budget is the number of URLs a search engine is willing and able to crawl…
Learn More
Flow for verifying googlebot using reverse dns and the published googlebot ip range json so spoofed user agents can be rejected.

List of Googlebot IP addresses released

Search Engine Land is the leading industry source for daily, must-read news and in-depth analysis about…
Learn More
Diagram of google two wave indexing where wave one crawls the raw html and wave two renders javascript in the web rendering service, listing what wrs will not do.

WRS (Web Rendering Service)

The WRS (Web Rendering Service) is the part of Google's pipeline that renders pages with a…
Learn More
Diagram of shopify with fast simon filters: one base collection stays canonical and indexable, while size, color, price and brand filters multiply into 1,600 plus urls that should be canonicalized, noindexed, and disallowed to protect crawl budget.

Shopify & Fast Simon: An E-commerce SEO Case Study

A case study showing the impact of adding Fast Simon Shopify integration and fixing technical issues.
Learn More
Diagram of a five step python workflow for finding striking distance keywords using search console data.

Python SEO Script: Top Keyword Opportunities Within Striking Distance

Learn how to identify keywords your pages are close to ranking for using this simple Python…
Learn More

Get new blog posts by email: