consulting services

Crawling

Three panel diagram of log file analysis for seo: raw googlebot access log lines classified as keep or waste, a four step verify and segment workflow, and a crawl budget verdict bar where only 44 percent of hits reach indexable canonical 200 urls.

Log File Analysis

Log file analysis is the practice of examining a server's raw access logs to see exactly…
Learn More
Annotated robots. Txt example showing user-agent, disallow, allow, and sitemap directives next to the difference between crawl control and index control.

Complete Guide to Robots.txt Configuration

Comprehensive guide to robots.txt covering implementation, best practices, troubleshooting, and advanced techniques.
Learn More
Diagram of cloaking: a server checks the user-agent and ip, serves googlebot a clean keyword-rich page, serves humans a different page, and google detects the mismatch as a spam-policy violation.

Disallow Humans (Bad Cloaking)

A cloaking case study seeing if Google penalizes the website if it were intentionally decieved.
Learn More
Twitter down 32% in 24 hours - sistrix

Twitter down 32% in 24 hours - SISTRIX

A case study from Sistrix on the impact of Twitter blocking Google from crawling and indexing.
Learn More
Diagram of how search engine crawling discovers and fetches pages before indexing on seoprocheck. Com

Crawling

Crawling is the process by which search engines discover and download pages on the web using…
Learn More
Rendering queue: google needs 9x more time to crawl js than html

Rendering Queue: Google Needs 9X More Time To Crawl JS Than HTML

An experiment to find out how long it would take Googlebot to crawl JavaScript vs HTML…
Learn More

Get new blog posts by email: