Case study: Fixing “Indexed, though blocked by robots.txt”

Diagram showing why a URL blocked in robots.txt can still be indexed by Google, and the ordered fix of unblocking the URL then adding a meta robots noindex tag.

Case study: Fixing “Indexed, though blocked by robots.txt”

A case study by Eoghan Henn on how he fixed Indexed, though blocked by robots.txt file.

Diagram of cloaking: a server checks the User-Agent and IP, serves Googlebot a clean keyword-rich page, serves humans a different page, and Google detects the mismatch as a spam-policy violation.

Disallow Humans (Bad Cloaking)

A cloaking case study seeing if Google penalizes the website if it were intentionally decieved.

Line chart showing the Twitter SISTRIX visibility index falling 32 percent within 24 hours after crawlers met a login wall, with before and after crawl states.

Twitter down 32% in 24 hours - SISTRIX

A case study from Sistrix on the impact of Twitter blocking Google from crawling and indexing.

Decision diagram mapping each URL removal situation to the correct method: the Removals tool for urgent temporary removal, noindex to drop a live page, a 410 response for deleted pages, and the X-Robots-Tag header for non HTML files.

Another Way To Deindex URLs

"An experiment to see if the ""unavailable_after"" directive can remove pages from search results."

Index bloat triage diagram: a bar splitting high value pages from bloat such as filters, tag archives, internal search, and URL parameters, then three actions of enriching thin pages, applying noindex to needed non search URLs, and returning a 410 for junk.

Index Bloat: Why Deleting Old Website Pages Is Good for SEO

A case study to show the impact of removing low-quality pages (index bloat) on a eCom…

Diagram showing three ways to hide text with display none, visibility hidden, and a JavaScript toggle, and how each is treated for crawling and indexing.

Long Term CSS Javascript Hidden Text Experiment

An experiment that shows Google gives less weight to text hidden via CSS and JavaScript.

Load More

Get new blog posts by email: