Case study: Fixing “Indexed, though blocked by robots.txt”

Few Search Console statuses generate as much confusion as Indexed, though blocked by robots.txt. It reads like a contradiction: the page is both blocked and indexed at the same time. It is not a contradiction, it is the predictable result of asking robots.txt to do a job it was never designed for. This practitioner guide, which builds on a case study by Eoghan Henn, explains exactly what the status means, why a blocked URL still lands in the index, and the precise order of changes that removes it for good. It also shows the exact robots.txt and tag edits so you can copy the pattern.

What the status actually means

The status appears in the Page indexing report under Why pages are not indexed, but the wording is slightly misleading because the page is indexed. What Google is really telling you is that it added the URL to its index despite being unable to crawl the page body. robots.txt controls crawling, not indexing. When you add a Disallow rule for a path, you tell Googlebot not to fetch the content, but if Google discovers the address through an internal link, an external backlink, or an XML sitemap, it can still record the URL. Because it never read the page, the listing usually shows the raw URL with a generic description such as "No information is available for this page," and it can quietly rank for branded or long tail queries you never intended to target.

This matters more than it looks. A thin, snippetless result can undercut a stronger page for the same query, waste the authority of inbound links on a dead end, and clutter your index with URLs you meant to keep private. The status is a signal that your crawling controls and your indexing controls are fighting each other.

A worked example: the exact edits

Suppose a shop blocked its faceted filter URLs to save crawl budget, and one of them, /collections/shoes?color=red, picked up a few backlinks and got indexed anyway. The robots.txt looked like this:

User-agent: *
Disallow: /collections/*?color=

The instinct to also drop a noindex into robots.txt is exactly the trap. This does nothing, because Google retired support for it:

User-agent: *
Disallow: /collections/*?color=
Noindex: /collections/*?color=   # ignored since 2019

The correct sequence is to first unblock the path, so Google can crawl it and read a real indexing directive:

User-agent: *
# Disallow rule removed so the URL can be crawled

Then serve a noindex on the URL itself. For an HTML page, put this in the head:

<meta name="robots" content="noindex, follow">

For a non HTML resource such as a PDF, or when you want a server level rule, send the same instruction as a response header:

X-Robots-Tag: noindex

Once Google recrawls the now crawlable URL, it reads the noindex and drops the page. If, and only if, you still want to conserve crawl budget afterwards, you can restore the Disallow rule. The follow value keeps link equity flowing through the page while it is being deindexed.

The fix, in the right order

The order is the whole game, and getting it wrong is why these pages linger for months. Before you touch anything, make one decision: do you want this URL indexed or not? That single choice sets the path.

If you want to keep the page indexed, the status is simply telling you the page is blocked. Remove the Disallow rule, make sure the page has no noindex tag, request indexing, and you are done. If you want to remove the page from the index, follow the four steps: remove the Disallow rule so Googlebot can crawl the URL again; add a meta robots noindex tag or an X-Robots-Tag: noindex header; wait for a recrawl and confirm in Search Console that the URL has dropped; and only after it is deindexed should you consider blocking it in robots.txt again. If you re block too early, Google cannot crawl the page, cannot see the noindex tag, and the URL stays stuck in the index indefinitely.

Validating the fix in Search Console

Do not assume the change worked. Verify it. Open the URL Inspection tool in Search Console and paste the affected address. Under Page indexing, check that Crawl allowed reads Yes and that Indexing allowed shows the noindex is now detected. Click Request indexing to push the URL to the front of the crawl queue. Over the following days, watch the Indexed, though blocked by robots.txt row in the Page indexing report: as pages are recrawled and deindexed, the count should fall. When the URL you fixed moves to Excluded by noindex tag, the change has fully taken effect and you can, if you wish, restore the robots.txt block.

Common mistakes to avoid

Two mistakes dominate. The first is adding a noindex line inside robots.txt: Google stopped supporting that undocumented directive in September 2019, so it does nothing at all. The second is reaching for the URL removal tool as a permanent fix. That tool only hides a URL from results for about six months, so it buys time but does not solve the underlying setup. A third, subtler error is re blocking the path the moment you add the noindex tag, which prevents the very crawl that would let Google read it. Fix the crawl and noindex configuration together, in order, for a durable result. For the broader context on how Google spends its crawl on your site, read our crawl budget explainer and the guide to index bloat. For the underlying mechanics, see the complete robots.txt reference and the meta robots and X-Robots-Tag reference, and browse more crawling and indexing case studies.

About this case study

Source: https://www.rebelytics.com/fixing-indexed-though-blocked-by-robots-txt-case-study/

Frequently asked questions