Case study: Fixing “Indexed, though blocked by robots.txt”
- September 4, 2023
- Crawling and Indexing
AI Summary
This case study covers the Search Console status Indexed, though blocked by robots.txt, where Google indexes a URL it was blocked from crawling. The fix is to unblock the URL, add a meta robots noindex tag, let Google recrawl, and only then re block it if you still need to.
- robots.txt controls crawling, not indexing, so a blocked URL can still rank.
- Google indexes such URLs on the strength of inbound links, usually with no snippet.
- A noindex directive placed in robots.txt has not worked since 2019.
- Unblock, add meta robots noindex, wait for a recrawl, then confirm removal.
This SEO case study documents a successful optimization initiative, providing actionable insights for practitioners. The documented approach demonstrates how strategic SEO implementation drives measurable results.
Initial Situation
Understanding the starting point is essential context for evaluating any case study. This documentation covers the initial challenges, competitive position, and business objectives that shaped the SEO strategy.
Strategy and Approach
The strategic approach combined multiple SEO disciplines to address identified opportunities. Key decisions around prioritization and resource allocation provide a template for similar initiatives.
Implementation
Moving from strategy to execution required specific technical implementations, content development, and process changes. This case study documents the practical steps that translated strategy into action.
Results and Learnings
The outcomes demonstrate effectiveness through measurable improvements in rankings, traffic, and business metrics. Analysis of successes and challenges provides learning value for practitioners.
Case studies like this contribute to the SEO knowledge base, helping practitioners learn from documented real-world experiences.
Source: https://www.rebelytics.com/fixing-indexed-though-blocked-by-robots-txt-case-study/
Why a blocked URL still gets indexed
The status “Indexed, though blocked by robots.txt” confuses a lot of site owners because it looks like a contradiction. It is not. robots.txt controls crawling, not indexing. When you add Disallow for a path, you tell Google not to fetch the page body, but if Google discovers the URL through internal or external links it can still add the address to its index. Because it never read the page, the listing usually appears with a missing or generic snippet, and it can quietly rank for branded or long tail queries you never intended.
The fix, in the right order
The order matters, and getting it wrong is why these pages linger for months. First remove the Disallow rule so Googlebot can crawl the URL again. Second, add a meta robots noindex tag in the page head, or send an X-Robots-Tag: noindex header. Third, wait for a recrawl and confirm in Search Console that the URL has dropped from the index. Only after it is deindexed should you consider blocking it in robots.txt again. If you re block too early, Google cannot crawl the page, cannot see the noindex tag, and the URL stays stuck in the index.
| Control | What it does | What it does not do |
|---|---|---|
| robots.txt Disallow | Blocks crawling of the URL body | Does not remove the URL from the index |
| meta robots noindex | Removes a crawlable page from the index | Does not save crawl budget, the page must be crawled first |
| rel canonical | Suggests a preferred URL to consolidate signals | Is a hint, not a strict directive |
Common mistakes to avoid
Two mistakes dominate. The first is adding a noindex line inside robots.txt: Google stopped supporting that undocumented directive in September 2019, so it does nothing. The second is reaching for the URL removal tool as a permanent fix. That tool only hides a URL from results for about six months, so it buys time but does not solve the underlying setup. Fix the crawl and noindex configuration for a durable result. For the broader context on how Google spends its crawl on your site, read our crawl budget explainer and the guide to index bloat, and see more crawling and indexing case studies.
Frequently asked questions
What does Indexed, though blocked by robots.txt mean?
Google has added the URL to its index but was blocked from crawling the page body by robots.txt. It can still rank on the strength of inbound links and the URL itself, usually with a missing or generic snippet.
Why can a blocked page still be indexed?
robots.txt controls crawling, not indexing. If Google discovers the URL through links, it can index the address without reading the content, and it never sees any noindex tag because the block prevents the crawl.
How do I remove a page that is indexed but blocked?
Unblock the URL in robots.txt so Google can crawl it, add a meta robots noindex tag or header, then wait for a recrawl. Once it drops from the index you can block it again if you wish.
Will adding noindex to robots.txt work?
No. Google stopped supporting a noindex directive in robots.txt in 2019. Use a meta robots tag or an X-Robots-Tag header on a crawlable URL instead.
Should I use the URL removal tool?
The removal tool only hides a URL for about six months, so it is a stopgap. Fix the underlying crawl and noindex setup for a permanent result.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
