What this check is telling you
You have a URL that your robots.txt file disallows, yet Search Console shows it earning clicks and impressions from Google. On the surface that feels impossible. If Google cannot crawl the page, how is it ranking? The answer is the single most misunderstood fact about robots.txt: it controls crawling, not indexing. A disallowed URL can absolutely sit in the index and rank, it just does so with one hand tied behind its back.
When Googlebot hits a disallowed URL it does not fetch the content. But it can still index the URL itself, usually based on links pointing at it and the anchor text those links use. That is why you sometimes see a result in the SERP with the message "No information is available for this page" under it. Google knows the URL exists and thinks it is relevant enough to show, but it never read the page. So yes, a blocked page can get traffic, and this check is flagging that mismatch so you decide what you actually want.
Why it matters
This is a strategy problem more than a bug. There are two ways this ends up being wrong, and they pull in opposite directions.
Case one: the page should be earning traffic properly. Someone disallowed a directory too broadly, or a template moved under a blocked path, and now a page that deserves to rank is crippled. It shows a bare URL in the results, has no meta description, and cannot use its own on page content to rank. You are getting traffic despite the block, not because of a healthy setup. Unblock it and it will do far better.
Case two: the page should not be public at all. Maybe it is a thin filter URL, an internal search page, or a staging path. You blocked it to keep it out of Google, but it slipped in before the block or got linked from elsewhere, and now it is drawing traffic you did not want and cannot control. Robots.txt alone will never get it back out, because Google cannot crawl it to see a noindex you might add later.
Robots.txt versus noindex, the distinction that fixes this
| Lever | Stops crawling | Removes from index | Use it when |
|---|---|---|---|
| robots.txt Disallow | Yes | No | Saving crawl budget on pages already out of the index |
| noindex meta or header | No | Yes | You want a page kept out of results |
| Both at once | Yes | No, and it backfires | Never, the block hides the noindex Google needs to read |
That last row is the trap. If you want a page out of Google, do not disallow it in robots.txt, because Google can no longer crawl the page to see the noindex you added. Let it crawl, let it read the noindex, let it drop, and only then consider a Disallow to save crawl budget.
How to detect and diagnose it
- Google Search Console Performance: the report that flags the click generating URLs. Cross reference those against your robots.txt rules.
- GSC URL Inspection: paste the URL, and it will show "Blocked by robots.txt" alongside its index status. That confirms the conflict on a real Google fetch.
- Screaming Frog: configure it to respect robots.txt, crawl, and check the Response Codes and robots.txt tabs. It marks which URLs are blocked and lets you compare against a list of indexed pages.
- The robots.txt Tester and a plain
site:search: searchsite:yourdomain.com/blocked-path/and see if blocked URLs show up with the "No information is available" note. - Server logs: confirm Googlebot is honoring the Disallow and not fetching the page body, which tells you the traffic is coming from URL only indexing.
- Decide first whether the page belongs in Google at all
- To keep it in, remove the Disallow and give it a real title and description
- To pull it out, allow crawling and add a noindex, then wait for it to drop
- Use robots.txt to save crawl budget only after a page is already deindexed
- Recheck in Search Console after the change lands
- Assume a Disallow removes a page from Google, it does not
- Stack Disallow and noindex on the same URL, the block hides the noindex
- Block a directory so broadly that real pages get caught
- Ignore the flag because "it is getting traffic anyway"
- Use a password or robots block when you actually need noindex
FAQ
How can a blocked page rank at all?
If I want it gone, why not just keep the robots.txt block?
Is this always a mistake?
How long until a noindex page drops out?
Robots.txt conflicts and index bloat hide in almost every large site. Our advanced SEO audit maps what is blocked, what is indexed, and what to do about each.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
