Internal Disallowed URLs: Linking to Pages Google Cannot Crawl

Quick version: this flag lists internal links that point at URLs your own robots.txt blocks. Every one of those links hands PageRank and crawl attention to a destination Google is not allowed to fetch — a signal poured down the drain. It is rarely catastrophic, but it is almost always sloppy plumbing worth cleaning up.

What this means

You have <a href> links inside your pages pointing to URLs that match a Disallow rule in robots.txt. Google follows your internal links to understand site structure and pass authority, but when the target is disallowed, the crawler stops at the door. The link still spends your internal linking equity; the destination just cannot receive any crawl in return.

Why it matters

Three things go sideways. First, wasted signals — internal PageRank flows toward a URL that can never be crawled or ranked, so it is authority you could have pointed at a page that actually competes. Second, mixed messages — you are telling Google "this is worth linking to" while robots.txt says "do not look at this," and that contradiction is a small but real crawl-efficiency tax. Third, it often reveals a genuine mistake: a link that should point at the indexable version of a page but instead targets a parameter or faceted variant you deliberately blocked.

A real example

An online store blocked its internal search results with Disallow: /search to keep infinite query URLs out of the index — a sound call. But the product template's "find similar items" button linked straight to /search?q=running+shoes&color=blue. Every product page shipped several of these. The result was thousands of internal links funneling equity into a blocked path, plus a steady stream of crawl attempts Google had to bounce off robots.txt.

The fix was to repoint the button at a real, crawlable category page instead of a blocked search query:

<!-- before: links into the blocked /search path -->
<a href="/search?q=running+shoes&color=blue">Find similar</a>

<!-- after: links to an indexable, curated category -->
<a href="/shoes/running/blue/">More blue running shoes</a>

Now the internal link points somewhere Google can actually crawl and rank, and the equity lands on a page that earns traffic.

Disallowed link targets and what to do

How to detect it on your own site

1. Screaming Frog with robots.txt respected. Crawl the site (Configuration > robots.txt > Respect robots.txt). Blocked URLs show a status of "Blocked by robots.txt." Then use Bulk Export > Response Codes > Blocked by robots.txt Inlinks to get the exact list of source pages linking to each blocked target — that inlinks export is the money view.
2. Search Console. The Page Indexing report has a "Blocked by robots.txt" category. Any URL there that also receives internal links is a candidate. Cross-reference with the Links report to see which of your pages point at them.
3. curl to confirm a single target. Verify a suspect URL really is blocked before acting: curl -s "https://example.com/robots.txt" and read the rules, or test the path against them. Do not assume — a rule you half-remember may not match the URL you think it does.
4. Grep your templates. Once you know the blocked pattern (say /search), search your codebase or rendered HTML for href="/search to find every place the template emits the bad link.

How to fix it

If the destination should be crawlable, unblock or repoint

Decide what you actually want indexed. If the target deserves to rank, remove the Disallow rule. If it does not, change the link to point at the version that should — usually a clean category or canonical URL instead of a parameterized one.

If the destination should stay blocked, fix the link

For action links (cart, login, print), the destination is correctly blocked; the link is the problem. Turn it into a real button, or if it must stay an anchor for users, this becomes a deliberate, accepted case rather than a mistake. There is no crawl benefit to passing, but there is also no harm once you have made the choice consciously.

If it is a faceted or parameter pattern, manage the pattern

Blocking facets in robots.txt while linking to them heavily is a common tension. The cleaner architecture is to let a curated set of filter combinations be crawlable and canonical, and stop internally linking to the blocked long-tail combinations at all.

FAQ

Is this an error I have to fix?

Not an emergency. It is a hygiene issue. On a small site with a handful of these, the impact is negligible. On a large site emitting the pattern site-wide, the wasted internal equity and crawl noise add up enough to be worth a template fix.

Will Google index a disallowed page just because I link to it?

It can, awkwardly. Google may index a disallowed URL with no snippet if enough links point to it, because it sees the link but cannot read the page to know it should not be indexed. That produces the ugly "indexed, though blocked by robots.txt" state — another reason to stop linking to blocked URLs.

Should I remove the link or add nofollow?

Prefer repointing to a good URL. If the link genuinely serves users and the target is meant to stay blocked, a rel="nofollow" stops you passing equity into a dead end, but it does not fix the underlying "why am I linking to something I block" question. Repoint when you can; nofollow when you cannot.

How do I find which robots.txt rule is blocking a URL?

Read robots.txt top to bottom and match against the path. Remember precedence: the most specific matching rule wins, and Allow can override a broader Disallow. Search Console's robots.txt tester will tell you exactly which line blocks a given URL.

Related checks

Blocking assets is a related trap — see disallowed JavaScript file: why blocking JS can break rendering. For the rules themselves and how precedence works, the complete robots.txt reference covers wildcards and gotchas. And if the blocked links are faceted URLs, faceted navigation SEO: the complete guide shows how to link to filters without wasting crawl. When these URLs also sit in your sitemap, fix that too: disallowed URL in XML sitemaps.