Disallowed URL in XML Sitemaps: How to Fix It

What "Disallowed URL in XML Sitemaps" means

This issue fires when your XML sitemap lists a URL that is simultaneously blocked from crawling by your robots.txt file. In other words, the same address is being told two opposite things at once. The sitemap entry says "here is an important page, please crawl and index it," while the matching robots.txt disallow rule says "do not crawl this path at all." Search engines cannot honour both instructions, so they flag the contradiction during sitemap processing.

An XML sitemap is fundamentally a discovery and prioritisation file. It exists to hand crawlers a clean list of the canonical, indexable URLs you want them to find and refresh. A robots.txt disallow, by contrast, is a crawl directive that stops a bot from even requesting the resource. When a single URL sits in both places, your sitemap loses some of its trustworthiness as a signal, and the affected page is effectively wasted real estate inside it.

Why robots-disallowed URLs do not belong in a sitemap

Google's own sitemap guidance is consistent on one point: a sitemap should contain only the URLs you genuinely want crawled and indexed, and those URLs should be indexable. A page blocked by robots.txt cannot be crawled, which means the crawler can never read its content, its meta tags, or its canonical signals. Listing it in the sitemap therefore asks the search engine to act on a page it is forbidden to look at.

Crawl tools formalise this. Screaming Frog surfaces internal URLs blocked by robots.txt under its Response Codes tab with a "Blocked by Robots.txt" filter, and it even shows the exact matched disallow line responsible. Sitebulb and similar crawlers report the same class of hint when a sitemap URL collides with a disallow rule. The shared message across these tools is simple: a sitemap URL that a bot cannot fetch is a defect worth resolving.

The mixed-signal problem

Search engines build trust in your sitemap over time. A sitemap full of clean, crawlable, canonical URLs is treated as a reliable map of your site. Every contradictory entry chips away at that reliability. When a crawler repeatedly finds entries it is not allowed to visit, the sitemap becomes a noisier signal, and that noise can dampen how aggressively the file is trusted as a whole.

There is also a practical cost. Crawl budget is finite. Pointing crawlers at addresses they must immediately refuse wastes attention that should be spent on the pages you actually care about. Blocking genuinely low-value paths in robots.txt is a perfectly valid tactic, but those same paths then have no business advertising themselves inside the sitemap.

Common causes

Most occurrences trace back to one of a handful of patterns:

Automatically generated sitemaps that include every published URL regardless of robots rules. Many CMS and SEO plugins build the sitemap from your content database, not from your crawl directives, so a disallowed section still gets listed.

A robots.txt disallow added after the sitemap was built, for example blocking a faceted-navigation parameter, a staging path, an internal search results path, or a "/tag/" or "/cart/" directory, without regenerating or pruning the sitemap.

Overly broad disallow patterns that catch more URLs than intended. A rule meant to block one parameter can accidentally match a whole branch of legitimate pages that are still listed in the sitemap.

Migrations and redesigns where old directory rules linger in robots.txt while a freshly built sitemap re-publishes the new equivalents under paths that overlap the stale block.

How to diagnose

Start by listing the exact URLs flagged, then check each one against your live robots.txt. Run a crawl in Screaming Frog or Sitebulb in list mode against your sitemap, then apply the blocked-by-robots filter to see every affected URL alongside the specific disallow line that matches it. That matched-line column tells you precisely which rule to edit.

You can also test individual URLs against your robots.txt rules directly. Confirm which directive is doing the blocking before you change anything, because a single broad pattern is often responsible for many flagged URLs at once.

# robots.txt currently blocking a path
User-agent: *
Disallow: /private/

# Sitemap entry that contradicts it
<url>
  <loc>https://example.com/private/report</loc>
</url>

# Pick ONE resolution, never both:
# A) page should stay private  -> remove the <url> from the sitemap
# B) page should be indexed    -> remove or narrow the Disallow rule

How to fix

There are exactly two correct outcomes, and the right one depends on a single decision: do you want this page in Google's index?

Option A: remove the URL from the sitemap

If the page genuinely should not be crawled, for example an internal, parameterised, or thin path, leave the robots.txt disallow in place and take the URL out of the sitemap. In most CMS and SEO plugins this means excluding that post type, taxonomy, or section from sitemap generation, or filtering out the matching pattern. After regenerating, confirm the URL is gone from the file.

Option B: unblock the URL in robots.txt

If the page should be crawled and indexed, the disallow rule is the mistake. Remove it, or narrow it so it no longer matches this URL. Be careful that loosening the rule does not accidentally open up paths you meant to keep blocked. Once the disallow is gone, the sitemap entry becomes valid because the crawler can finally fetch the page.

A worth noting subtlety: if you want a page crawled but kept out of the index, robots.txt is the wrong tool entirely. Allow the crawl and use a noindex meta tag on the page instead, since a blocked page can never have its noindex tag read.

Best practice

Treat your sitemap as a strict allow-list of canonical, crawlable, indexable URLs and nothing else. Every entry should return a 200 status, be free of robots.txt blocks, carry a self-referencing canonical, and not be set to noindex. Whenever you add a disallow rule, immediately check whether any sitemap URLs fall under it, and regenerate the sitemap if they do. Re-crawl after every robots.txt change so contradictions are caught before search engines do.