How to Audit Your XML Sitemap for SEO Issues
- March 9, 2022
- Crawlability and Indexation, Technical SEO
AI Summary
Auditing an XML sitemap means verifying three things: that every URL in it returns a 200, self canonical, indexable page; that the file itself is discoverable, valid and within size limits; and that Search Console's discovered count roughly matches the pages you actually want indexed. Anything else in the file wastes crawl budget and weakens the quality signal your sitemap sends.
- Hard limits are 50,000 URLs and 50 MB uncompressed per file. Split larger sets behind a sitemap index.
- Crawl the file in list mode and filter for 3xx, 4xx,
noindexand canonicalised URLs. Fix the generator, never the file by hand. - Google ignores
changefreqandpriority. Onlylastmodearns its place, and only when it reflects real content changes. - A per sitemap indexation ratio below 50% usually signals a content or duplication problem, not a technical one.
To audit an XML sitemap, verify three things: that every URL in it returns a 200, self-canonical, indexable page; that the sitemap itself is discoverable, valid XML, and within size limits; and that Google Search Console's "Discovered" count roughly matches your count of pages you actually want indexed. Anything else in the file (redirects, 404s, noindexed or non-canonical URLs) wastes crawl budget and muddies the quality signal your sitemap sends. Most sitemap problems fall into a dozen recurring patterns, and all of them are checkable with Search Console, a crawler, and curl.
Why Sitemap Audits Matter for Crawling and Indexation
Crawling represents a critical component of modern SEO strategy. As search engines continue to evolve and user expectations shift, understanding the nuances of crawling becomes essential for maintaining and improving organic visibility. The XML sitemap is your most direct lever here: it is a curated list of the URLs you are asking search engines to crawl and index. Google treats it as a strong hint for discovery and canonical selection, not a command, which is exactly why a dirty sitemap hurts. If 30% of the URLs you submit are redirects or noindexed pages, you are telling Google your own map of the site is unreliable, and crawl scheduling on the pages that matter suffers for it. This guide covers the key concepts, implementation approaches, and strategic considerations that SEO practitioners need to master when auditing sitemaps.
The Sitemap Audit, Step by Step
1. Find every sitemap that actually exists
Check robots.txt for Sitemap: lines, then request the common defaults directly: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml (WordPress core), /sitemap_index.xml (Yoast), /page-sitemap.xml, /post-sitemap.xml. Sites frequently serve two competing sitemap systems (a plugin sitemap and a leftover static file) with different URL sets. Kill the stale one. Also confirm what is registered in Google Search Console under Indexing > Sitemaps; a sitemap that exists on the server but was never submitted still works via robots.txt discovery, but you lose GSC's per-sitemap reporting. Specialist sitemaps follow exactly the same rules: an image sitemap or a video sitemap should list only assets attached to canonical, indexable pages, and both count toward the same 50,000 URL ceiling as any other sitemap file.
2. Validate the file itself
Fetch it from the command line so you see what a bot sees:
curl -sI https://example.com/sitemap.xml: expect 200 and Content-Type: application/xml or text/xml. A 301 to another location, a 404, or an HTML error page served with a 200 are all common failures. Then check structure: the root element must be <urlset> (or <sitemapindex> for an index file) with the namespace xmlns="http://www.sitemaps.org/schemas/sitemap/0.9". Limits: 50,000 URLs and 50 MB uncompressed per file; beyond that, split into multiple sitemaps referenced by an index. Gzipped sitemaps (sitemap.xml.gz) are fine, but the 50 MB limit applies to the uncompressed size.
3. Crawl the sitemap URLs
In Screaming Frog, switch to Mode > List, choose Download XML Sitemap, and paste the sitemap URL. Every URL should come back 200, indexable, and self-canonical. Filter the crawl for: 3xx (redirected URLs, replace with final destinations), 4xx/5xx (remove), noindex (remove: a URL should never be in the sitemap and noindexed simultaneously), and canonicalised URLs where rel=canonical points elsewhere (list the canonical target instead). In a full site crawl, Screaming Frog's Sitemaps tab also surfaces the inverse problem: indexable pages that are not in any sitemap ("Orphan" / "URLs Not in Sitemap").
4. Reconcile against Search Console
In GSC's Indexing > Sitemaps report, open each submitted sitemap and note "Discovered pages," then click through to the filtered Page Indexing report for that sitemap. This is the single most useful view in a sitemap audit: it shows exactly which submitted URLs are excluded and why, "Crawled, currently not indexed," "Duplicate without user-selected canonical," "Excluded by 'noindex' tag," "Page with redirect." Each bucket maps to a specific fix (see the table below). If the sitemap says 12,000 URLs and only 4,000 are indexed, the gap is your work list, prioritised by template.
5. Check freshness signals
Google has publicly stated it ignores <changefreq> and <priority>, so don't spend time tuning them, but it does use <lastmod> when it's consistently accurate. A sitemap where every URL shows the same lastmod timestamp (usually the sitemap generation time) trains Google to ignore the field on your site. Either emit the real content-modified date per URL or omit the tag entirely.
Sitemap Issues: Symptom, Check, Fix
| Symptom | How to check | Fix |
|---|---|---|
| GSC says "Couldn't fetch" | curl -sI the sitemap URL; check robots.txt isn't blocking it | Serve 200 with an XML content type; remove any Disallow covering the sitemap path |
| Redirected URLs in sitemap | Screaming Frog List mode → 3xx filter | Regenerate sitemap from final canonical URLs; fix the generator, not the file |
| Noindexed URLs in sitemap | List-mode crawl → Directives → noindex | Remove from sitemap (or reconsider the noindex): the two signals contradict each other |
| Non-canonical URLs submitted | List-mode crawl → Canonicals → Canonicalised | Submit only canonical versions; parameter/paginated variants stay out |
| Indexed pages missing from sitemap | Full crawl → Sitemaps tab → "URLs not in sitemap" | Fix generator inclusion rules (custom post types and taxonomies are the usual gaps on WordPress) |
| Large "Crawled, currently not indexed" bucket | GSC Page Indexing filtered to the sitemap | Quality problem, not sitemap problem: consolidate or enrich thin templates (see discovered, currently not indexed for the sibling bucket) |
| Stale sitemap after site changes | Compare sitemap URL count vs. CMS published count | Check sitemap caching (plugins cache aggressively); flush and confirm regeneration hooks fire on publish |
| Identical lastmod on every URL | View source of the sitemap file | Emit true per-URL modification dates or drop the tag |
| Multiple conflicting sitemaps | robots.txt + default paths + GSC submitted list | Keep one system of record; 404 or redirect the legacy files and resubmit |
Key Considerations for How to Audit Your XML Sitemap for SEO Issues
When approaching crawling, several factors require careful attention. Technical implementation must align with broader SEO objectives while maintaining site performance and user experience. The balance between optimization and over-optimization requires ongoing monitoring and adjustment based on performance data and algorithm changes. Industry benchmarks and competitor analysis provide context for evaluating your own implementation. For sitemaps specifically, the decision criterion is simple: a URL belongs in the sitemap if and only if it is a canonical, indexable page you would defend as worth indexing. On large sites, split sitemaps by template (products, categories, articles) even when well under the 50,000-URL cap: per-sitemap indexation rates in GSC then become a free monitoring dashboard showing exactly which section of the site Google is declining to index.
Implementation Best Practices
Successful implementation begins with thorough auditing of current state and clear goal definition. Document baseline metrics before making changes to enable accurate impact measurement: for sitemaps, record per-sitemap Discovered vs. Indexed counts from GSC before you touch anything. Prioritize changes based on potential impact and implementation effort, focusing on high-impact items first: dead and redirected URLs are quick generator fixes, while "crawled, currently not indexed" gaps are content projects. Test changes in staging environments when possible, and monitor closely after deployment to production. Maintain documentation of changes for future reference and troubleshooting.
Common Mistakes to Avoid
Several patterns consistently cause problems in crawling implementations. Rushing implementation without proper planning leads to errors that can be costly to fix. Ignoring the interplay between different SEO factors creates conflicts that undermine results: the classic sitemap example is submitting URLs the very same deploy noindexed. Failing to monitor and iterate based on performance data means missing optimization opportunities. Two more sitemap-specific traps: hand-editing a generated sitemap file (your fix disappears on the next regeneration, always fix the generator), and treating sitemap submission as a ranking tactic. A sitemap gets pages discovered; it has never made a weak page rank.
Measuring Success
Effective measurement requires defining appropriate KPIs aligned with business objectives. Track both leading indicators (rankings, impressions, technical metrics) and lagging indicators (traffic, conversions, revenue) to build a complete picture. For a sitemap audit the leading indicator is the per-sitemap indexation ratio in GSC (indexed divided by discovered), checked monthly. A healthy content site should sit well above 80%; ratios below 50% almost always indicate a quality or duplication problem rather than a technical one. Establish reasonable timeframes for evaluation, as SEO changes often take weeks or months to fully manifest in results. Compare performance against historical baselines to contextualize results.
Strategic Recommendations
Approach crawling as an ongoing program rather than a one-time project. Build processes for regular auditing, monitoring, and optimization: a quarterly list-mode crawl of every submitted sitemap takes under an hour and catches generator regressions before they compound. Stay current with industry developments and algorithm changes that may affect your strategy. Invest in education for team members to build internal capabilities. Consider how crawling fits within your broader digital marketing and business strategy for maximum impact.
Frequently Asked Questions
Run the full audit quarterly, and immediately after any migration, redesign, or CMS/plugin change that touches URL generation. In between, a monthly glance at GSC's per-sitemap Discovered vs. Indexed counts is enough to catch regressions: a sudden jump in discovered URLs or drop in indexed ones is your cue to re-crawl.
No, with one temporary exception: during a site migration, some SEOs briefly submit a sitemap of the old (now redirecting) URLs to accelerate Google's processing of the redirects. Outside that window, every sitemap URL should be the final, 200-status canonical. Remove the migration sitemap once the old URLs are reprocessed.
Not directly. It accelerates discovery and helps Google pick canonicals, which matters on large, deep, or frequently changing sites. On a 40-page brochure site with clean internal linking, the sitemap is nearly irrelevant; on a 200,000-SKU store it is critical infrastructure. Clean sitemaps improve crawl efficiency: the ranking impact is downstream of the pages themselves.
Because inclusion is a request, not a guarantee. Open the Page Indexing report filtered to that sitemap and read the exclusion reasons. "Duplicate" and "alternate page with proper canonical" buckets mean you are submitting non-canonical URLs; "Crawled, currently not indexed" at scale means Google judged the pages too thin or too similar to bother indexing, a content problem the sitemap cannot fix.
No. Google ignores both, and always has for practical purposes. The only optional tag worth maintaining is lastmod, and only if it reflects real content changes: an always-fresh fake lastmod is worse than none because Google learns to distrust it site-wide.
50,000 URLs or 50 MB uncompressed per sitemap file, whichever comes first. Larger sites use a sitemap index file pointing to multiple child sitemaps; the index itself can list up to 50,000 sitemaps, which is far more than any real site needs. Compressing with gzip reduces transfer size but does not raise the uncompressed limit.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
