Finding a Competitor's Sitemap, and Turning It Into a Change Log

Why the file is worth watching

An XML sitemap is a declaration of intent. It lists the URLs a site wants crawled, and most generators stamp each one with a lastmod date. That combination is more useful than it looks. New entries tell you what they launched. Disappeared entries tell you what they retired or quietly folded into something else. A cluster of lastmod dates all moving in the same week tells you a team went through a section on purpose.

None of that requires a tool licence or a login. It is a text file they publish for crawlers, and you are allowed to read it. The only thing you need is a copy from last month to compare against, which is the part nobody has when they want it.

Step one, find the file

Before any of this works you have to locate the thing, and that turns out to be the step that fails. I ran two routes against 184 sites: guess the conventional paths, and read the Sitemap: line out of robots.txt. Neither route is close to complete on its own.

The guess covers /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap1.xml. Of the 78 hits, 74 were plain /sitemap.xml, which is why guessing feels more reliable than it is.

The practical lesson sits in the middle row. Reading one extra file takes you from 78 sites to 116, and the sites you pick up that way are not obscure ones. Microsoft, YouTube, Wikipedia, Pinterest, X and Vimeo are all in the group where a guess returns nothing and the declaration works.

It is worth seeing how little you would have guessed:

wikipedia.org   /w/rest.php/site/v1/sitemap/0
nytimes.com     /sitemaps/new/news.xml.gz
vimeo.com       /marketing.xml.gz
paypal.com      /paypal-sitemap-index.xml
discord.com     /servers/servers-sitemap-index.xml
ibm.com         /content/sitemap/href-sitemap-index.xml
dropbox.com     /sitemapindex.xml

And the declaration is not a guarantee. Of the 100 sites that declared a sitemap, the first declared URL served real XML on 81. 15 served none at all, split between four that answered with a 403 or a 404 and eleven that answered 200 with something that was not XML, which in most cases looked like the same bot handling I measured in the crawler access study. A further 4 declared a gzipped file, which is perfectly valid and simply beyond the plain reader I used.

Step two, take the baseline

Fetch the file, parse out every URL with its lastmod, and save that as a dated snapshot. That is the whole job. The snapshot is worthless on its own, which is the thing worth being honest about: a change log cannot be created retroactively. The archive services do not reliably keep copies of XML sitemaps, and a site that rebuilt its sitemap last week has already overwritten the evidence.

So the first run buys you nothing and the fourth run is where it starts paying. Weekly is the sensible cadence for most sectors, because it is frequent enough to attribute a change to a week and slow enough that the diffs stay readable.

Two things to store beyond the URL list. Keep the declared sitemap set from robots.txt, because sites add and remove sitemap files and that is itself a signal. And keep the URL count per file, since a file that drops from 40,000 entries to 12,000 is worth knowing about even before you look at which URLs went.

Step three, read the diff

Three categories come out of a comparison, and they are worth different amounts.

The lastmod caveat is the one that catches people. Plenty of platforms stamp the current date on every URL whenever anything is published anywhere on the site, which makes the field pure noise. The test is cheap: if a diff shows every single URL updated on the same date, the field is decorative on that site and you should ignore it from then on. If a diff shows forty URLs in one directory moving and nothing else, somebody did a content refresh and you have just watched it happen.

The traps, all of which I hit

The sitemap is usually not one file. A sitemap index points at children, and the children are what hold URLs. Some sites go much further than that: samsung.com declares 94 sitemap lines in robots.txt, hp.com 89, hostinger.com 88. Fetching "the sitemap" on one of those and calling it done gives you a fraction of the picture.

Gzipped files are normal. Six of the sites I checked declare a .gz first. Anything reading these needs to decompress, or it will report an empty sitemap and you will believe it.

A 200 is not a sitemap. Eleven declared URLs answered 200 with HTML. Check that the body actually starts with an XML declaration before you parse it, or a bot challenge page becomes a sitemap with zero URLs and your diff reports a catastrophic overnight deletion that never happened.

Sitemaps sometimes point at the wrong host. Staging hostnames and localhost URLs turn up in production sitemap indexes more often than anyone would like. If a child sitemap will not resolve, read the URL before assuming the fetch failed.

Watch your own request volume. A site with 90 sitemap files is 90 requests, and doing that to one host in a burst is how you get rate limited or firewalled. Space them out. If you are fetching a lot of files from a single origin, treat it the same way you would treat a crawl.

What this is not

A sitemap tells you what a site publishes. It says nothing at all about what works. A competitor can add 400 pages that earn nothing, and the diff will look identical to 400 pages that earn well. Structure describes activity, and only ranking or traffic data describes results, so a change log is the question rather than the answer. It is very good at telling you where to point the next question.

It is also not a substitute for reading the pages. The most useful version of this habit is a weekly diff that takes two minutes, followed by opening the three most interesting new URLs and actually looking at them.

Related reading on the same file from the technical side: the 403 URLs in XML sitemaps check covers what it means when your own sitemap lists a URL that refuses crawlers, which is the mirror image of the trap above. And if you are here because of the AI crawler question, the current crawler names and which ones sites block is the companion piece to this one.

Questions people ask about this

Method: for each of 184 hosts, robots.txt was fetched once and every Sitemap: line recorded, then the first declared URL was fetched, then up to five conventional paths were tried. A hit means HTTP 200 with a body beginning in an XML declaration. Measured 18 September 2026 from a single residential connection in the United States. The sample is the same 184 hosts that returned 200 and HTML to a browser in the September crawler access study.