Crawl Priority

What crawl priority means

Crawl priority is how Google decides which of your URLs to fetch, and how often, out of everything it could be crawling. You don't set it with a switch — it's the output of signals like internal link prominence, XML sitemaps, freshness, and a page's overall importance — but on a big or slow site those decisions determine how fast new and updated pages actually show up in results.

This matters most when you have more URLs than Google cares to crawl in a sitting. If your important pages sit low in the priority order, they get fetched late or infrequently, updates lag, and fresh content waits days to be discovered. On a 50-page brochure site you'll never notice. On a 500,000-URL store, it's the difference between "indexed today" and "indexed next month."

A real example

An ecommerce site launches 300 new product pages on a Monday. The ones linked straight from the homepage and top-level category pages get crawled within hours. The ones buried five clicks deep, reachable only through a paginated "page 14 of 22" listing and absent from the sitemap, don't get touched for a week and a half. Same site, same day, same crawler — different priority, because the first set sent strong importance signals and the second set sent almost none.

The fix wasn't to "increase crawl budget." It was to surface the buried products in the sitemap, link them from relevant category hubs, and stop wasting fetches on faceted-filter URLs that produced thousands of near-duplicate pages. Reallocating attention beat begging for more of it.

Signals that shape crawl priority

Crawl priority is closely tied to crawl budget: budget is roughly how much Google is willing to fetch, and priority is the order it fetches things in. You influence both mostly through the same levers.

How to check your crawl priority

1. Open Search Console → Settings → Crawl stats. Look at total crawl requests over time, average response time, and the breakdown by response code and file type.
2. Run a log-file analysis. Server logs show exactly which URLs Googlebot fetched, how often, and which it ignores — the ground truth no other tool gives you.
3. Compare your important URLs against that crawl activity. Are your money pages being fetched regularly, or are they missing from recent crawls?
4. Crawl the site yourself and record click depth. Flag anything important sitting more than three clicks from the homepage.
5. Audit what's eating crawls: faceted URLs, session IDs, endless pagination, redirect chains, soft 404s. These pull fetches away from pages that matter.
6. Check your XML sitemap covers every important URL and that lastmod is accurate — lying about freshness trains Google to distrust it.

Common mistakes and how to fix them

• Thinking you can set crawl priority directly. The old sitemap <priority> tag is effectively ignored. Fix: influence priority through links, freshness, and site health — not a number in a file.
• Letting faceted navigation spawn infinite URLs. Filter combinations create millions of thin URLs that soak up crawling. Fix: block or canonicalize filter parameters and keep crawlers on canonical pages.
• Burying key pages deep in the architecture. Five clicks from home tells Google the page barely matters. Fix: flatten the path and link important pages from prominent hubs.
• Ignoring server performance. Slow responses and 5xx errors make Google back off. Fix: cut response times and error rates so the crawler ramps up instead of throttling.
• Stale or bloated sitemaps. Sitemaps full of redirects, 404s, or noindex URLs waste discovery. Fix: keep them clean, current, and limited to indexable canonicals.

FAQ

Can I directly control which pages Google crawls first?

Not directly. You steer it with internal links, sitemap coverage, freshness, and by removing the low-value URLs that waste crawler attention. There's no dial that says "crawl this one next."

Does the sitemap priority tag do anything?

Practically nothing. Google has said it largely ignores the <priority> value. The useful part of a sitemap is which URLs you include and an honest lastmod, not a made-up priority score.

Is crawl priority the same as crawl budget?

They're related but distinct. Budget is how much Google is willing to crawl on your site; priority is the order it works through URLs. Fixing one usually helps the other, since both respond to link structure, freshness, and site health.

How do I know if crawl priority is actually a problem for me?

If your site is under a few thousand pages and content indexes within a day or two, it isn't your bottleneck — look elsewhere. Worry about it when you have large URL counts, slow discovery of new pages, or Crawl stats showing Google spending fetches on junk URLs.

Will improving crawl priority raise my rankings?

Only indirectly. Better crawling gets your pages discovered and refreshed faster, which is a prerequisite for ranking — but crawling isn't a ranking factor by itself. A page still has to earn its position on relevance and quality.