Crawl Budget

Crawl budget is the number of URLs a search engine will fetch from your site in a given window — Google frames it as crawl capacity (what your server can take without degrading) multiplied by crawl demand (how much Google actually wants your URLs). When the budget gets eaten by garbage URLs, your real pages wait: new products take weeks to appear, price changes go stale in the index, and on big sites entire sections quietly fall out of rotation.

First, the honesty check most articles skip: if your site has under roughly 10,000 URLs and a healthy server, crawl budget is almost certainly not your problem, and "optimizing" it is procrastination. This becomes a real constraint on large e-commerce catalogs, classifieds, publishers with deep archives, and any site whose URL count is generated by parameters rather than editors. The full diagnosis workflow is in Crawl Budget: What It Is and When You Should Actually Care.

Where budgets actually go to die

Faceted navigation is the classic killer because it multiplies URLs combinatorially — this breaks shit fast. Ten filters with five values each isn't fifty URLs, it's every combination of them. A real pattern from a fashion retailer's logs:

/dresses?color=red&size=m&sort=price_asc&page=3
/dresses?size=m&color=red&sort=price_asc&page=3   ← same page, new URL
/dresses?color=red&size=m&sort=price_desc
/dresses?color=red&size=m&sessionid=8f3a...        ← infinite

Count what share of bot hits land on parameter URLs versus clean ones:

grep Googlebot access.log | awk '{print $7}' | grep -c '?'     # parameterized fetches
grep Googlebot access.log | awk '{print $7}' | grep -vc '?'    # clean fetches

On the site that prompted this entry, 71% of Googlebot's requests were hitting filter permutations of 30 category pages while 12,000 product URLs sat in "Discovered — currently not indexed." The crawler wasn't lazy; it was busy.

What wastes budget vs. what earns it

Notice budget is only ever spent at the fetch. That's why robots.txt is the one tool that truly saves requests, while noindex and canonicals only tidy up after the request already happened — the page must be fetched for either to be read.

How to check crawl budget on your own site

1. GSC → Settings → Crawl stats. Total crawl requests per day, divided by your count of indexable URLs, tells you the refresh cycle. 5,000 fetches/day against 40,000 URLs means a full pass takes over a week — assuming zero waste, which is never true.
2. In the same report, open "By response." Redirect and error percentages are pure overhead. Over ~10% combined deserves a ticket.
3. Analyze a week of access logs (Screaming Frog Log File Analyser, or the greps above). Segment fetches into money pages / parameters / redirects / errors. This is the single most persuasive chart you can put in front of engineering.
4. Cross-reference with the GSC Pages report: a big "Discovered — currently not indexed" bucket alongside heavy parameter crawling is the textbook budget-starvation signature.
5. Recheck 30 days after each fix. Freed budget shows up as rising fetch counts on clean URLs, then as faster indexing of new content.

Common mistakes

• Optimizing crawl budget on a 500-page site. Google will crawl all of it regardless. Spend the time on content or links instead.
• Using canonicals to "fix" faceted sprawl. Canonical tags consolidate signals but every variant still gets fetched to read them. Fix: block the patterns in robots.txt once you've confirmed the variants earn no organic entrances.
• Blocking parameters that were the only crawl path to products. Seen it: a robots.txt cleanup that orphaned half a catalog because pagination lived behind a blocked parameter. Fix: verify every product remains reachable through allowed links before you deploy the rule.
• Leaving expired listings returning 200. Classifieds and job boards bleed budget refetching thousands of dead ads. Fix: 410 them and they leave the crawl queue.
• Measuring nothing. If you can't say what percentage of yesterday's Googlebot hits reached indexable URLs, you're guessing. Logs first, opinions second.

FAQ

What's a "good" crawl budget number?

There isn't one — it's relative. The useful metric is coverage: bot fetches per day ÷ indexable URLs, and the share of fetches hitting URLs you actually want indexed. Improve the ratio, not the raw count.

Does robots.txt Disallow increase my crawl budget?

It doesn't raise the ceiling, it stops the leak. Google reallocates requests it would have wasted on blocked paths toward the URLs still allowed — which is what you wanted anyway.

Do sitemaps increase crawl budget?

No, but they aim it. A clean sitemap with truthful lastmod values helps Google spend its existing budget on the URLs that changed instead of re-verifying ones that didn't.

Does page speed affect crawl budget?

Server response time does, directly: crawl capacity is calibrated to how fast your server answers without erroring. Client-side metrics like LCP are a ranking-side concern, not a crawl-side one.

Is crawl budget shared across bots?

Each crawler manages its own schedule — Googlebot's budget is separate from Bingbot's or GPTBot's. But they all hit the same server, so aggregate bot load can drag response times and shrink everyone's capacity at once.