Crawl Rate

What crawl rate actually is

Crawl rate is how often and how fast Googlebot fetches pages from your site - the number of requests it makes and the pace it makes them at. Get it wrong in either direction and you either hammer your own server or leave fresh content sitting undiscovered for days.

The important thing to understand up front: you no longer set this manually. Google retired the crawl-rate limiter tool in Search Console in 2023. Crawl rate is now decided automatically by Google based on your site's health and how much new or updated content it thinks is worth fetching. Your job shifted from "turning a dial" to "making the automatic system want to crawl you more."

How Google sets it now

Google balances two things it calls crawl capacity and crawl demand. Capacity is how much crawling your server can take without slowing down; demand is how much Google wants to crawl based on freshness and popularity. The rate you actually get is the lower of the two.

So a slow server does double damage: it lowers your capacity ceiling directly, and the resulting timeouts make Google back off further. Speed is the single biggest lever most sites have.

What "reporting a problem" looks like

The old limiter is gone, but Google left a safety valve. If Googlebot is genuinely overloading your server, you can file a crawl-rate reduction request through the Search Console Help flow, and in an emergency you can return a temporary 503 or 429 status to slow it immediately. That last one is a blunt tool - use it only for real overload, and only briefly, because sustained 5xx responses hurt indexing.

# Emergency-only: temporarily tell Googlebot to back off.
# Return this for a short window if the bot is truly overloading you.
HTTP/1.1 503 Service Unavailable
Retry-After: 3600

Leaving a 503 up for days is a self-inflicted wound - Google will drop pages from the index if it keeps getting told the site is unavailable.

How to check your crawl rate

1. Open Search Console, go to Settings > Crawl stats > Open report. This is your primary window into rate over the last 90 days.
2. Read Total crawl requests and its trend line - that is your effective crawl rate over time.
3. Check Average response time. If it is climbing, your capacity ceiling is dropping and rate will follow.
4. Look at Host status. A red flag on server connectivity or robots.txt means Google is being forced to throttle.
5. Open By purpose - a healthy site refreshes known pages and discovers new ones in reasonable proportion. Runaway Discovery signals a crawl trap eating your rate.
6. For per-URL truth, cross-check with server log file analysis. Logs show exactly when Googlebot hit each URL and what it got back.

The full breakdown of every chart lives in our Crawl Stats reference - crawl rate is basically what that report measures.

Common mistakes and how to fix them

• Looking for the old crawl-rate slider. It is gone as of 2023. If a tutorial tells you to set it in Search Console settings, that tutorial is stale. Focus on server speed and content freshness instead.
• Using 503s to "manage" crawling routinely. A 503 is emergency-only. Sustained 503/429 responses tell Google the site is broken and pages start dropping out of the index.
• Blaming crawl rate for indexing problems. Plenty of pages get crawled often and still are not indexed. If pages are crawled but not indexed, the issue is quality, duplication, or canonicals - not rate.
• Ignoring server response time. Teams obsess over content and never notice their TTFB crept to two seconds. That quietly caps how much Google will crawl. Fix the server and rate usually recovers on its own.
• Feeding Google junk URLs. Faceted navigation, session IDs, and endless parameters burn crawl capacity on pages you do not want indexed, starving your real content. Block or consolidate them.

FAQ

Can I make Googlebot crawl my site faster?

Not with a switch. You raise it indirectly: speed up your server, publish and update valuable content, keep sitemaps clean, and strengthen internal links. Google raises the rate when your site can handle more and there is more worth fetching.

Google stopped crawling a section of my site. Why?

Usually low demand (the pages look stale, thin, or duplicative) or a capacity problem (that section is slow or throws errors). Check Crawl Stats response times and the By response breakdown for that path, and look at whether those pages actually change.

Is a higher crawl rate always good?

No. Crawling is a means, not a goal. A lean site that crawls efficiently beats a bloated one that gets crawled constantly but wastes it on parameter junk. What you want is Google spending its crawling on pages that matter.

Does crawl rate affect rankings directly?

Not directly. It affects how quickly new and updated pages get discovered and refreshed, which affects how current your search results are. The ranking impact is downstream of that freshness, not a direct signal.

How do crawl rate and crawl budget relate?

Crawl rate is the pace of fetching; crawl budget is the total capacity-plus-demand that pace draws from. Most small and mid-size sites never hit a budget ceiling - see crawl budget explained for when it genuinely matters.

Related reading

• Crawl Stats - the Search Console report that measures your crawl rate.
• Googlebot - the crawler doing the fetching.
• Log File Analysis - per-URL ground truth on what Googlebot fetched and when.