Crawl Stats

What the Crawl Stats report is

Crawl Stats is the report inside Google Search Console (Settings > Crawl stats) that shows how Googlebot actually hit your site over the last 90 days: total crawl requests, total download size, and average response time. If crawling quietly breaks, this is usually the first place you can see it in hard numbers instead of guessing.

It matters because a healthy site should show a steady or gently rising request line with fast responses. A cliff, a spike in server errors, or response times climbing into the seconds are early warnings that Google is having a harder time reading your pages, which eventually shows up as slower indexing and stale search results.

Where to find it and what each chart means

The report is not in the main left-hand menu. Open a domain or URL-prefix property, click the gear-icon Settings at the bottom of the sidebar, then Crawl stats > Open report. Note that Crawl Stats is only available for domain properties and the root URL-prefix property, not for a subfolder property.

At the top you get three totals with a 90-day trend line:

• Total crawl requests - every URL Googlebot requested, including repeats, redirects, and failed fetches.
• Total download size - bytes downloaded across all those requests. Sudden jumps often mean bloated HTML, uncompressed assets, or a new section being discovered.
• Average response time - milliseconds Google waited for your server. This is the number that quietly wrecks crawl efficiency when it drifts up.

Below the totals, Google breaks the same requests down four ways: By response (200, 301, 404, 5xx, etc.), By file type (HTML, image, JS, CSS, JSON), By purpose (Discovery of new URLs vs Refresh of known ones), and By Googlebot type (Smartphone, Desktop, Image, Page resource load, and so on).

A worked example

Say your property shows 1.2M total requests, 42 GB downloaded, and a 90-day average response of 780 ms. You drill into By response and see this:

OK (200)                    68%
Moved permanently (301)     14%
Not found (404)              9%
Server error (5xx)           6%   <- was 1% last month
Other client error (4xx)     3%

That 5xx jump from 1% to 6% is the story. Six percent of Google's requests are hitting server errors, which is why your average response time crept up too - timeouts and error pages drag the average. That is a server or hosting problem to chase, not a content one.

How to read the numbers together

No single metric tells you much alone. The combinations are where the diagnosis lives.

The Host status box at the top deserves its own glance every time. It rolls up three availability checks - robots.txt fetch, DNS resolution, and server connectivity - into a green check or a red flag. A red flag here means Google physically struggled to reach you, and that outranks every other number on the page.

How to check it on your own site

1. Open the correct Search Console property (domain or root URL-prefix, not a subfolder), click Settings, then Crawl stats > Open report.
2. Look at Host status first. If it is not green, click in and fix availability before analyzing anything else.
3. Read the three totals and their trend lines. Note any cliff, spike, or steady climb in response time.
4. Open By response. Confirm 200s dominate. Flag any growth in 5xx, 404, or 3xx.
5. Open By purpose. If Discovery dwarfs Refresh, you probably have URL bloat feeding Google junk to crawl.
6. Open By Googlebot type. Smartphone should lead. Heavy "Page resource load" can mean render-blocking assets Google keeps re-fetching.
7. Cross-check against your server logs for the same window if you have them - log data is the ground truth Crawl Stats summarizes.

Common mistakes and how to fix them

• Treating total requests as a KPI to maximize. More crawling is not better. A tight site with clean URLs crawls efficiently. If requests balloon, look for what junk Google is being fed, not for a reason to celebrate.
• Ignoring the response-time trend. An average creeping from 300 ms to 900 ms over a few weeks is a real signal even if nothing looks "broken." Slow responses shrink how much Google will crawl per visit.
• Panicking at 404s that belong there. Some 404s are correct - deleted junk should 404. The problem is 404s coming from your own internal links or sitemaps. Chase those, ignore the rest.
• Confusing Crawl Stats with the Pages (Indexing) report. Crawl Stats is about fetching. Indexing is about what made it into the index. A page can be crawled constantly and still not be indexed.
• Only checking after traffic drops. By then the damage is weeks old. Glance at Host status and the response trend monthly so you catch server trouble while it is small.

FAQ

How far back does the Crawl Stats report go?

90 days. There is no way to extend the window inside the report, which is one reason server-log archiving is worth setting up for longer-term trend analysis.

My crawl requests dropped. Is that bad?

Not automatically. If you removed a pile of low-value URLs, fewer requests is healthy. It is only alarming when the drop pairs with a 5xx spike, a robots.txt problem, or a fall in indexed pages. Read it alongside Host status and the response breakdown.

Is Crawl Stats the same as crawl budget?

No, but they are related. Crawl Stats is the reporting; crawl budget is the underlying capacity and demand that produces those numbers. See our explainer on crawl budget for when budget actually matters.

Why does response time matter so much here?

Google adjusts how much it crawls partly based on how fast your server answers. Faster responses let Google fetch more URLs in the same window. This ties directly into your crawl rate - slow servers pull it down.

Can I see which exact URLs Google crawled?

Not in this report - it is aggregated. For per-URL detail you need server log file analysis, which shows every individual Googlebot request with timestamps and status codes.

Related reading

• Crawl Rate - how fast and how often Googlebot fetches.
• Using Crawl Stats to troubleshoot a site move.
• Crawl Trap - what feeds runaway Discovery requests.