Log File Analysis

Log file analysis is the practice of reading your web server's raw access logs to see exactly what search engine crawlers requested, when, and what response they got. Unlike a crawl simulation or Search Console's sampled charts, logs are the ground truth: every single Googlebot hit, timestamped, with the actual status code your server returned.

Why it matters: everything else in technical SEO is inference. A crawler like Screaming Frog shows what a bot could fetch; logs show what Googlebot actually fetched. On sites past a few tens of thousands of URLs, the gap between those two is where crawl budget dies: bots hammering parameter junk, faceted URLs, and dead redirects while your new content waits days for a first visit. You cannot see that anywhere except the logs.

What a log line actually says

Here are three real-format lines (Apache/nginx combined log format) of the kind you'll see when you grep for Googlebot:

66.249.66.35 - - [28/Jun/2026:04:12:33 +0000] "GET /products/blue-widget/ HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.72 - - [28/Jun/2026:04:12:41 +0000] "GET /products/?color=blue&sort=price_asc HTTP/1.1" 200 47950 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.35 - - [28/Jun/2026:04:13:02 +0000] "GET /old-category/widgets/ HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Line one is healthy. Line two is Googlebot spending a request on a filtered-and-sorted parameter URL that's near-duplicate of the category page. Line three is a crawl hit wasted on a redirect that's probably been live for years. Multiply lines two and three by 40,000 a day and you've found your indexing lag.

Reading the fields

Note what's absent: no referrer of value (bots send -), no rendering, no JavaScript execution. Logs tell you about requests, not about what the bot did with the response. Pair them with crawl data for the full picture.

How to check it on your own site

1. Get the raw access logs. cPanel: Metrics → Raw Access. nginx: usually /var/log/nginx/access.log. Behind Cloudflare or a load balancer, make sure logs record the real client IP or bot verification is impossible. This is honestly the hardest step: half of log-analysis projects die at "ops can't give us the logs."
2. Isolate verified Googlebot: grep Googlebot access.log for the first pass, then verify a sample of IPs with host 66.249.66.35 (must resolve to *.googlebot.com or *.google.com and back).
3. Run the file through GoAccess (goaccess access.log --log-format=COMBINED) for instant per-status, per-URL breakdowns, or use the log file analyzer for an SEO-oriented view without shell work. Screaming Frog Log File Analyser is the desktop route and joins log data to crawl data.
4. Answer four questions: What share of bot hits land on URLs you actually want indexed? What share hit parameters, redirects, and 404s? Which important sections get crawled rarely or never? How fast does new content get its first Googlebot hit?
5. Segment by directory and by bot, and trend it over at least 30 days: single-day samples lie, because Googlebot's attention is bursty. For rolling your own, there are ready-made Python log file parsers to build on.

Ten-minute triage: the one-liners

Before any tooling, these shell one-liners against a combined-format access.log answer the first-pass questions. Each assumes the standard field order shown in the sample lines above (status is field 9, path is field 7).

# Status code distribution for Googlebot hits
grep Googlebot access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Top 20 most-crawled URLs
grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Share of bot hits wasted on parameter URLs
grep Googlebot access.log | awk '{print $7}' | grep -c '?'

# Every 404 Googlebot keeps re-requesting
grep Googlebot access.log | awk '$9==404 {print $7}' | sort | uniq -c | sort -rn

# Crawl activity per directory (first path segment)
grep Googlebot access.log | awk -F'/' '{print $2}' | sort | uniq -c | sort -rn | head

# AI crawlers hitting the site, by user-agent
grep -E 'GPTBot|ClaudeBot|PerplexityBot|Bytespider|CCBot' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

# Unique verified-range Googlebot IPs to sample for reverse DNS
grep Googlebot access.log | awk '{print $1}' | sort -u | head

Rotated logs are usually gzipped; swap grep for zgrep (or zcat access.log.*.gz | grep ...) to include them, and run across at least 30 days of files before trusting any percentage.

Which bots to expect, and what each one proves

Segmenting by user agent turns one number into a diagnosis. A drop in total bot hits means nothing until you know which bot stopped coming. Run the per-bot counts separately, then read them like this:

The AI crawlers deserve their own pass, because their robots.txt tokens, rendering behaviour and retrieval models differ from each other in ways that change what you should allow. The AI crawler comparison of GPTBot, ClaudeBot and PerplexityBot breaks down each one. Before you block anything, remember that crawl-level blocks and index-level directives are different controls: see the meta robots and X-Robots-Tag reference for which lever does what.

Red-flag thresholds worth acting on

Rules of thumb from repeated audits, starting points for judgment rather than lab constants:

Common mistakes

• Trusting the user-agent string. A hell of a lot of "Googlebot traffic" in raw logs is scrapers wearing a costume. Always reverse-DNS-verify before drawing conclusions, or your "crawl budget problem" is actually a scraping problem.
• Analyzing one day of logs. Googlebot crawls in waves. A 24-hour slice will convince you a section is ignored when it was crawled thoroughly last Tuesday. Thirty days minimum, ninety for seasonal sites.
• CDN logs vs origin logs confusion. If Cloudflare serves a cached copy, the request may never reach your origin log. Audit at the layer that sees all bot traffic: for cached sites that means the CDN's logs, not the server's.
• Counting hits instead of unique URLs. 100,000 Googlebot hits sounds great until you see 60% of them went to 200 infinite-calendar URLs. Always report both hit volume and unique-URL coverage.
• Finding waste and stopping there. The log analysis isn't the deliverable; the robots.txt rule, the parameter cleanup, or the redirect fix is. Tie every finding to a change, then re-run the logs a month later to prove the shift.

For the full workflow (sampling strategy, joins against crawl data, and reporting templates) see the complete log file analysis guide. If you're dealing with multi-CDN setups and log volumes in the terabytes, the enterprise log file analysis piece covers the plumbing.

FAQ