Screaming Frog

Screaming Frog SEO Spider is a desktop website crawler that fetches your site the way a search engine bot would and lays out everything it finds — URLs, status codes, titles, canonicals, links, directives — in a giant sortable table. It's the workhorse of technical SEO: when you need to know what's actually on a site rather than what the CMS claims, this is the tool that tells you.

Why a 2010-era desktop app still runs the industry

Screaming Frog wins on three things: control, transparency, and price. You decide exactly what gets crawled, how fast, with what user-agent, rendered or not. It shows you raw data instead of opinions — no scores, no "site health" gauge, just the facts in columns you can filter and export. And the license is a few hundred dollars a year (a free tier crawls 500 URLs), which makes it the cheapest serious crawler by a wide margin.

The honest flip side: the interface is a wall of tabs that intimidates newcomers, it tells you what but never why it matters (prioritization is your job — that's the core trade-off versus Sitebulb), and because it runs on your machine, very large crawls are constrained by your RAM and disk unless you configure database storage mode. It's a power tool, not a report generator. Handing a raw Frog export to a client is a hazing ritual, not a deliverable.

Crawl configurations that map to real jobs

The default "enter URL, press Start" crawl is maybe a fifth of what the tool does. The leverage is in configuration:

For copy-paste starting points — migration audits, hreflang checks, log-file cross-references and more — the site keeps a set of Screaming Frog crawl recipes worth saving.

How to run a first useful crawl

1. Switch to database storage mode (Config → System → Storage Mode) before anything else if the site exceeds ~100k URLs or you want crawls saved automatically. Memory mode dies mid-crawl on big sites and takes your afternoon with it.
2. Set a sane speed (Config → Speed): 2–5 URLs/second for sites you don't own the infrastructure for. Hammering a shared host is how you get your IP blocked mid-audit.
3. Crawl, then work the tabs in order: Response Codes (4xx/5xx and redirect chains), then Canonicals, then Directives, then Page Titles/Meta. Structure problems first, cosmetics last.
4. Use Crawl Analysis (Crawl Analysis → Start) after the crawl completes — link scores, orphan detection via sitemap/GSC comparison, and several reports are empty until you run it. Everyone forgets this once.
5. Export targeted reports, not the whole table: Reports → Redirects → Redirect Chains, and Bulk Export → Links → internal link sources for whatever you're fixing.

Common mistakes (and fixes)

• Crawling in memory mode until it falls over. Fix: database storage mode as your default, always. There is no downside on a modern SSD.
• Auditing a JS-heavy site without rendering and concluding half the content is missing. Fix: check how the site is built first; if the raw HTML is a shell, enable JS rendering and compare raw vs rendered.
• Trusting the crawl as the full picture. The Frog only sees what's linked from your start point. Orphan pages, parameter URLs Google found years ago, and log-file reality are invisible. Fix: layer in sitemap crawls, GSC data, and log file analysis.
• Ignoring robots.txt settings. By default the Frog respects robots.txt, so blocked sections silently vanish from your audit — or worse, someone sets it to ignore and reports "indexable" URLs that Google can never fetch. Fix: know which mode you're in and say so in the report.
• Shipping raw exports as deliverables. Fix: your job is the diagnosis and the priority order; the CSV is your working material, not the report.

FAQ

Does Screaming Frog crawl like Googlebot?

Close, but not identical. It follows links and can render JavaScript with a Chromium engine, and you can set its user-agent to Googlebot. But it doesn't share Google's crawl scheduling, index selection, or years of URL memory — a clean Frog crawl doesn't guarantee clean indexing.

Free version or paid?

Free crawls 500 URLs with core features; paid (per-seat annual license) unlocks unlimited URLs, JS rendering, custom extraction, API integrations, scheduling, and config saving. Anyone doing technical SEO professionally recoups the license on the first project.

Screaming Frog or Sitebulb?

Frog for control, extraction, and raw-data workflows; Sitebulb when you want built-in prioritization and client-friendly visuals. Many consultants run both. The full argument is in Screaming Frog vs Sitebulb.

How big a site can it handle?

In database storage mode on decent hardware, millions of URLs — crawls just take time. For very large enterprise sites, cloud crawlers ease the hardware constraint, at 10–50x the price.

Why does my crawl show fewer URLs than Google has indexed?

Usually orphan pages (indexed but no longer linked internally), parameter URLs, or sections behind robots.txt rules the Frog is respecting. Compare the crawl against your sitemap and GSC's page report to find the gap — the gap itself is often the audit's most valuable finding.