Screaming Frog: Advanced Usage Guide

Screaming Frog Fundamentals

Screaming Frog SEO Spider is the industry-standard desktop crawler for technical SEO audits. It crawls websites like a search engine, collecting data about URLs, status codes, meta data, links, images, and more. While the free version crawls up to 500 URLs, the paid license unlocks unlimited crawling, JavaScript rendering, custom extraction, and advanced features. Understanding Screaming Frog's capabilities transforms audit efficiency and enables deeper technical analysis than browser-based tools typically allow.

Configuration for Different Audits

Proper configuration ensures useful crawl data. For standard audits, crawl in "Spider" mode respecting robots.txt. For comprehensive audits, consider ignoring robots.txt to see everything (but only on sites you control). Configure JavaScript rendering (Configuration > Spider > Rendering) when auditing JavaScript-heavy sites. Set crawl limits appropriately for large sites. Configure extraction of custom elements using CSS selectors or regex. Enable API connections for PageSpeed, Analytics, and Search Console data enrichment. Save configuration profiles for different audit types.

Essential Reports and Exports

Export data strategically for analysis. The "All Inlinks" export shows complete internal linking with anchor text. "Crawl Overview" provides executive summary statistics. Custom exports filter specific issues like pages missing meta descriptions or with duplicate titles. Use the "Compare" feature to diff crawls over time, identifying changes between audits. Schedule crawls for regular monitoring. Export to Google Sheets or Excel for further analysis, filtering, and visualization. The built-in reports (Issues, Directives, Canonicals) quickly identify common problems.

Custom Extraction and XPath

Custom extraction pulls specific data from crawled pages using CSS selectors, XPath, or regex. Extract product prices, review counts, author names, schema data, or any page element for analysis at scale. Use CSS selectors like "h1.product-title" for simple extraction. Use XPath for complex selections like "//script[@type='application/ld+json']" for JSON-LD schema. Extract multiple elements to columns for comprehensive content audits. Combine with List Mode to audit specific URLs rather than full crawls. Custom extraction transforms Screaming Frog from crawler to data extraction tool.

Integration with Other Tools

Screaming Frog integrates with external APIs for enriched data. Connect Google Analytics to see sessions, bounce rate, and conversions per URL. Link Search Console for impressions, clicks, and position data. Enable PageSpeed Insights API to gather Core Web Vitals and performance scores during crawl. Connect Ahrefs or Majestic for backlink metrics per page. These integrations create comprehensive datasets combining crawl data with performance, search, and link metrics. Export combined data for advanced analysis in spreadsheets or BI tools. API integrations require appropriate accounts and API keys.