How To: Build A Content Audit Assistant With ScreamingFrog and AI

The emergence of AI-powered search and large language models has created new considerations for content optimization. This resource explores how content creators can adapt their strategies to maintain visibility as AI increasingly mediates information discovery.

Understanding AI Content Consumption

Large language models process content differently than traditional search algorithms. They seek factual, well-structured information that can be confidently cited. Understanding these consumption patterns helps optimize content for AI visibility alongside traditional search performance.

Optimization Approaches

Research indicates specific approaches improve AI citation likelihood: including statistics with sources, maintaining factual accuracy, structuring content for easy extraction, and demonstrating clear expertise. These tactics complement rather than replace traditional SEO fundamentals.

Strategic Implications

As AI search grows, early optimization provides compounding advantages. Building citation presence now establishes your content as an authoritative source that AI systems learn to trust. The convergence of good SEO and good GEO practices suggests quality-focused strategies benefit both channels.

This resource provides actionable guidance for the evolving intersection of content strategy and AI visibility.

Source: https://www.ianlurie.com/content/audit-existing-content-screamingfrog-ai/

SEO ProCheck practitioner notes

The idea in the source is simple and powerful: let the crawler gather the pages and let a language model make the qualitative call that used to eat days of manual review. Below is how we wire it up in practice, including the Custom Extraction selector, the prompt shape, and the decision rules that turn a spreadsheet of scores into a work queue.

Extract the content, not the chrome

Point Screaming Frog at the main content container so the model scores the article and not the navigation, footer, and sidebar. In Configuration, Custom, Extraction, add an extractor that targets the body wrapper with a CSS path such as main article or .entry-content and set it to return the text content. Add a second extractor for the visible word count. Extracting the real copy matters because a page can look thin in raw HTML yet be padded with template markup, and a page can look long yet be mostly navigation.

Write a scoring prompt that returns strict JSON

The model needs a tight contract or the output column becomes unusable. Ask for a fixed shape and nothing else. A prompt like the one below keeps every row parseable:

You are an SEO content auditor. Given the URL, title, and body text,
return ONLY valid JSON: {"score":0-100,"verdict":"keep|enrich|merge|redirect",
"reason":"one short sentence"}. Score on depth, search intent match,
uniqueness, and freshness. Do not add commentary outside the JSON.

Feed it the extracted title and body as variables. Because the reply is strict JSON, you can split it into columns after export, or parse it directly if you call the API from a script rather than inside the crawl.

Turn scores into decisions

The verdict field is the point of the whole exercise. Treat keep as pages that already earn their place, enrich as pages with the right intent but thin execution, merge as near duplicates that should be consolidated into one stronger URL, and redirect as pages with no unique value and no traffic. Consolidating overlapping pages and enriching thin but relevant ones is almost always higher leverage than deleting, which is why the queue below leads with enrich and merge rather than removal.

Validate before you act

An AI verdict is a strong first pass, not a final ruling. Cross reference the queue against Search Console clicks and impressions before you redirect anything, because a page the model rates as thin may still catch a steady trickle of qualified queries. Keep the original crawl export as a backup, apply changes in small batches, and recrawl to confirm the fixes landed. The assistant compresses the triage, but a human still signs off on removals.

Verdict to action map

What has changed since this how to

When this walkthrough was published, connecting a language model to Screaming Frog meant wiring in your own OpenAI API key through the Custom JavaScript feature. Screaming Frog has since added native AI features: recent versions let you run AI prompts directly against crawled pages and configure them without hand written scripts, which lowers the setup cost considerably. The workflow is otherwise identical, extract the content, prompt for a verdict, decide per URL, but you now have a choice between the built in integration and your own API call. If you already run scripted audits, the external call still gives you the most control over the prompt and the model.

Related reading

• A practical framework for LLM consumption
• Browser extensions and daily SEO stack
• More SEO articles and guides

Frequently asked questions