Validating Structured Data at Scale

Why scale changes the problem

Validating one page is trivial: paste the URL into the Schema Markup Validator or the Rich Results Test and read the result. That approach collapses the moment you have thousands of URLs generated from a handful of templates. A single bug in a Product template can invalidate ten thousand product pages at once, and you will never catch it by spot checking. Validation at scale means treating schema like code: extract it programmatically from every URL, run it through validators, and surface errors grouped by type and template so one fix repairs the whole class of pages.

The pipeline: crawl, extract, validate, report

Every scalable approach follows the same four stages. Crawl the URL set, extract the JSON-LD from each page, validate each block against the vocabulary and Google's rules, then report the errors grouped so they are actionable.

Screaming Frog for mid sized sites

For sites up to a few hundred thousand URLs, Screaming Frog does the crawl and validation in one pass. Enable structured data extraction under Config > Spider > Extraction, and turn on both Schema.org Validation and Google Rich Results Validation. After the crawl, the Structured Data tab lists every URL with error and warning counts, and the filters let you export just the pages with errors. The advanced workflow, including scheduled crawls and custom extraction, is covered in the Screaming Frog advanced usage guide.

A scripted validator for CI

To catch errors before deploy, run extraction and validation in your build pipeline. The pattern is to fetch each URL, pull the JSON-LD blocks, and check required properties for the types you care about. A minimal Node sketch:

// validate.js : node validate.js urls.txt
import fs from 'node:fs';
const urls = fs.readFileSync(process.argv[2],'utf8').split('\n').filter(Boolean);
const required = { Product:['name','offers'], FAQPage:['mainEntity'], Article:['headline'] };
for (const url of urls) {
  const page = await (await fetch(url)).text();
  const re = /<script[^>]*application\/ld\+json[^>]*>(.*?)<\/script>/gs;
  for (const b of page.matchAll(re)) {
    const data = JSON.parse(b[1]);
    const type = data['@type'];
    for (const prop of (required[type] || []))
      if (!data[prop]) console.log(url, type, 'missing', prop);
  }
}

For richer coverage without maintaining rule logic yourself, run each URL through the Rich Results Test, or lint the JSON-LD against the published Schema.org shapes. Run the script on a URL sample in every pull request and on the full set nightly.

Errors worth catching first

A few error classes account for most rich result losses. Missing offers or price on Product, empty acceptedAnswer text on FAQPage, non integer position values in BreadcrumbList, and date fields that are not valid ISO 8601 are the usual culprits. Because these come from templates, fix them once in the template and the whole cohort clears. When your schema is also meant to feed knowledge panels and AI systems, align it with your broader entity strategy described in entity HTML for a machine readable brand. Local businesses have their own frequent gaps, covered in missing local schema.

Make it continuous

Validation is not a one time cleanup. Templates change, plugins update, and a CMS migration can silently break markup across the site. Schedule the full crawl weekly, wire the scripted check into CI so a broken template fails the build, and keep a dashboard that trends error counts by type over time. The goal is that a schema regression is caught in a pull request or a nightly report, never by a drop in rich results weeks later.

Frequently asked questions