AI Training Data Policy

Element Code: AI-005

What "AI Training Data Policy" means

Large language models are trained on scraped web content. Over the last few years the major AI companies started publishing named crawlers and honoring robots.txt rules for them, so site owners finally have a lever. This check looks for two things: a machine readable directive (in robots.txt or via response headers) telling AI crawlers whether they may use your content, and a human readable policy page stating your position. If both are missing, the check flags it, because silence is a decision you did not consciously make.

Worth being precise here: allowing an AI crawler to train on your content is a different question from allowing it to fetch your page live to answer a user's question (retrieval, sometimes called grounding or RAG). Some crawlers do one, some do both, and the same company often runs separate user agents for each. Blocking the wrong one can cut you out of AI answer surfaces you actually want to appear in.

Why it matters

Three reasons, and they pull in different directions, which is exactly why you need a deliberate stance.

Rights and licensing. If your content is your product, original research, paywalled analysis, photography, proprietary data, you may not want it absorbed into a model for free. Publishers from the New York Times to Reddit have taken formal positions and, in some cases, struck paid licensing deals. Your robots.txt is the first place you assert that.

Visibility. The flip side: if you want to show up in ChatGPT answers, Perplexity citations, or Google AI Overviews, blocking every AI user agent can make you invisible there. Retrieval crawlers are how you get cited. Blindly disallowing everything with "AI" in the name is a common mistake I see that quietly kills a growing traffic channel.

Trust and compliance. A published policy page tells partners, regulators, and contributors what happens to content on your site. For membership sites, communities, and anywhere users submit content, that transparency is increasingly expected and sometimes legally relevant under text and data mining opt out regimes like the EU's.

The crawlers you are actually deciding about

User agent names change and new ones appear, so verify current tokens against each operator's published documentation before you commit rules. Never treat a list like this as frozen.

The decision, mapped

How to detect the current state

• Fetch your robots.txt at https://yourdomain.com/robots.txt and look for named AI user agents. No mention of them means no policy is being enforced.
• Server logs: filter for user agent strings containing GPTBot, ClaudeBot, CCBot, PerplexityBot. This tells you who is actually crawling you and how often, which is the ground truth.
• Screaming Frog: you can set a custom user agent and run a crawl as, say, GPTBot to confirm your robots rules block or allow it the way you intended.
• Check for a policy page: search your own site for terms like "AI", "machine learning", or "training data" in your terms or a dedicated policy URL. If nothing exists, that half of the check is failing too.

How to fix it, step by step

1. Decide your position using the map above. Training, retrieval, or both, and for which crawlers.
2. Encode it in robots.txt. To block a training crawler, add a block such as:
   User-agent: GPTBot
Disallow: /
   Repeat per user agent you want to block. To opt out of Google's model training without hurting Search rankings, use User-agent: Google-Extended with Disallow: /, which does not affect Googlebot.
3. Leave retrieval bots allowed if you want AI answer visibility. Do not blanket disallow everything that looks AI shaped.
4. Publish a short policy page stating your stance in plain language and link it from your footer or terms. This is the human readable half the check wants.
5. Verify and monitor. Refetch robots.txt, crawl as each user agent to confirm, and keep watching logs, since bots that ignore robots.txt need blocking at the firewall or CDN instead.

Do vs Don't

What "good" looks like

A robots.txt with explicit, current rules for the AI crawlers you care about, matching a decision you actually made. Retrieval bots handled separately from training bots so you keep AI answer visibility if you want it. A short published policy page that a human can read and a link to it from your footer. And a habit of rechecking every quarter, because the crawler list and the etiquette around it keep moving.

FAQ