AI Summary
AI training data is the corpus of text, code, and media a model learns from, and if you publish openly, some of your work is probably in one. Training decides what a model knows about your topic by default, and your only levers are which crawlers you admit and which opt out tokens you set.
- Training is not retrieval: training bakes in baseline knowledge, retrieval fetches your live pages at answer time.
- Every opt out (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent) is forward looking, never a recall button.
- Common Crawl (CCBot) is the hole most no training policies forget.
- Blocking retrieval agents kills the citations that drive traffic without stopping training via old archives.
What AI training data is
AI training data is the corpus of text, code, and media a model learns from, and if you publish on the open web, some of your work is probably in one. The stakes for a site owner are concrete: training decides what models "know" about your topic and your brand by default, and your only levers are which crawlers you admit and which opt-outs you set.
Two pipes, one common confusion
Training is not retrieval. A model's training bakes in its baseline knowledge months before you ever query it; retrieval fetches your live pages at answer time and can cite them. Your site can ride either pipe or both, and the robots.txt decisions are separate. (The timing side of this, why trained knowledge lags reality, is the knowledge cutoff problem.)
How content enters training sets, vendor by vendor
| Vendor | How your content gets in | Opt-out mechanism | What opting out does, and doesn't do |
|---|---|---|---|
| OpenAI | GPTBot crawls public pages | User-agent: GPTBotDisallow: / | Keeps future crawls out of future training. Doesn't touch ChatGPT's live browsing/citations, that's OAI-SearchBot and ChatGPT-User. |
| Anthropic | ClaudeBot crawls public pages | User-agent: ClaudeBotDisallow: / | Same forward-only logic. Claude-SearchBot and Claude-User handle retrieval and are governed separately, the ClaudeBot entry maps all three. |
| Content Google already crawls for Search can feed Gemini training | User-agent: Google-ExtendedDisallow: / | Google-Extended is a control token, not a separate crawler. Blocking it opts out of Gemini training without harming your Google Search presence. | |
| Common Crawl | CCBot builds public web archives that many labs train on | User-agent: CCBotDisallow: / | The big one people forget. Block every named AI bot but leave CCBot open, and your content still reaches trainers through the archive. |
| Apple | Applebot crawls for Siri and Spotlight; the same crawl can feed Apple's AI training | User-agent: Applebot-ExtendedDisallow: / | Another control token. Opts out of AI training while Applebot keeps serving Siri/Spotlight features. |
| Meta | meta-externalagent crawls for AI training and related uses | User-agent: meta-externalagentDisallow: / | Forward-only, like the rest. Check Meta's docs for their full agent list, it has churned more than most. |
Notice the pattern: every mechanism is prospective. Nothing in that column removes what a shipped model already learned. Opt-out is a valve on future flow, not a recall button.
A policy worth copying: split by intent, not by vendor
The cleanest robots.txt configurations pick a stance on training and a stance on retrieval, then apply each consistently. A publisher who wants AI citations but not training exposure would disallow GPTBot, ClaudeBot, CCBot, meta-externalagent, Google-Extended, and Applebot-Extended, while explicitly leaving OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, and PerplexityBot alone. A publisher chasing maximum AI presence allows everything and spends the energy on being worth citing instead. Both are coherent. What's incoherent is the accidental middle most sites live in, where the rules reflect whichever blog post the last admin read.
How to check where you stand
- Grep a month of access logs for the tokens above:
grep -icE "GPTBot|ClaudeBot|CCBot|meta-externalagent|Applebot" access.log. This is your actual exposure, not your assumed one. - Read your own robots.txt and list which of the six mechanisms you've set. Most sites have decided by accident, a plugin default or a copied template.
- Search your domain in the Common Crawl index (index.commoncrawl.org). If you're in past dumps, that content is already in circulation regardless of today's rules.
- Check your CDN's bot-management panel. Several providers now block or challenge AI training bots by default, which may already be enforcing a policy you never chose.
- Recheck quarterly. New agents appear, tokens get renamed, and the AI crawler map is the running inventory.
Common mistakes and fixes
- Blocking training bots to get out of current AI answers. Wrong pipe. Answers citing you today come from retrieval agents. Fix: identify which agent does what before writing rules.
- Blocking retrieval agents when you only object to training. This is the expensive inverse, you keep feeding old training dumps via Common Crawl while killing the citations that send traffic. Fix: separate the two intentions explicitly in robots.txt.
- Forgetting CCBot. The most common hole in "we blocked AI" configurations. Fix: if your policy is no-training, CCBot belongs on the list.
- Setting rules on the main domain only. Subdomains have their own robots.txt; your blog on
blog.example.comfollows its own file. Fix: audit every host you serve. - Trusting unofficial signals like a "noai" meta tag. No major vendor commits to honoring it. Fix: use the documented tokens above; they're the only ones with stated support.
FAQ
Can I get my content removed from a model that's already trained?
Practically, no. There's no deletion mechanism for learned weights; your options are blocking future collection and, where it applies, legal or licensing routes. Several copyright suits on exactly this question are still working through the courts, unsettled territory, so don't take robots.txt for a legal remedy.
Is being in training data good or bad for SEO?
Genuinely double-edged. Presence in training data means models can know and mention your brand unprompted; absence means you rely entirely on retrieval for any AI visibility. Sites that want AI-driven discovery usually come out ahead allowing it, the reasoning is laid out in training data implications for GEO.
Do publishers get paid for training data?
Some do, the big licensing deals have gone to large publishers and platforms. For the long tail there's no functioning payment channel today, which is why the allow-or-block decision is mostly about visibility, not revenue.
If I block everything, am I invisible to AI?
Mostly, eventually, and that includes the upside. No citations, no referrals, no presence in AI answers. Blocking is a legitimate stance; just price it honestly.
Which AI bots should I block to opt out of training?
For a no training stance, disallow GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and meta-externalagent. Leave the retrieval agents such as OAI-SearchBot, Claude-SearchBot, ChatGPT-User, and PerplexityBot alone if you still want AI citations.
Does blocking Google-Extended hurt my Google Search rankings?
No. Google-Extended is a control token for Gemini training only, not a separate crawler. Blocking it opts you out of AI training while Googlebot keeps crawling and ranking your pages exactly as before.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
