Element Code: TE-012
What this check verifies
People increasingly ask ChatGPT, Claude, Perplexity, and Google's AI features the questions they used to type into a search box. Every one of those systems depends on crawlers or fetchers reaching your pages. This check confirms that the major AI user agents can actually retrieve your content: no robots.txt disallow aimed at them, no CDN or firewall rule serving them a 403, no bot challenge they cannot pass.
Here is the part that trips teams up: a block is invisible from inside the building. Your site looks fine in a browser, looks fine to Googlebot, and meanwhile every AI fetcher has been bouncing off a Cloudflare rule for six months. Nobody notices until someone asks why competitors keep getting cited in ChatGPT answers and you never do.
Know your crawlers, they are not interchangeable
The single most common mistake I see is treating "AI bots" as one thing. There are three different jobs, and blocking each has a completely different consequence.
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect content that may inform future model training. Blocking them is a legitimate licensing stance and costs you little visibility today. Search index crawlers (OAI-SearchBot, PerplexityBot) build the retrieval indexes behind AI search products. Block these and you disappear from those answers. User triggered fetchers (ChatGPT-User, Claude-User) grab a page live because a human asked. Block these and the assistant tells its user your page could not be opened.
| User agent | Operator | What it feeds | Blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Excluded from future OpenAI training crawls |
| OAI-SearchBot | OpenAI | ChatGPT search index | Pages stop surfacing in ChatGPT search results |
| ChatGPT-User | OpenAI | Live fetches for a user | ChatGPT cannot open your page on request |
| ClaudeBot | Anthropic | Model training | Excluded from Anthropic training crawls |
| Claude-User | Anthropic | Live fetches for a user | Claude cannot retrieve your page in a session |
| PerplexityBot | Perplexity | Perplexity's index | Dropped from Perplexity answers and citations |
| Google-Extended | Gemini training controls | No effect on Google Search rankings or crawling | |
| CCBot | Common Crawl | Open corpus many labs train on | Removed from a dataset many models ingest |
Two Google specifics worth knowing. AI Overviews are fed by regular Googlebot, so there is no robots.txt lever that removes you from AI Overviews without also removing you from Google Search; the granular controls are snippet directives like nosnippet and max-snippet. And Bingbot feeds Microsoft Copilot as well as Bing search, so blocking it takes out both.
Where blocks actually happen
Access failures stack in layers, and robots.txt is only the top one. A crawler has to survive every layer to reach your HTML.
The layers, in the order they bite:
- CDN defaults. Cloudflare announced in July 2025 that new domains block AI crawlers by default unless the owner opts in. If you onboarded to Cloudflare since then, assume you are blocking until proven otherwise.
- Bot management and WAF rules. "Block AI scrapers" toggles, JavaScript challenges that a non-browser fetcher can never pass, and managed rulesets that score AI user agents as threats.
- robots.txt. A lot of sites added blanket disallows for GPTBot and friends in 2023 when training opt-outs were the loud topic, then forgot. Those lines now also shape AI search visibility.
- Security plugins and rate limits. WordPress security plugins with aggressive bot lists, and rate limiters that 429 crawlers with bursty fetch patterns.
The tradeoff, honestly
I am not going to tell you that every site must fling the doors open. If your content is your product, opting out of training crawlers is a defensible business decision, and robots.txt is the documented way OpenAI, Anthropic, and Google let you express it. But make it a decision, not an accident. My default recommendation for businesses that publish in order to be found: allow the search index crawlers and the user triggered fetchers at minimum, and decide on training bots separately with whoever owns content licensing.
How to detect blocks
- Read your robots.txt. Search it for GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, CCBot. Blanket
User-agent: *disallows count too. - Grep your server logs. Filter for the AI user agent strings and look at status codes. Lots of 403s or zero hits at all are both answers. Logs are the only place user triggered fetchers show up clearly.
- Test from outside.
curl -A "GPTBot" https://yoursite.com/ -Ifrom a machine outside your network exposes WAF rules that key on the user agent. Caveat: some WAF rules verify crawlers against published IP ranges, so a spoofed UA can be blocked while the real bot passes. Logs beat curl when they disagree. - Check the CDN dashboard. Cloudflare's bot and AI audit screens show exactly which AI crawlers were blocked, challenged, or allowed. Five minutes, definitive.
- Run this check. SEO Pro Check evaluates your robots.txt and live responses against the major AI user agents so the panic-era rules surface before they cost you citations.
How to fix it, step by step
- Decide policy first, in writing: which of the three crawler classes (training, search index, user triggered) you allow. Get the licensing owner to sign off on the training call.
- Edit robots.txt to match. Explicit
User-agentgroups per bot beat clever wildcard logic that nobody can read six months later. - Align the CDN. Turn off default AI blocking for the agents you decided to allow, and whitelist them in any custom WAF rules. This step gets skipped constantly because robots.txt feels like the whole job. It is not.
- Confirm your rate limiting treats verified crawlers gently enough that they do not collect 429s.
- Re-test with curl, then watch logs for a week. You want a steady pattern of 200s for each allowed agent.
DO
- Treat training, search index, and user triggered bots as three separate decisions
- Check the CDN and WAF layer, not just robots.txt
- Verify with server logs, where real fetches and real status codes live
- Keep robots.txt rules explicit and per agent
- Re-audit after any CDN migration or security plugin install
DON'T
- Assume you are open because you never edited robots.txt
- Block Bingbot to keep content out of Copilot: you exit Bing search too
- Expect blocking Google-Extended to change anything about Google Search
- Leave 2023 panic disallows in place without an owner and a reason
- Trust a spoofed curl test over what your logs say the real bots got
FAQ
Does blocking GPTBot remove my content from ChatGPT today?
Will allowing AI crawlers hurt my Google rankings?
Do AI crawlers actually respect robots.txt?
Should I add an llms.txt file?
How do I know a bot claiming to be GPTBot is really OpenAI?
Not sure what the AI crawlers are actually seeing?
An Advanced SEO Audit includes a full crawler access review: robots.txt, CDN and WAF behavior, log analysis per user agent, and a written allow/block policy your whole team can point at.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
