GPTBot Blocked

Element Code: IN-057

What GPTBot actually is

GPTBot is the crawler OpenAI uses to gather web content for training its models. It identifies itself as GPTBot/1.1 and requests pages the way Googlebot does: fetch robots.txt first, check if the path is allowed, then request the page. If your robots.txt has a block rule for User-agent: GPTBot, it will not crawl anything under that block, full stop.

People conflate GPTBot with two other OpenAI bots, and that confusion is the root of most misconfigured robots.txt files. OAI-SearchBot indexes pages so ChatGPT can cite and link to them in search-style answers. ChatGPT-User fires in real time when a live user asks ChatGPT to open a specific page. Three separate user agents, three separate rules. Blocking GPTBot doesn't block the other two, and plenty of site owners think they've opted out of "AI" entirely when they've only blocked training, not retrieval. OpenAI publishes IP ranges for each bot at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json, the way to confirm a hit in your logs is really OpenAI and not a spoofed agent.

Why a site might block it on purpose

This is not automatically a mistake, and I want to say that before this article talks you into unblocking it. Some site owners block GPTBot deliberately for content licensing reasons: they don't want articles or proprietary data folded into a training set without compensation. News publishers pushed back hardest, and several signed direct licensing deals with OpenAI instead of leaving the door open. Bandwidth is a smaller but real concern too, since one more crawler adds server load on top of whatever scraper traffic you're already fighting.

So if this check flagged your site, the right question isn't "how do I fix this," it's "did we mean to do this." If legal made a documented call to keep GPTBot out for licensing reasons, the check is working as intended: it's surfacing a fact, not a bug.

Why blocking it usually hurts AI visibility

Here's the part that costs you something. If GPTBot can't crawl your site, your content isn't part of what future model versions learn from. When someone asks ChatGPT a question your page would have answered well, the model never saw your phrasing or data, because it was never in the training run. That's separate from whether ChatGPT cites you in a live answer, which is OAI-SearchBot and ChatGPT-User's job, but it still matters for GEO.

This is different from classic SEO. Google and Bing rankings are untouched by a GPTBot block, since Googlebot and Bingbot are separate crawlers with separate rules. You won't see a ranking drop in Search Console from this. What you lose is upside in a channel referral data shows is growing: traffic from ChatGPT answers, and presence in whatever corpus shapes the next model generation.

How to detect the block

A few ways to confirm what's happening, roughly in order of speed:

1. Check robots.txt directly. Visit yourdomain.com/robots.txt and look for User-agent: GPTBot followed by Disallow: /, or a wildcard User-agent: * block catching GPTBot with everything else.
2. Screaming Frog's robots.txt checker. Test specific user agents against your rules under Configuration > robots.txt > Custom, simulating exactly what GPTBot sees.
3. Server logs. Filter access logs for GPTBot and check response codes. A wall of 403s, or zero hits despite robots.txt allowing it, points to a WAF blocking it at a layer robots.txt can't control.
4. WAF and CDN dashboards. Cloudflare has a "Block AI Bots" toggle under Security > Bots. These sit above robots.txt entirely: a site can look wide open in the text file and still 403 every GPTBot request because someone flipped a switch in the CDN panel.

The combination that trips people up most: robots.txt looks fine, but the WAF silently 403s GPTBot's IP ranges. Check both layers.

How to fix it, step by step

1. Open robots.txt, usually at the site root or through your WordPress SEO plugin's editor.
2. Find any User-agent: GPTBot block with Disallow: / and remove it, or change it to Allow: /.
3. If GPTBot is swept up by a wildcard User-agent: * disallow, add a specific User-agent: GPTBot section above it with Allow: /, since specific rules beat wildcards.
4. Check your CDN or WAF for an "AI bot" blocking toggle. Disable it, or confirm it's intentionally on if you're keeping the block for licensing reasons.
5. Verify by refetching robots.txt and cross-checking against OpenAI's crawler docs for current syntax.
6. Watch server logs for GPTBot hits returning 200s instead of 403s. OpenAI doesn't recrawl instantly, so don't panic if nothing shows up same-day.

What good looks like

Good isn't "always allow every bot." Good is a robots.txt and WAF configuration that reflects an actual decision someone made and can explain. If your legal team flagged content licensing risk, that's a defensible, intentional position, and this check is just confirming it matches your intent. If a hosting platform shipped "block AI bots" as the default and nobody looked at it, that's the accident this check exists to catch. Either way, the fix isn't really technical, it's a five-minute conversation with whoever owns the decision, followed by a robots.txt edit that matches what was actually decided.

GPTBot Requests Page User-agent: GPTBot/1.1 Checks robots.txt and WAF / firewall rules Allowed Disallowed Page is crawled Content eligible for training data 403 or no fetch Never enters training data Possible upside Shapes future model answers (separate from live citations) Confirmed intentional? Licensing concern: leave as is Accidental default: fix robots.txt

The main AI crawlers you'll see in your logs

Google-Extended is the closest analog to GPTBot: a separate, opt-out-able signal controlling AI training without touching classic Google Search rankings. If you already have a policy for Google-Extended, apply the same logic here instead of treating GPTBot as a one-off decision.

FAQ