GPTBot

What GPTBot is

GPTBot is OpenAI’s web crawler that collects publicly accessible pages as training data for future GPT models. That’s it — one job. It does not power ChatGPT’s web search (that’s OAI-SearchBot) and it does not fetch pages when a user asks about them (that’s ChatGPT-User), which means your GPTBot decision is purely a policy call about training — and a damn lot of sites got it wrong by treating it as an all-purpose “block ChatGPT” switch.

Why this bot matters historically

When OpenAI announced GPTBot in August 2023, it was the first mainstream, documented opt-out for AI training crawls — a named token, published behavior, published IPs. That single move turned robots.txt from a search-engine courtesy file into the de facto policy layer for AI data collection, and every major lab followed the pattern with tokens of its own. It also created today’s confusion: thousands of sites shipped GPTBot blocks in 2023 as a statement, then OpenAI launched search with a different bot, and those old blocks quietly became irrelevant to the thing people now care about.

How to recognize it

GPTBot identifies itself with a user-agent string along these lines:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot

OpenAI publishes the IP ranges its crawlers use, so a UA claiming to be GPTBot from an unlisted IP is an impostor — scrapers borrow famous bot names constantly. Verify before you blame OpenAI for load, and before you whitelist anything by UA string alone.

What each directive actually does

This table is the whole argument for precision. Every row is a different decision with a different consequence:

The sentence to remember: blocking GPTBot blocks training, not ChatGPT search. Search visibility lives and dies with OAI-SearchBot — the split is explained from the platform side on our ChatGPT glossary page.

How to check it

1. Measure GPTBot’s actual footprint:

   grep -i "gptbot" access.log | wc -l
   grep -i "gptbot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

2. Verify the source IPs of heavy hitters against OpenAI’s published ranges before attributing load to the real bot:

   grep -i "gptbot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

3. Audit robots.txt for all three OpenAI tokens and check what your User-agent: * group implies for anything you didn’t name explicitly.
4. Check the edge layer: 403s and challenge pages for GPTBot in your logs mean the CDN decided for you — a case documented in is your Cloudflare or WAF secretly blocking GPTBot?
5. Re-test after changes: robots.txt is cached; give crawlers a day or two, then confirm the response codes flipped.

Common mistakes

• Blocking GPTBot to “get out of ChatGPT,” then wondering why you still appear — or worse, blocking all OpenAI bots and wondering why AI referrals flatlined. Two different wrong levers, same root cause: not knowing the three tokens. Run the GPTBot blocked check to see your current stance.
• Expecting a GPTBot block to remove existing knowledge. Models already trained on your content keep whatever they learned. Robots.txt is forward-looking only.
• Fuzzy WAF regexes. A pattern like (?i)gpt catches OAI-SearchBot referral infrastructure and ChatGPT-User along with GPTBot. Match exact tokens or suffer silent collateral damage.
• Trusting UA strings without IP checks. Plenty of “GPTBot” traffic is scrapers wearing a costume. Decisions based on fake traffic are fake decisions.
• Copy-pasting a 2023 blocklist and never revisiting. OpenAI added OAI-SearchBot after most of those lists were written; a stale blanket block now costs search citations that didn’t exist when the list was made.

FAQ

Does GPTBot respect robots.txt?

OpenAI documents that it does, and observed behavior matches. What robots.txt can’t do is stop third-party scrapers impersonating the UA — that requires IP verification and edge rules.

If I allow GPTBot, will ChatGPT recommend my business?

Not necessarily. Training inclusion is a lottery ticket, not a listing. The reliable path to being surfaced is ChatGPT search, which depends on OAI-SearchBot access and citable content — see how all the AI bot user agents divide that labor.

Is GPTBot heavy on server resources?

On large sites it can crawl in noticeable bursts. If that’s a problem, rate-limit it at the server or CDN level instead of banning it — you keep the policy choice while capping the load.

Should publishers block GPTBot?

It’s a business decision, not a technical one: blocking preserves licensing leverage over training data while keeping search visibility intact — which is exactly why the GPTBot/OAI-SearchBot split exists. Many major publishers block the former and allow the latter.

How does GPTBot differ from ClaudeBot and PerplexityBot?

Same species, different owners and diets: ClaudeBot feeds Anthropic’s training, PerplexityBot feeds a live answer index rather than training at all. Behavior, tokens and tradeoffs are compared in the AI crawler comparison: GPTBot, ClaudeBot, PerplexityBot.