What ChatGPT is (for site owners)
ChatGPT is OpenAI’s assistant, and since it gained web search it is two distribution channels in one: a model that may already “know” your brand from training data, and a search product that crawls, indexes and cites live pages. Treat it as a place your content can be found, not just a writing tool — because your robots.txt decisions directly decide whether ChatGPT can ever send you a visitor.
How ChatGPT gets web content: three different bots
OpenAI runs three crawlers with separate robots.txt tokens, and conflating them is the single most common screw-up in AI-bot policy. Each one answers to a different product decision:
| Crawler (robots.txt token) | What it does | Block it and you get | Allow it and you get |
|---|---|---|---|
GPTBot | Collects public pages as training data for future OpenAI models | Opt-out from future training crawls. No effect on ChatGPT search results or live browsing | Your content eligible for future model training; possible “model knowledge” of your brand |
OAI-SearchBot | Builds and refreshes the search index behind ChatGPT search | You vanish from ChatGPT’s cited, linked web results — the surface that actually refers traffic | Eligibility for citations with links, and referral clicks |
ChatGPT-User | Fetches a specific URL live when a user asks about it in a conversation | Users can’t pull your pages into a chat; shared links of yours can’t be read or summarized | Your live page readable in-conversation, at the exact moment someone cares about it |
The trap in plain terms: blocking GPTBot is a statement about training. It does not remove you from ChatGPT search — that’s OAI-SearchBot’s index — and it doesn’t stop a user-initiated fetch, which arrives as ChatGPT-User. Full detail on that distinction lives on our GPTBot glossary page.
Example: a deliberate, split policy
A publisher that wants ChatGPT referrals but not training inclusion writes exactly this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /That combination is coherent: no training, full search visibility, full in-chat readability. Most sites that “blocked ChatGPT” in 2023 actually wanted this and shipped a blanket ban instead.
How visibility in ChatGPT is earned
ChatGPT doesn’t search on every prompt. It reaches for the web when the question smells time-sensitive or factual — prices, news, product comparisons, “best X for Y” — or when the user toggles search on explicitly. That routing decision matters for site owners: informational evergreen queries often get answered from model memory with zero citations, while commercial-investigation queries trigger retrieval and hand out source links. The queries most likely to cite you are, conveniently, the ones closest to money.
When search mode does trigger, ChatGPT queries OpenAI’s index, reads candidate pages, and composes an answer with inline source links. Cited links typically carry utm_source=chatgpt.com, which makes this the easiest AI channel to measure in analytics. Pages win those slots the boring way: crawlable raw HTML, a direct answer near the top, verifiable specifics, and a title that works as a link label. The tactical layer is covered in ChatGPT SEO: optimizing for citations.
How to check it
- Grep for all three tokens separately — the split tells you which pipeline is active on your site:
grep -ioE "gptbot|oai-searchbot|chatgpt-user" access.log | sort | uniq -c - Read your robots.txt top to bottom. Confirm which of the three tokens each rule actually matches, including inheritance from
User-agent: *. - Test the WAF path: if logs show 403s or challenge responses for these UAs, your edge layer overrides robots.txt.
- Segment analytics for
utm_source=chatgpt.comand referrerchatgpt.com; annotate the date of any robots change so you can see cause and effect. - Prompt-test with search enabled: ask ChatGPT your key questions in fresh sessions and log whether you appear as a cited source.
Common mistakes
- Blocking GPTBot and expecting to leave ChatGPT search. Wrong lever. Search eligibility is OAI-SearchBot; check yours with the AI bot user-agents explainer.
- Blanket-banning all AI agents, then reporting “ChatGPT sends us nothing.” Of course it doesn’t. You told it not to.
- Regex UA blocking at the CDN. A pattern written for
GPTcatches OAI-SearchBot’s fetches and ChatGPT-User too. Match exact tokens. - JS-only content. OpenAI’s fetchers read server-delivered HTML; a page that is empty before hydration is empty to ChatGPT.
- Assuming a GPTBot hit means citations are coming. Training crawls produce no links and no traffic. Only the search index does.
FAQ
Does ChatGPT use Google or Bing results?
ChatGPT search runs on OpenAI’s own crawling and index (built by OAI-SearchBot), supplemented by partnerships. It is not a skin over Google, which is why your Google rankings and ChatGPT citations can diverge sharply.
How do I see ChatGPT traffic in GA4?
Filter session source for chatgpt.com. Cited links usually append utm_source=chatgpt.com, so a simple source filter captures most of it.
Can content blocked from GPTBot still be cited in ChatGPT?
Yes. Citations come from the search index and live fetches, which use OAI-SearchBot and ChatGPT-User respectively. GPTBot only governs training collection.
Should an e-commerce or lead-gen site allow all three bots?
Usually yes — there’s no revenue in model training either way for most businesses, and search plus live-fetch visibility is pure upside. Media companies licensing content are the main case for a GPTBot-only block.
How does OpenAI’s crawling compare to Anthropic’s and Perplexity’s?
Same three-way split of jobs, different tokens and behaviors — mapped side by side in the AI crawler comparison: GPTBot, ClaudeBot, PerplexityBot.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
- AI Assistant, GEO, LLM, OpenAI
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
