AI Crawler User Agents: The Current Names, and Which Ones Sites Actually Block

Why the names go stale

An AI crawler rule is a name in a text file. The name is decided by the company running the crawler, and they change it when the crawler's job changes. OpenAI split one crawler into three. Anthropic renamed its and added two more. Cohere retired a name and published a longer one. Every one of those changes leaves the old name sitting in thousands of robots.txt files, doing nothing.

Nothing warns you. A rule aimed at a name no crawler uses is not an error. It does not show up in Search Console, it does not break a build, and it does not appear in any report. It simply has no effect, quietly, for as long as it stays there.

So I went and counted. Not what the advice blogs say to block, what the biggest sites on the web are actually blocking right now, checked line by line against the vendors' own current pages.

What I measured

The sample is the 184 sites that qualified for the access study I published earlier this month, which is the top of the Tranco list after removing DNS roots, CDN edges and telemetry endpoints. For this pass I fetched one file from each, /robots.txt, and parsed every user agent group in it. 148 of the 184 returned a robots.txt that parsed, and that is the base for every number here.

A name counts as blocked when its own group carries Disallow: / with no matching Allow: /. Names covered only by a wildcard group are not counted, because this is a question about whether site owners named the crawler, not about whether a catch all happens to sweep it up.

Then I fetched each vendor's current crawler documentation the same day and recorded which names appear on it. That comparison is the whole point. One side is what sites block, the other is what exists.

The current names, by vendor

This is the reference half of the page. Every name here appeared on the vendor's own documentation when I checked it on 18 September 2026, and the last column is how many of the 148 sites block it today.

Vendor links go to the page I read on 18 September 2026. Meta publishes its crawler names in its own developer documentation, which refused my request that day, so its two names are listed from the robots.txt evidence rather than from a page I was able to open.

The names worth deleting

These five turn up repeatedly and none of them is on a current vendor page. A rule naming one of them is not harmful, it is simply inert, and it takes up the space where a working rule would go.

Put the two Anthropic rows together and the shape of the problem is obvious. The retired names anthropic-ai and claude-web are blocked by 13 and 10 sites. The two current names for the crawlers that fetch pages to answer a question, claude-searchbot and claude-user, are blocked by 5 and 6. More sites are aiming at the names that no longer exist than at the ones that do.

Training, search and a person clicking a link are three different things

The most useful thing in the data is not a mistake, it is a deliberate choice that a fair number of sites are making. Every big vendor now separates its crawlers by job, and the three jobs deserve different answers.

Training collects pages to train a model. Blocking it costs you nothing in visibility today. Search indexing builds the index the assistant searches when somebody asks a question, so blocking it removes you from answers. User fetch is one page pulled because a person pasted a link or asked about your site specifically, and blocking that one is closest to refusing a visitor.

The numbers show sites acting on exactly that distinction. OpenAI's training crawler gptbot is blocked by 18 sites. Its search crawler oai-searchbot is blocked by 8, and chatgpt-user, the one that fires when a person asks, by 11. Same company, three names, and roughly half as many sites are willing to shut the search one out.

The two most blocked names in the whole sample fit the same logic. bytespider and ccbot are blocked by 21 and 21 sites, more than any assistant crawler, and both are pure collection with no answer surface attached to them. Blocking those two has no visibility cost at all, which is presumably why they lead.

How to check your own file

Three steps, and the whole thing is a reading exercise rather than a technical one.

One, list what your file actually names. Open your robots.txt and write down every user agent line that is not a search engine and not a wildcard. That list is your real AI policy, whatever the intention was.

Two, cross it against the table above. Anything in the retired table is doing nothing and can go. Anything missing from your list that you meant to cover is a gap, and the likely ones are the search and user fetch names, because they are newer than most of the advice in circulation.

Three, decide per job, not per company. Write the rule you mean. If the goal is to stay out of training data while remaining answerable, block the training names and leave the search and user fetch names alone. If the goal is to be absent from assistants entirely, you need the search names too, and you should expect that to cost you.

One caution on all of this, and it is the same one from the access study earlier this month: robots.txt is a published request that well behaved crawlers choose to honour. It is not a lock. If your reason for blocking is contractual or legal rather than a preference, the rule belongs at the edge as well, and the two need to agree. They frequently do not.

Worth reading next to this: how these directives interact with page level rules is in the meta robots and X-Robots-Tag reference, and the separate question of files that describe your site to an assistant rather than restrict it is covered in AGENTS.md vs llms.txt vs llms-full.txt.

Questions people ask about this

Method: robots.txt fetched once per host on 18 September 2026 from a single residential connection in the United States. Sample is the 184 hosts that returned a 200 and HTML to a browser in the September access study; base for every figure is the 148 of those that also served a parseable robots.txt. A name counts as blocked when its own group carries Disallow: / with no matching Allow: /. Vendor documentation read the same day.