Can AI Crawlers Actually Read Your Site? I Measured 400 of the Biggest

The check a dashboard cannot make

An AI visibility tool asks a model a set of questions and counts how often you turn up in the answers. That measures the answer. It does not measure your site. The tool never has to touch your server to produce a number, so the number arrives looking much the same whether your pages are reachable or not.

Access is a different question and it sits in front of all the others. If the crawler cannot fetch the page, nothing further down the list matters. Not the schema, not the heading structure, not the answer-first paragraph at the top that took an afternoon to get right.

Running that check on one site takes about ten minutes. I wanted to know what it looks like across a few hundred sites at once, so I ran it and wrote down what came back.

What I measured

The sample came from the Tranco top million, daily list dated 4 September 2026. I took the first 400 hostnames after removing DNS roots, CDN edge hosts and telemetry endpoints, which are not websites and have no homepage to serve.

For each of the 400 I fetched /robots.txt once, then the homepage four times, once per user agent, with a pause between requests. Then I applied one mechanical inclusion rule: the browser request had to return HTTP 200 with a text/html content type. That rule drops parked domains, API hosts, redirect-only domains and anything that failed for reasons unrelated to crawler policy, and it keeps me from hand-picking the sample.

184 of the 400 qualified. 126 of those 184 also served a robots.txt that parsed, which is the base for every comparison between what a site declares and what it does.

What came back

Base: 184 sites where a browser got a 200 and HTML. "Not served real content" combines every non-200 answer, every refused or timed out connection, and the handful of 200 responses whose body came back under a tenth the size of the browser version.

The three totals sitting close together hides the interesting part. 71 sites (38.6%) denied at least one of the three crawlers, and only 19 (10.3%) denied all three. Whatever is happening here, it is mostly not a site deciding it wants nothing to do with AI crawlers.

The two boxes worth your attention

Put the declared policy on one axis and the observed behaviour on the other and the picture gets sharper. Of the 126 sites with a robots.txt I could parse:

The green box is the ordinary case and it is the large one. The grey box is the coherent one: 5 sites say no in robots.txt and also say no at the server. That leaves 20 plus 14 sites, 34 of 126, where the file and the server tell different stories.

Blocked without saying so

20 sites never name GPTBot in robots.txt and deny it anyway. This is the case that costs a site something it did not choose to give up, because the policy nobody wrote is the policy nobody reviews.

What the denials look like from outside: cloudflare 4, varnish 4, (not disclosed) 3, nginx 2, cloudfront 2 by Server header, and inside the response bodies, 18 showing captcha, 10 showing access denied, 6 showing akamai ref. Those are the fingerprints of general bot management rather than a deliberate rule about AI crawlers, which fits: a rate limiter or a managed challenge does not know or care what the client claims to be.

This is the finding the IP caveat lands on hardest. Some of these sites are correctly refusing a client that claims to be GPTBot from an address GPTBot does not use, and the real crawler would sail through. Others are matching on the user agent string alone, in which case the real crawler is refused too. From outside you cannot tell which. From inside you can, in about a minute, and the last section says how.

Declared, and then ignored

The other direction is the one people find surprising. 10 of the 15 sites that explicitly disallow GPTBot in robots.txt served my client the full homepage anyway.

Nothing is broken there. That is robots.txt working as designed. It is a published request that well behaved crawlers choose to honour, and the major AI crawlers do honour it. The server is not enforcing anything, because robots.txt was never an enforcement mechanism.

Two practical consequences follow. First, if the reason for your disallow line is that you do not want your pages used for training, that line only stops the crawlers that read it and obey it, and it does nothing at all about anything else sending the same user agent. Second, you cannot confirm your own robots policy by watching your access logs for blocks, because a correctly working disallow produces no blocks on your side. It produces an absence somewhere else.

The three crawlers do not get the same answer

58 of the 184 sites (31.5%) returned a different status code to each of the three. That is roughly a third of the sample, and it is the reason a single spot check is close to worthless.

Observed 5 September 2026 from a residential connection. 999 is LinkedIn's own refusal code rather than a standard one. A site can have a good reason for every row here, and none of these are accusations, they are just what the server returned.

LinkedIn is the cleanest illustration in the set. Its robots.txt disallows ClaudeBot, and ClaudeBot got the page. Its robots.txt does not disallow the client I sent as GPTBot, and that one got refused. The file and the edge are managed by different people with different goals, which is exactly how a site ends up in this state without anyone deciding to put it there.

Two of them changed their mind two hours later

I re-ran the check on a sample of the sites that had denied GPTBot, about two hours after the first pass. Six of the eight repeated the refusal exactly. Two served the homepage normally the second time.

That is a small number and it carries a large instruction: a single request is a sample of one and bot management is adaptive. Rate limits reset. Reputation scores decay. A challenge that fired an hour ago may not fire now. If you are going to make a claim about whether your site is reachable, take the measurement more than once, from more than one address, and write the date on it.

The same discipline is what keeps the rest of AI visibility reporting honest, and it is the argument running through monitoring your AI search visibility: a number without a method and a date attached is not a measurement.

Run this on your own site in ten minutes

Two commands, run from outside your own network, so no office allowlist or VPN rule flatters the result.

# what a browser gets
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36" \
  https://example.com/

# what a client identifying as GPTBot gets
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://example.com/

Repeat for ClaudeBot, PerplexityBot, Google-Extended and any other agent you care about, on your homepage and on three or four templates that actually matter commercially. Two things to read in the output. A status code that is not 200 is the obvious one. The one people miss is a 200 whose byte count is a fraction of the browser response, which is a challenge page or a stripped shell wearing a success code.

Then read your own robots.txt next to those results and see which of the four boxes each agent lands in. If you are unsure what your directives are actually saying once they meet meta robots and response headers, the meta robots and X-Robots-Tag reference lays out how they interact.

The part only you can answer

Everything above is measured from outside, which leaves one question open that outside measurement cannot close: when your edge refuses a client claiming to be GPTBot, is it refusing on the user agent string, or on the address?

The answer is in your own logs and rules, and it takes a few minutes. Look for requests carrying the AI crawler user agents, check whether they were served or challenged, and check the source addresses against the published ranges each vendor documents. If the refusals are landing on requests from those documented ranges, your edge is turning away the real crawler. If they are landing only on requests from everywhere else, your edge is doing its job and the outside view was misleading, which is the good outcome.

Either way you now know, and knowing changes what the visibility number means. A share of voice measured on a site the crawlers cannot reach is not a low score. It is not a score at all.

Once access is settled, the next questions are what to publish and how to keep it citable, which is where the pre-publish quality gate and the difference between AGENTS.md, llms.txt and llms-full.txt come in. Both assume the door already opens.

Questions people ask about this

Method note: sample drawn from the Tranco top million list dated 4 September 2026. Measurements taken 5 September 2026 from a single residential connection in the United States. Base for serving figures is 184 sites, base for policy comparisons is 126. Every figure in this article is a count from that run, and the run is reproducible with the two commands above.