Is Your Cloudflare or WAF Secretly Blocking GPTBot?

Here is a problem that hides in plain sight: your content renders perfectly, your robots.txt looks fine, and yet ChatGPT, Claude and Perplexity never cite you — because a security layer is quietly turning their crawlers away at the door with a 403. It is more common than you would think.

Why it happens

Cloudflare and most WAFs ship bot-management rules that block "non-browser" traffic. Cloudflare even added one-click "Block AI bots" toggles. These are easy to enable (or leave on by default) without realizing they also block the search and user-fetch bots that decide whether you get cited (see the three jobs every AI bot has).

The receipts

When I tested a stack of polished, well-funded sites with a GPTBot user agent, five of them — Airbnb, Coinbase, Medium, Product Hunt and Udemy — returned an outright 403. Some of that is deliberate; plenty of it is a WAF rule nobody revisited. The point: most teams have no idea which camp they are in.

How to check in 60 seconds

Hit your own site with each bot's user agent and read the status code:

for ua in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot; do
  echo -n "$ua: "; curl -s -o /dev/null -w "%{http_code}n" -A "$ua" https://yoursite.com/
done

A 200 means the bot gets in; a 403 (or a challenge page) means you are blocking it.

How to fix it

1. In Cloudflare, review Security → Bots and any WAF custom rules. If "Block AI Scrapers/Crawlers" is on, decide deliberately which bots to allow.
2. Allowlist the bots you want (the search and user-fetch ones) even if you block training crawlers.
3. Check for verified-bot settings — allow verified AI crawlers explicitly.
4. Re-test with the curl loop until the bots you want return 200.

Blocking AI crawlers is a valid choice — but it should be a decision, not an accident.