AI Crawler Checker

Not sure which AI bots your robots.txt lets in?

A rule written for one crawler can quietly apply to others through the catch-all group. This page parses the file into user-agent groups, picks the group each AI bot would follow, and applies the matching rules for the site root, so you see a plain Allowed or Blocked for 27 tokens used by OpenAI, Anthropic, Google, Perplexity, Common Crawl, ByteDance, Amazon, Apple, Meta and others.

Step by step

1. Enter your domain (https:// is added if you leave it off) and press the fetch button. The request goes to /robots.txt on that host via a public proxy, so the address you type is shared with that proxy.
2. Once loaded, the file appears in the paste box and the results fill in, split into three sections by purpose.
3. If the proxy times out or errors, the page offers a direct link to your robots.txt. Open it, copy everything, paste it into the box, and press "Check Pasted robots.txt".
4. Editing a draft? Paste it straight in and run the pasted check. That route stays in your browser, so you can test changes before they go live.

What to watch for

The verdicts describe access to the home path "/" and nothing deeper. A file that disallows GPTBot from /private/ still shows GPTBot as Allowed here, because the root is open. Bots tagged "not named" inherit whatever your * group says, often by accident, so review those rows first. When you block part of the list but not all of it, the summary names up to ten of the bots your file leaves out. Files that mention the same bot in more than one group are handled the way the robots.txt standard asks: every such group is combined into one rule set. An empty file is reported as allowing everything, which matches the standard.

Questions people ask

Does Allowed mean the bot can actually reach my pages?

The checker only reads robots.txt. A firewall, CDN bot setting or server rule can still refuse the request, and none of that is visible from a text file.

Do AI companies have to follow robots.txt?

It is a voluntary protocol. Most major vendors document that their crawlers honor it, though some fetchers that act on a direct user request say they may not. The file cannot force compliance, and this tool cannot verify behavior.

Why are Google-Extended and Applebot-Extended on the list if they are not crawlers?

They are control tokens. Googlebot and Applebot do the fetching, and these names in robots.txt tell each company whether it may use your content for its AI models. Blocking them does not stop normal search crawling.

Further reading: AI Crawler User Agents: The Current Names, and Which Ones Sites Actually Block and Is Your Cloudflare or WAF Secretly Blocking GPTBot?.

The same tool is also published on the static mirror at seoprocheck.github.io, and the rest of the set is on the free tools page.