Robots.txt Generator

Need a clean robots.txt without memorizing the syntax?

A robots.txt file is short, but one wrong character can block far more than you meant to. This generator gives you a safe baseline: a catch-all group for every crawler with only the exclusions you list, optional full blocks for five named AI crawlers, and a pointer to your sitemap. It produces the text on the page and offers it as a download named robots.txt. It does not fetch your current file or test any URL against the rules.

Step by step

1. Put the complete, absolute address of your sitemap (for example https://example.com/sitemap.xml) in the sitemap box.
2. Decide on AI crawlers. Each ticked box adds its own group that shuts that bot out of the whole site. The ClaudeBot box covers that one name; Anthropic has retired the older anthropic-ai and Claude-Web tokens.
3. Type any private or low-value sections, such as /cart/ or /wp-admin/, into the paths box, one per line.
4. Press "Generate robots.txt", read every line, and then copy it or download it and upload it so it loads at yourdomain.com/robots.txt.

What to watch for

The paths you type are copied into the file exactly, so a missing leading slash or a misspelled folder name slips through. With no paths at all you get an empty Disallow line, which means nothing is blocked. Also remember that a crawler obeys only the most specific group that names it: a bot with its own group ignores the rules under the asterisk. That is harmless here, since the named groups block everything, but it matters if you later add other groups by hand.

Questions people ask

Will blocking a page here remove it from Google?

No. robots.txt controls crawling, not indexing. If other pages link to a blocked URL, Google can still list it, just without reading its content. To keep a page out of results, allow crawling and use a noindex directive, or put it behind a login.

Does ticking GPTBot keep my site out of ChatGPT?

It stops OpenAI's training crawler. OpenAI documents other agents, including OAI-SearchBot for search features and ChatGPT-User for pages a person asks ChatGPT to open, and treats them separately from GPTBot. Other vendors split their bots in a similar way, so review their current documentation.

Will Google-Extended hurt my Google rankings?

Google states that the token has no effect on inclusion or ranking in Search. It only governs whether your content can be used for Gemini. Googlebot keeps following the catch-all group.

Further reading: Which AI Bots Are You Actually Blocking? (GPTBot, ClaudeBot, Perplexity & More) and Robots.txt FAQ: Complete Guide to Crawler Directives.

The same tool is also published on the static mirror at seoprocheck.github.io, and the rest of the set is on the free tools page.