Robots.txt Mistakes That Quietly Tank Your SEO

Robots.txt is a few lines of text with outsized power: get it wrong and you can wall off your whole site or quietly block the crawlers you most want. Here are the mistakes that actually cause damage.

The mistakes that hurt

• Disallowing CSS/JS. Blocking assets Google needs to render the page degrades how it understands and ranks you. Let rendering resources through.
• Using robots.txt to deindex. Disallow stops crawling, not indexing — a blocked URL can still appear in results (without a snippet). To remove a page, use noindex and let it be crawled.
• A stray sitewide block. A leftover Disallow: / from staging is the classic catastrophe. Always check after launches.
• Stacked, conflicting groups. Multiple User-agent: * blocks confuse maintenance and invite mistakes. Keep one clean group per agent.
• Blocking the AI crawlers you wanted. A blanket rule can exclude the search and user-fetch bots that drive AI citations. (See The Forgotten HTML.)
• Assuming it is enforcement. Robots.txt is a request. Misbehaving bots ignore it; for real enforcement use a firewall.

How to get it right

1. Default to open. Only disallow what genuinely should not be crawled (internal search, cart, admin).
2. Keep it minimal and readable — one group per user agent, comments for intent.
3. Reference your sitemap with a Sitemap: line.
4. Test changes in GSC's robots.txt report before and after.
5. Re-check after every launch or migration.

Related: Crawl budget · What AI crawlers really see