Complete Guide to Robots.txt Configuration
- February 18, 2024
- Crawlability and Indexation, Technical SEO
robots.txt is a plain text file at your domain root that tells crawlers which paths they may request. It controls crawling, not indexing: a page blocked in robots.txt can still appear in results if other pages link to it, so use meta robots or the X-Robots-Tag header when you actually need a page kept out of the index.
- Directives are User-agent, Disallow, Allow, Sitemap, and the non-Google Crawl-delay.
- robots.txt manages crawl budget; meta robots and X-Robots-Tag manage indexing.
- Keep CSS, JS, and images crawlable so Google can render your pages.
- Always test critical URLs before deploying changes to production.
What is Robots.txt?
The robots.txt file is a plain text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they should or shouldn't request. While it's not a security mechanism (crawlers can ignore it), major search engines like Google, Bing, and others respect these directives. Understanding robots.txt is fundamental to controlling how search engines interact with your site.
Robots.txt Syntax and Directives
The file uses a simple syntax with specific directives. The User-agent directive specifies which crawler the rules apply to (use * for all crawlers). The Disallow directive blocks access to specified paths, while Allow permits access to specific paths within a disallowed directory. The Sitemap directive points crawlers to your XML sitemap location. Each directive must be on its own line, and the file is case-sensitive for paths.
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Specifies the crawler | User-agent: Googlebot |
| Disallow | Blocks crawling of path | Disallow: /admin/ |
| Allow | Permits crawling within disallowed path | Allow: /admin/public/ |
| Sitemap | Points to XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Request delay (not Google) | Crawl-delay: 10 |
Common Robots.txt Patterns
Several patterns appear across well-optimized websites. Blocking parameter URLs (Disallow: /*?*) prevents duplicate content from query strings. Blocking internal search results (Disallow: /search/) keeps thin pages out of the index. Blocking staging or development paths protects unfinished content. Always ensure your CSS, JavaScript, and image files remain crawlable, as Google needs these to render pages properly.
Testing and Validation
Google Search Console provides a robots.txt tester that shows how Google interprets your file and whether specific URLs are blocked. Test critical URLs before deploying changes to production. Common mistakes include blocking entire sites accidentally, using incorrect path syntax, or forgetting that robots.txt doesn't prevent indexing if pages have inbound links. For pages you want completely removed from search, use noindex meta tags instead of robots.txt blocking.
Robots.txt vs Meta Robots vs X-Robots-Tag
Understanding the difference between these directives is crucial. Robots.txt controls crawling, not indexing. Meta robots tags and X-Robots-Tag HTTP headers control indexing. A page blocked by robots.txt can still appear in search results if other pages link to it (Google will show the URL without a snippet). For complete control, use robots.txt to manage crawl budget, and meta robots or X-Robots-Tag to control indexing.
A Worked Example You Can Adapt
Here is a compact, safe starting point for a typical content site. It blocks a private admin area and query-string duplicates, keeps the public assets crawlable, and points crawlers at the sitemap.
User-agent: *
Disallow: /admin/
Disallow: /*?*
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xmlOn WordPress, the platform serves a virtual robots.txt unless you place a physical file in the web root. If you add rules, never blanket block /wp-content/, because that can stop Google fetching CSS and JavaScript and break rendering. Block only what genuinely should not be crawled, and lean on noindex for pages that must stay out of the index.
Mistakes That Quietly Cost You Traffic
| Mistake | What it causes | Fix |
|---|---|---|
| Disallow: / left over from staging | Whole site blocked from crawling | Remove the rule, then request validation in Search Console |
| Blocking /wp-content/ or asset folders | Google cannot render the page correctly | Allow CSS, JS, and image paths |
| Using robots.txt to deindex a linked page | URL still appears, just without a snippet | Allow the crawl and add a noindex directive instead |
| Path case or trailing slash errors | Rule matches nothing or too much | Match the exact path; robots.txt paths are case sensitive |
A Simple Testing Workflow
Before you push a change, run these steps. Open the current file at yourdomain.com/robots.txt and save a copy. Make the edit in staging or a draft. Use the Search Console robots.txt report to confirm how Google parses each rule. Inspect a handful of important URLs to confirm they are still allowed. Only then deploy, and re-inspect in production. If you manage a large site, pair this with crawl budget management and keep your XML sitemap clean so crawlers spend their time on the URLs that matter.
Frequently Asked Questions
Does robots.txt stop a page from being indexed?
No. robots.txt controls crawling, not indexing. A disallowed page can still appear in results if other pages link to it, usually shown as a bare URL without a snippet. To keep a page out of the index, allow the crawl and use a noindex meta robots tag or the X-Robots-Tag header.
Where does the robots.txt file go?
In your domain root, so it resolves at yourdomain.com/robots.txt. It only applies to the host and protocol it sits on, so a separate subdomain needs its own file. Search engines look for it at the root and nowhere else.
Should I block /wp-content/ on WordPress?
No. Blocking it can stop Google fetching the CSS and JavaScript it needs to render your pages, which can hurt how those pages are assessed. Block only specific private paths and keep assets crawlable.
What is the difference between Disallow and noindex?
Disallow in robots.txt asks crawlers not to request a URL, which manages crawl budget. noindex, set via a meta tag or the X-Robots-Tag header, tells search engines not to keep the page in the index. If you disallow a page, crawlers may never see its noindex, so the two should not be combined on the same URL.
Does Google obey Crawl-delay?
No. Google ignores the Crawl-delay directive and manages its crawl rate through Search Console settings and its own systems. Other engines such as Bing do respect Crawl-delay, so it can still be useful for them.
How do I test my robots.txt safely?
Keep a copy of the current file, make changes in staging, and use the Search Console robots.txt report to see how Google parses each rule. Inspect several important URLs to confirm they stay allowed, then deploy and re-check in production.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
