Technical Definition
Robots.txt is a text file at your site's root (example.com/robots.txt) that provides crawling instructions to search engine bots. It uses directives like Disallow (block paths), Allow (permit within blocked paths), and Sitemap (reference XML sitemaps). Important: robots.txt blocks crawling, not indexing. Blocked URLs can still be indexed if linked from other sites.
Simple Explanation (ELI13)
Robots.txt is a file that tells search engines which parts of your website they can and can't visit. It's like a 'staff only' sign. But here's the tricky part: blocking Google from visiting a page doesn't stop it from knowing the page exists. If other websites link to that page, Google might still list it in search results.
Related Terms
Crawling, Googlebot, Noindex, User-Agent
Learn More
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
Recent Posts
- No Social Schema December 7, 2025
- Missing Social Profile Links December 7, 2025
- Social Image Wrong Size December 7, 2025