Improving Crawling & Indexing with Noindex, Robots.txt & Rel Attributes | Sitebulb
- February 19, 2025
- General
AI Summary
robots.txt blocks crawling, meta robots and X-Robots-Tag noindex block indexing, rel canonical consolidates duplicates, and rel nofollow shapes link signals, so each directive does exactly one job. The most common mistake is combining Disallow with noindex, which cancels out and leaves the page indexed.
- A robots.txt Disallow can still leave a URL only listing in search results.
- noindex must be on a crawlable page, so never also block it in robots.txt.
- Canonical is a consolidation hint, not a removal tool, use noindex to remove.
- noindex in robots.txt is unsupported, and rel next and prev are no longer used.
The linked Sitebulb guide walks through the directives that control how search engines crawl and index a site: robots.txt, the meta robots noindex tag, and the rel attributes such as canonical and nofollow. Each of these does exactly one job, and most technical SEO mistakes come from confusing them. This article lays out what each lever controls, shows the syntax, and highlights the classic conflict that silently keeps pages in the index when you wanted them out.
robots.txt controls crawling, not indexing
robots.txt lives at the root of your domain and tells crawlers which paths they may request. A Disallow rule prevents crawling, but it does not prevent indexing. If other pages link to a blocked URL, Google can still list it in results as a URL only entry, without a snippet, because it was told not to fetch the page. Use robots.txt to save crawl budget on genuinely unimportant sections, not to hide pages from search.
# /robots.txt
User-agent: *
Disallow: /cart/
Disallow: /search/
Allow: /search/popular/
Sitemap: https://seoprocheck.com/sitemap_index.xmlmeta robots and X-Robots-Tag control indexing
To keep a page out of the index, use a noindex directive. For HTML pages, place a meta robots tag in the head. For non HTML files such as PDFs, or when you want to apply the rule at the server level, send the equivalent X-Robots-Tag HTTP header. The critical requirement is that the page must be crawlable for Google to see the directive, so the URL must not be blocked in robots.txt.
In the HTML head:
<meta name="robots" content="noindex,follow">
As an HTTP response header, for PDFs and feeds:
X-Robots-Tag: noindexThe classic conflict that traps pages
The single most common mistake is combining a robots.txt Disallow with a noindex tag on the same URL. It feels thorough, but the two cancel out. Because the page is blocked from crawling, Google never fetches it, so it never reads the noindex, and the URL can linger in the index indefinitely. The correct pattern to remove a page is to allow crawling and serve noindex, wait for Google to recrawl and drop it, and only then, if you wish, add a Disallow. Sequence matters.
rel canonical consolidates, it does not block
The rel canonical link element tells Google which URL is the preferred version among duplicates or near duplicates, so ranking signals consolidate on one address. It is a hint, not a directive, so Google can ignore it if the signals disagree. Canonical is the right tool for parameter variations, printer friendly versions, and syndicated content, but it is the wrong tool for pages you truly want out of the index, which need noindex instead.
rel nofollow, sponsored, and ugc shape link signals
The rel attribute on a link controls how that link passes signals. Since 2019, Google treats nofollow, along with the newer sponsored and ugc values, as hints rather than strict directives. Use sponsored for paid or affiliate links, ugc for user generated content such as comments, and nofollow for links you cannot vouch for. None of these values affect whether the linked page is crawled or indexed through other paths, they only shape the signal carried by that specific link.
What has changed since this guide
Two updates matter most. First, Google confirmed that noindex placed inside robots.txt is unsupported, so the only reliable places for noindex are the meta tag and the X-Robots-Tag header. Second, the pagination attributes rel next and rel prev are no longer used by Google as an indexing signal, so paginated series are handled through normal crawling and internal links rather than those tags. Google also ignores the crawl delay directive in robots.txt, controlling crawl rate instead through server response and its own systems. The core model in the Sitebulb guide, one lever per job, still holds.
Directive quick reference
| Directive | What it does | Use it when |
|---|---|---|
| robots.txt Disallow | Blocks crawling of a path | Saving crawl budget on unimportant sections |
| meta robots noindex | Removes an HTML page from the index | Thin, private, or duplicate HTML pages |
| X-Robots-Tag | Applies noindex at the header level | PDFs, feeds, and non HTML resources |
| rel canonical | Consolidates duplicate URLs | Parameters, syndication, near duplicates |
| rel nofollow, sponsored, ugc | Shapes the signal a link passes | Paid, user generated, or untrusted links |
Related reading
- Noindex explained in the technical SEO glossary
- How index bloat wastes crawl budget
- Crawl budget explained
- Fixing crawled currently not indexed pages
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. robots.txt only blocks crawling. A disallowed URL can still appear in search results as a URL only listing if other pages link to it, because Google knows the address even though it cannot fetch the content. To remove a page from the index you need a noindex directive on a crawlable page.
What happens if I use noindex and Disallow together?
They cancel each other out. Blocking the URL in robots.txt stops Google from crawling it, so it never sees the noindex tag, and the page can stay indexed. The fix is to allow crawling, serve noindex, wait for a recrawl, and only add a Disallow afterward if you still want to save crawl budget.
When should I use a canonical tag instead of noindex?
Use rel canonical when you have duplicate or near duplicate pages and want ranking signals to consolidate on one preferred URL that should still be indexed. Use noindex when a page should not appear in search at all. Canonical is a consolidation hint, noindex is a removal directive.
Is rel nofollow still respected by Google?
Since 2019 Google treats nofollow, sponsored, and ugc as hints rather than strict directives, meaning it may use them for crawling and ranking at its discretion. They remain the correct way to label paid, user generated, or untrusted links, but they no longer guarantee a link is ignored.
How do I noindex a PDF or other non HTML file?
You cannot add a meta tag to a PDF, so use the X-Robots-Tag HTTP response header with a value of noindex. Configure it at the server or CDN level for the file or a matching path, and make sure the file is not also blocked in robots.txt, or Google will never read the header.
Are rel next and rel prev still needed for pagination?
No. Google stopped using rel next and rel prev as an indexing signal, so paginated content is handled through normal crawling and clear internal links between pages. You can keep the tags for other user agents that still use them, but they no longer influence how Google indexes a series.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
