Noindex is a crawler directive that tells search engines to drop a page from their index, which means it stops appearing in search results entirely. You apply it with a robots meta tag in the page's <head> or with an X-Robots-Tag HTTP header, and unlike a canonical (a hint) it is a command Google actually obeys.
The stakes cut both ways. Used deliberately, noindex is how you keep faceted URLs, internal search results, and thank-you pages from bloating your index. Applied by accident — a staging flag that shipped to production, a plugin checkbox someone ticked — it silently deletes your money pages from Google. I've audited a site where a template-level noindex sat on an entire product category for five months. Nobody noticed because the pages still loaded fine in a browser. The traffic just quietly went to zero.
What noindex actually looks like
Two legitimate delivery mechanisms. In HTML:
<meta name="robots" content="noindex, follow">Or as a response header, which is the only option for PDFs, images, and other non-HTML files:
curl -I https://example.com/whitepaper.pdf
HTTP/2 200
content-type: application/pdf
x-robots-tag: noindexTo set that header on Apache for every PDF in a directory, drop this in .htaccess:
<FilesMatch ".pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>The one non-negotiable rule: Googlebot has to be able to fetch the page to see the directive. Block the URL in robots.txt and the noindex becomes invisible — the URL can then get indexed anyway from external links, showing up in results as a bare link with no snippet. If your goal is removal, allow crawling and serve noindex. I cover the precedence rules in depth in the meta robots and X-Robots-Tag reference.
Noindex vs. X-Robots-Tag vs. robots.txt: what each one actually does
| Mechanism | Stops crawling? | Stops indexing? | Works on non-HTML? | Where it lives | Gotcha |
|---|---|---|---|---|---|
| Meta robots noindex | No — page must be crawled to be seen | Yes, on next recrawl | No, HTML only | Page <head> | Ignored if placed in <body> or blocked by robots.txt |
| X-Robots-Tag header | No | Yes, on next recrawl | Yes — PDFs, images, feeds | Server config / CDN | Easy to apply too broadly with a sloppy regex |
| robots.txt Disallow | Yes | No — URL can still index from external links | Yes | /robots.txt | Actively prevents Google from seeing a noindex on the same URL |
| Combining Disallow + noindex | Yes | Unreliable | — | — | The worst of both: crawl blocked, directive unseen, URL lingers |
That last row is the classic self-own. Pick one tool per URL: robots.txt to save crawl budget on URLs you don't care about, noindex to remove URLs from results.
How to check it on your own site
- Crawl the site with Screaming Frog and open the Directives tab. Filter for "Noindex" and read every URL in that list asking one question: did we mean this?
- Cross-reference against your XML sitemap. A URL that is both in the sitemap and noindexed is sending Google contradictory signals — fix one side.
- In Google Search Console, open Indexing → Pages and look for "Excluded by 'noindex' tag." Sort by URL pattern; a whole template showing up here is the red flag.
- Spot-check individual URLs with
curl -Ifor the header variant, or use the meta robots checker for a quick single-URL read of both mechanisms at once. - For JavaScript-heavy sites, verify the rendered HTML in GSC's URL Inspection tool. A noindex injected (or removed) by client-side JS is a different animal than one in the raw source, and Google acts on the rendered version.
Common mistakes
- Disallowing the URL you're trying to noindex. Covered above, still the number one offender. Fix: remove the robots.txt block, let Google recrawl, and only re-block (if ever) after the URL has dropped out.
- Leaving the staging noindex on at launch. WordPress's "Discourage search engines" checkbox, an environment variable that didn't flip, a hardcoded header on the CDN. Fix: make "verify indexability of five key templates" a launch-day checklist item, every deploy.
- Canonicalizing to a noindexed URL. You're telling Google "the real version of this page is that one over there" and "that one over there shouldn't exist" at the same time. Google resolves the contradiction however it feels like. Full write-up on fixing that contradiction here.
- Reflexively adding nofollow to every noindex. "noindex, nofollow" cuts internal PageRank flow through the page for no benefit in most cases. There are legitimate pairings — here's when the combination is right and when it hurts — but "follow" is the sane default.
- Using noindex where the page should just be gone. If the content is permanently removed and has no replacement, serve a 404 or 410 instead. Noindex keeps the URL alive and recrawled forever; a Gone status lets Google close the file.
FAQ
How long does it take for a noindexed page to leave Google's index?
As long as it takes Google to recrawl the page — days for frequently-crawled URLs, weeks or longer for deep ones. You can speed a single URL up with the URL Inspection tool's recrawl request, or use the Removals tool for a temporary hide while the recrawl catches up.
Does noindex waste crawl budget?
Somewhat. Google still has to fetch the page to keep confirming the directive, though it recrawls long-noindexed URLs progressively less often. If you have millions of worthless URLs, robots.txt is the better crawl-budget tool — see the robots.txt reference for how to scope patterns safely.
Does a noindexed page still pass link equity?
Initially yes, if you leave "follow" in place. But Google has said long-term noindexed pages eventually get treated as nofollow too, since they stop being crawled regularly. Don't build internal-link architecture through pages you've noindexed.
Can a noindexed page still get organic traffic?
It can, and that's diagnostic gold — it means Google hasn't recrawled the page since the tag was added, or the tag isn't being served to Googlebot at all. Here's how to run that check down.
Is noindex the same as removing a page from the sitemap?
No. Leaving a URL out of the sitemap just means you're not advertising it; Google can still find and index it through links. Noindex is the only page-level directive that actually removes it from results.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
