Mismatched Nofollow Directives in HTML and Header: How to Fix It
- December 17, 2020
- HTML Structure, Robots Directives
What this check flags
This check fires when a page gives two different answers about nofollow: one value in the HTML <meta name="robots"> tag and a conflicting value in the X-Robots-Tag HTTP response header. It matters because search engines combine the two by taking the most restrictive instruction — so a stray nofollow in a header you forgot about can quietly kill link equity flow from a page you meant to have followed, and you'll never see it in the HTML.
Both the meta tag and the header are legitimate ways to deliver robots directives. The problem isn't having both — it's having them disagree, because then the source of truth depends on rules most people don't have memorized.
A real failing example, and the fix
Say your page's HTML looks intentional and permissive:
<!-- In the <head> -->
<meta name="robots" content="index, follow">But the response header, set months ago in an Nginx/Apache config or a CDN rule and long forgotten, says otherwise. Check it with a HEAD request:
$ curl -sI https://example.com/resources/guide/
HTTP/2 200
content-type: text/html; charset=UTF-8
x-robots-tag: nofollow
cache-control: max-age=3600The HTML says follow; the header says nofollow. Google resolves the conflict by combining directives and honoring the more restrictive one — so this page is treated as nofollow. Every link on it stops passing signals, and your <meta> tag was a red herring the whole time.
The fix is to make the two agree. If you actually want links followed, remove the header directive (or set it to match). At the server level:
# Apache (.htaccess) — REMOVE or correct a stale rule like this:
# Header set X-Robots-Tag "nofollow"
# Nginx — drop or fix the offending add_header line:
# add_header X-Robots-Tag "nofollow";Then re-check with curl -sI until the header no longer contradicts the meta tag. If you genuinely want nofollow, pick one place to say it and make the other match — don't leave them fighting.
Which one wins? The rules, plainly
| Meta robots (HTML) | X-Robots-Tag (header) | What the crawler actually does |
|---|---|---|
follow | nofollow | nofollow — most restrictive wins |
nofollow | follow | nofollow — most restrictive wins |
index, follow | (header absent) | follow — nothing to conflict |
| (meta absent) | nofollow | nofollow — header applies |
noindex | nofollow | noindex + nofollow — directives combine, not override |
Two things people get wrong here. First, it is not "the header always beats the meta tag" — Google combines both and takes the strictest. A permissive header does not rescue a restrictive meta tag, or vice versa. Second, directives stack: a noindex from one source and a nofollow from the other give you a page that's both, even though neither source said both.
One thing that genuinely does override: a directive only counts if the crawler can read it. If robots.txt blocks the URL, the crawler never fetches the page, never sees the header or the meta tag, and the directive is ignored entirely. Don't block a page you're trying to give robots instructions to.
How to detect it
curl -sI <url>— read the rawx-robots-tagheader, then view source for<meta name="robots">. If they disagree, you've reproduced it in two commands.- Chrome DevTools → Network tab — click the document request and read Response Headers for
X-Robots-Tagalongside the rendered meta tag in Elements. - Screaming Frog — it reports both the meta robots value and the X-Robots-Tag per URL in the Directives tab, so mismatches across a whole site jump out.
- Google Search Console URL Inspection — shows how Google actually resolved the directives for a live URL, which is the final word on which value won.
How to fix it
- Decide the intended state for the page: follow or nofollow. There's no fix without knowing what you meant.
- Find the source of the wrong value. Header directives live in server config (
.htaccess, Nginxadd_header), a CDN/edge rule, or an app-level middleware. Meta directives come from the template, an SEO plugin, or a theme setting. - Make one source authoritative. Pick the meta tag or the header for robots directives and set the other to match — ideally, don't emit both for the same directive at all.
- Watch for wildcard config. A blanket
X-Robots-Tag: nofollowapplied to a whole directory or file type is the classic cause — scope it, or remove it. - Re-verify with
curl -sIand re-inspect in Search Console until both agree.
FAQ
Does X-Robots-Tag always override the meta robots tag?
No — that's the myth this whole check exists to correct. Google reads both and applies the most restrictive combination. A followed header will not undo a nofollow meta tag; the page ends up nofollow either way. Treat them as additive, not as a priority order.
Is a nofollow mismatch as serious as a noindex one?
nofollow is usually less catastrophic than a rogue noindex — it doesn't drop the page from the index, it stops the page from passing signals through its links. But on a hub, category, or resource page whose job is to distribute link equity, an accidental sitewide nofollow is a real, quiet drain. Fix it with the same urgency.
Where does nofollow in the header even come from if it's not in my HTML?
Almost always a server or CDN rule someone added for a specific reason (staging, a file type, a legacy directory) that grew too broad or outlived its purpose. Search your .htaccess, Nginx configs, and edge/CDN rules for X-Robots-Tag. It's rarely the CMS.
Should I just delete both and let the page default?
If you want the page indexed and its links followed, yes — the default when no robots directive is present is exactly index, follow. Emitting explicit directives only creates surface area for them to disagree. Say nothing unless you mean to restrict something.
Related checks
- Robots Meta Tag vs X-Robots-Tag HTTP Header: When to Use Each — the full picture on choosing between the two.
- Meta Robots and X-Robots-Tag: The Complete Reference — every value and how they combine.
- Noindex and Nofollow Together — when both directives on one page are right, and when they hurt.
- Meta Robots Outside Head — a meta directive placed where crawlers won't honor it.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
