Mismatched Nofollow Directives in HTML and Header: How to Fix It
- December 17, 2020
- HTML Structure, Robots Directives
When the meta robots tag and the X-Robots-Tag header disagree about nofollow, Google obeys the most restrictive directive, so the page is treated as nofollow; pick one source of truth and remove the stray directive.
What this check flags
This audit hint fires when a single URL declares robots directives in two places at once, and those declarations contradict each other on the nofollow rule. Typically the page's HTML contains a meta robots tag that says follow (or omits nofollow entirely, which means the same thing), while the server response carries an X-Robots-Tag HTTP header that says nofollow. The reverse is also possible.
Crawling tools such as Sitebulb flag this because it almost always indicates a configuration accident: one of the two directives snuck in somewhere.
The two delivery methods
Meta robots in the HTML
The classic approach is a tag in the document <head>, for example <meta name="robots" content="index, follow">. On WordPress sites this is usually written by the SEO plugin (Rank Math, Yoast and similar). It only works for HTML documents, since there is nowhere to put a meta tag in a PDF or an image.
X-Robots-Tag in the HTTP header
The second approach delivers the same instructions in the HTTP response itself: X-Robots-Tag: nofollow. Google's documentation is explicit that any rule valid in a robots meta tag can also be set as an X-Robots-Tag header. The header method exists precisely because it covers non-HTML resources, and it is typically configured at the server level in Apache's .htaccess, an Nginx config, or a CDN rule. Both methods are fully valid; the problem is only that the two layers disagree.
What happens on conflict
Google does not pick the HTML over the header or vice versa. Per Google's robots meta tag documentation, conflicting directives are combined and the most restrictive rule applies. Nofollow is more restrictive than follow, so nofollow wins. Every time.
The practical consequence is nasty because it is silent. Your page renders normally, the SEO plugin dashboard shows "index, follow", and yet a stray nofollow header means Google will not pass signals through any of the links on that page. Internal linking is one of the few levers a site fully controls, and a forgotten header quietly cuts those wires with no error message anywhere. You only see it if you inspect the raw HTTP response or run a crawler that checks both layers.
✖ Conflict: nofollow wins, links pass nothing
HTML: <meta name="robots" content="index, follow">
Header: X-Robots-Tag: nofollow
✔ Aligned: one source of truth, links counted
HTML: <meta name="robots" content="index, follow">
Header: (no X-Robots-Tag sent)Where mismatches come from
Usually the meta tag is intentional and the header is the accident. Common origins:
Server or CDN configuration. An .htaccess block, Nginx directive, or CDN edge rule adds X-Robots-Tag to a path pattern that matches more URLs than intended, so a rule meant for a staging folder or file type ends up applying to live pages.
Copy-pasted security or hardening snippets. Many "harden your server headers" guides include an X-Robots-Tag line alongside genuinely useful headers like X-Content-Type-Options. Paste the whole block unread and you have nofollowed your site at the server level while your SEO plugin keeps writing follow into the HTML.
Two plugins or two teams. A security plugin sets headers while the SEO plugin sets meta tags, or developers manage server config while SEO manages the CMS, and neither layer knows the other exists.
How to diagnose it
You need to look at both layers for the same URL and compare. The header first:
curl -I https://example.com/affected-page/
HTTP/2 200
content-type: text/html; charset=UTF-8
x-robots-tag: nofollowThen the HTML. Open view-source on the same URL and search the <head> for name="robots":
<meta name="robots" content="index, follow">If one says follow and the other says nofollow, the mismatch is confirmed. Check a handful of affected URLs: a shared folder, file pattern, or template points straight at the config rule responsible.
How to fix it
First decide what the page should be. For a normal public page whose links should count, the correct state is follow, which is also the default when no directive exists at all.
Then establish one source of truth and remove the unintended directive at the layer that owns it. On most CMS sites the SEO plugin should own robots directives, so the fix is to delete the X-Robots-Tag line from .htaccess, the server config, or the CDN rule, then rerun curl -I to confirm the header is gone. If instead the header is the intended behavior (rare, but it happens for non-HTML content), correct the meta tag in the template or plugin so both layers agree, or better, drop the meta tag and let the header speak alone.
Best practice, echoed by Sitebulb's guidance, is to specify robots directives exactly once per URL. Two layers means two places to keep in sync forever.
Common mistakes
Fixing the meta tag instead of the header. Making the HTML say nofollow too clears the warning, but now you have deliberately broken link flow on a page that should pass it. Resolve the conflict toward the directive you actually want.
Editing the origin server but forgetting the CDN. If a CDN or reverse proxy injects the header at the edge, cleaning .htaccess changes nothing. Always re-verify with curl against the live URL after the change.
Adding directives to compensate. Piling on more robots rules to "override" the bad one cannot work, since the most restrictive rule always wins. Subtract the wrong directive; never try to outvote it.
FAQ
A: Neither. Google treats both as equally valid sources of robots rules, combines whatever it finds in both places, and applies the most restrictive result. A nofollow in either layer makes the page nofollow.
A: Not by itself. Nofollow affects whether crawlers follow and pass signals through the page's links, not whether the page stays indexed. But check the full directive string, because the same misconfigured header often carries noindex too, and that one will remove the page.
A: Allowed as long as they agree, but it buys you nothing and doubles the maintenance risk. Use the meta tag for HTML pages, reserve X-Robots-Tag for non-HTML files like PDFs, and never declare the same rules in both places.
Need a full technical audit?
SEO ProCheck runs deep crawls that catch issues like this across your whole site.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
