The Vary Header and Caching for SEO: Avoiding Cloaking and Stale Pages

The Vary HTTP header tells caches when a single URL can legitimately return different responses depending on the request. Get it right and your CDN serves the correct variant to humans and Googlebot alike. Get it wrong and you either poison your cache, hand crawlers a stale or mobile-only page, or trip Google's cloaking detection. This is one of the quietest sources of indexing problems because nothing looks broken in a browser.

What the Vary header actually does

When a cache stores a response, it keys that entry by URL. The Vary header adds request headers to that cache key, so the cache knows the response is conditional on them. The classic example:

Vary: Accept-Encoding

This says "the gzip'd version and the uncompressed version are different responses for the same URL, don't serve a compressed body to a client that can't decode it." Caches respect this at every layer: the browser cache, your CDN, and intermediate proxies.

The SEO relevance starts the moment your server changes content (not just encoding) based on a request header. If you serve different HTML by device, language, or cookie and you don't declare it in Vary, a shared cache will happily serve the first variant it stored to everyone who follows, including Googlebot.

Vary and Googlebot: the dynamic serving case

Google's official guidance for "dynamic serving", serving different HTML on the same URL to mobile vs. desktop user agents, is to send:

Vary: User-Agent

This is a signal to Google that the URL's content depends on the requesting user agent, prompting it to crawl with both its smartphone and (legacy) desktop agents. With mobile-first indexing now universal, the mobile Googlebot is the one that matters: it is what indexes your site. So the failure mode that hurts you is serving desktop-only or degraded HTML to a mobile user agent.

A critical caveat: Vary: User-Agent is a hint, not a cloaking exemption. If the content you serve to Googlebot differs materially from what a real user on the same device sees, that's cloaking regardless of the header. Vary manages caches; it does not grant permission to show search engines something different.

The cloaking trap built into caching

The most common way teams accidentally cloak has nothing to do with intent. It's a caching misconfiguration:

• Geo or A/B logic with no Vary. Your edge serves different HTML by country or experiment bucket but doesn't vary the cache key. The CDN caches whatever variant Googlebot's crawl produced and serves it to everyone, or caches a user's variant and serves it to Googlebot.
• Cookie-driven personalization cached publicly. A logged-in or personalized response gets stored in a shared cache because the response was marked public. Now Googlebot indexes a personalized page, or users get someone else's session view.
• Bot-specific rendering paths. An SSR layer that detects Googlebot's user agent and serves a prerendered snapshot. If that snapshot drifts from the user experience, you've built a cloaking machine, and it's invisible until a manual action or ranking collapse surfaces it.

The tell is almost always that the page looks fine when you load it, because your request headers happen to match the cached variant. The discrepancy only exists for a different header combination, which is exactly what a crawler sends.

Why Vary: User-Agent destroys cache efficiency

Here's the tension nobody warns you about. Varying on User-Agent is technically correct for dynamic serving, but the User-Agent string has near-infinite cardinality, every browser version, OS build, and bot produces a distinct value. A CDN that honors Vary: User-Agent literally will store a separate cache entry per unique string, shredding your hit rate to almost nothing.

In practice this is why the industry moved away from dynamic serving toward responsive design (one HTML payload, CSS handles layout) and toward edge logic that normalizes the user agent into a small number of buckets before it becomes part of the cache key. If you must serve device-specific HTML behind a CDN, your options are:

1. Normalize User-Agent at the edge into mobile / desktop / bot and vary the cache key on that derived value, not the raw header.
2. Split content onto separate URLs (e.g., an m. subdomain) with proper bidirectional annotations, though this is legacy and discouraged.
3. Go responsive and stop varying content by user agent entirely. This is the recommendation for almost every site.

Vary and Cache-Control: read them together

Neither header is safe in isolation. Vary defines the cache key; Cache-Control defines whether and how long anything is stored. The combinations that bite SEO:

• Personalized HTML must not be publicly cacheable. Use Cache-Control: private, no-store (or private) for any response whose body depends on a logged-in cookie. private permits the browser cache but forbids shared CDN/proxy caches from storing it.
• If you vary on a cookie, you almost never want public caching. Vary: Cookie on a publicly cacheable response is a footgun, cookie cardinality is unbounded and you'll leak variants. Strip cookies at the edge for cacheable pages instead.
• Stale variants outlive their welcome. A long max-age combined with content that changes by header means an old variant can be served to a crawler for hours after the page changed. If a page is meant to be fresh for crawlers, keep its TTL short or revalidate.

How to diagnose it

You can't trust a normal browser load, it sends your headers, not a crawler's. Test the way a cache and a crawler actually behave:

• Fetch with Googlebot's mobile user agent and compare the rendered HTML to a real mobile browser load. Use the URL Inspection tool's "View crawled page" in Search Console as ground truth for what Google actually received.
• Curl the URL with different User-Agent and Accept-Language values and diff the bodies. Any meaningful difference that isn't declared in Vary is a latent cloaking/caching bug.
• Inspect response headers at the edge. Look for the CDN's cache-status header (e.g., x-cache: HIT) alongside Vary and Cache-Control. A HIT on a personalized page is a red flag.
• Check whether your CDN honors Vary at all. Some CDNs ignore most Vary headers by default (respecting only Accept-Encoding) and require explicit configuration. If yours ignores Vary: User-Agent, your dynamic serving is silently broken at the edge.

Common mistakes

• Assuming Vary: User-Agent makes dynamic serving "safe." It manages caches; it does not authorize content differences that constitute cloaking.
• Sending Vary: *. This makes every response uncacheable by design. Occasionally intentional, usually an accident that crushes performance.
• Letting a CDN cache cookie- or geo-personalized HTML as public. The single most frequent cause of "Googlebot indexed the wrong page" reports.
• Forgetting Vary: Accept-Encoding. Rare now, but omitting it can deliver gzip'd bytes to a client expecting plain text, a hard failure, not a soft one.
• Trusting your own browser as the test. Your headers match the cached variant; the bug lives in the header combinations you never send.

The defensible setup

For the overwhelming majority of sites: go responsive, serve one HTML payload per URL, mark public pages public with a sensible TTL, mark anything personalized private and strip cookies before caching. Reserve Vary for Accept-Encoding and let device differences live in CSS. If business requirements genuinely force header-dependent HTML, normalize that header into a tiny set of buckets at the edge and vary the cache key on the derived value, never the raw, high-cardinality string. Do that, and caching speeds your site up without ever showing a crawler something a user wouldn't see.