Duplicate Content

Duplicate content is the same or substantially similar content reachable at more than one URL — on your own site or across domains. There's no "duplicate content penalty" for normal sites; the real cost is consolidation failure: Google picks one version to index, splits signals across the copies while it decides, and sometimes picks the version you didn't want ranking.

What a duplicate cluster actually looks like

Here's a genuinely ordinary product page cluster from an ecommerce audit — one piece of content, six URLs, all returning 200:

https://shop.example.com/widgets/blue-widget
https://shop.example.com/widgets/blue-widget/
https://shop.example.com/widgets/blue-widget?utm_source=newsletter
https://shop.example.com/widgets/blue-widget?color=blue
https://shop.example.com/sale/blue-widget
http://shop.example.com/widgets/blue-widget

Verify a cluster is real duplication, not just similar URLs, by checking what each serves and declares:

$ curl -s "https://shop.example.com/widgets/blue-widget?utm_source=newsletter" 
  | grep -i 'rel="canonical"'
<link rel="canonical" href="https://shop.example.com/widgets/blue-widget/" />

If every variant canonicalizes to one clean URL, Google usually consolidates fine. When they don't — or when canonicals contradict redirects — you get GSC's "Duplicate without user-selected canonical" and "Duplicate, Google chose different canonical than user" buckets, which are Google telling you it's guessing.

Match the duplication type to the right remedy

This is where audits go wrong. Noindex is almost never the answer to duplication — it removes a page from the index without consolidating anything, so the signals just evaporate. The actual decision:

Rule of thumb: 301 when only one URL should exist, canonical when variants must stay reachable but one should rank, rewrite/merge when the duplication is editorial. Reach for noindex only when a page must exist for users but has no business in search at all — and know you're discarding its signals, not consolidating them.

How to check it on your own site

1. Crawl with Screaming Frog and open Content → Exact Duplicates and Near Duplicates (enable near-duplicate analysis in Config → Content, then run Crawl Analysis). Export the clusters.
2. Read GSC Page Indexing for the two "Duplicate…" reasons. Sample URLs from each and ask: did Google choose the canonical I would have chosen?
3. Test the infrastructure variants by hand: curl -sI the http, non-www, trailing-slash, and uppercase versions of a few key URLs. Anything answering 200 instead of 301 is minting duplicates sitewide.
4. Hunt cannibalization in GSC Performance: filter by a money query and check whether multiple URLs trade impressions for it week over week.
5. Spot-check external duplication by searching an exact-match sentence from your key pages in quotes. Honest caveat: site: and quoted-phrase results are rough discovery tools, not complete inventories — use them for leads, not conclusions. A dedicated duplicate content checker does the sentence-level comparison properly.

Common audit mistakes

• Slapping noindex on duplicates first. The laziest fix and usually the worst: signals to noindexed pages consolidate nowhere, and long-noindexed pages eventually stop being crawled at all. Fix: consolidate with canonical or 301; noindex is a last resort for pages that must exist but never rank.
• Treating canonical as a directive. It's a hint. Google ignores canonicals that contradict stronger signals — internal links, sitemaps, redirects pointing elsewhere. Fix: align every signal on the same preferred URL, then the canonical sticks.
• Scaring clients with penalty talk. Ordinary duplication is a consolidation and crawl-efficiency problem. Deliberate scraped-content spam is a different, rarer beast. Fix: diagnose which one you're looking at before writing the executive summary.
• Fixing tags while internal links keep pointing at variants. If your nav links the parameterized URL, you're voting against your own canonical on every page. Fix: internal links point at canonical URLs, no exceptions.
• Canonicalizing genuinely different pages together. Pointing every location page at one city, or every color variant at a "main" product with different specs, gets the canonical ignored and confuses consolidation further. Fix: canonicals connect true duplicates; different-enough pages get differentiated or merged.

FAQ

Is there a duplicate content penalty?

Not for normal-site duplication — no manual action, no algorithmic demotion for having parameter URLs. Google filters duplicates at serving time and consolidates signals. The exception is scaled, deliberately copied content, which falls under spam policies and is a different diagnosis entirely.

How much overlap counts as duplicate?

There's no published threshold. Crawlers like Screaming Frog default to flagging ~90% textual similarity as near-duplicate, which is a sane working line. Below that, worry less about percentages and more about whether two pages compete for the same query with the same substance.

Canonical tag or 301 — how do I choose?

Ask whether users ever need the duplicate URL to load. If no — old paths, http, trailing-slash chaos — 301 it. If yes — tracking parameters, sort orders, print views — canonical it and keep it reachable. Full reasoning in the duplicate content guide.

Does republishing my articles on Medium or LinkedIn hurt the original?

It can compete with the original, especially if your site is newer or weaker. Republish with a canonical back to your version where the platform allows it, and let the original get indexed first. Without those, accept that the syndicated copy may win the SERP.

Do near-duplicates matter if each page targets a different keyword?

If the pages are 90% identical, Google frequently treats the "different keywords" as one intent and consolidates anyway — and thin unique portions across many URLs feed a bigger problem: index bloat. Either make the differences substantial or merge the set.