/blog/blog/ or /shoes/mens/shoes/mens/. It almost always signals a broken relative link or a CMS routing bug, it spawns duplicate crawlable URLs, and it wastes crawl budget on pages that should not exist.What a repetitive path actually is
Look at the URL and you will spot the same folder name appearing more than once in the path: example.com/blog/blog/my-post/, example.com/products/shoes/products/shoes/, or the nastier recursive kind, example.com/a/a/a/a/ that keeps growing every time a crawler follows a link. The segment repeats. That is the whole tell.
Here is why it happens. Somewhere on the page there is a link written as a relative reference, something like <a href="blog/my-post/"> instead of a root-relative <a href="/blog/my-post/"> or an absolute URL. When the browser or the crawler is already sitting on /blog/, a relative blog/my-post/ resolves against the current directory and produces /blog/blog/my-post/. Now that new page also serves the same relative link, so the crawler follows it again and gets /blog/blog/blog/my-post/. That is the recursive spider trap, and I have watched it generate tens of thousands of junk URLs from a single bad template.
How the loop forms
Why it hurts your SEO
Each repeated-path variant is a separate URL that a crawler can request. Google does not know these are garbage, it just sees addresses that return content, so it crawls them. That drains crawl budget you would rather spend on your real, revenue-driving pages. On a large site the effect is real: Googlebot spends its allotted requests hammering /blog/blog/blog/ variants instead of discovering your new product pages.
There is a duplication problem too. If /blog/blog/my-post/ returns the same content as /blog/my-post/, you now have duplicate pages competing with each other. Even when canonicals eventually sort it out, you have handed Google an ambiguity it has to resolve, and internal PageRank leaks into dead-end variants. Worst case, the recursive version generates an effectively infinite URL space that a crawler treats as a spider trap, and crawling of the healthy site slows to a crawl.
How to detect it
This is one of the easier issues to catch once you know the pattern.
- Screaming Frog: crawl the site, then use the search or a custom regex filter on the URL column for a repeated segment, for example
/([^/]+)/\1/. Frog will surface every URL where a folder name repeats back to back. Its built-in reports also flag pages many clicks deep, which is where these traps hide. - Sitebulb: its crawl-depth and URL structure hints will light up when it sees the same directory duplicated, and it visualizes the runaway depth clearly.
- Google Search Console: check Crawl Stats for a spike in requests to a folder, and the Pages report for a flood of Crawled, not indexed or Duplicate URLs. Server logs tell the same story: filter access logs for a path segment that appears twice in one line.
- A quick site query:
site:example.com inurl:blog/blogin Google will show whether the junk variants are already indexed.
How to fix it, step by step
- Find the source link. Open one bad URL and view source. Look for the relative
hreforsrcthat lacks a leading slash. That single reference in a header, footer, nav, or a template partial is almost always the culprit. - Switch to root-relative or absolute URLs. Change
href="blog/my-post/"tohref="/blog/my-post/". This resolves the same way no matter which directory the page lives in, which kills the loop at its root. - Set a base href only if you understand it. A correct
<base>tag can help, but a wrong one makes things worse, so I usually just fix the links directly. - 301 redirect the existing junk. For repetitive variants already indexed or linked, add a redirect rule that strips the duplicate segment and sends the request to the canonical single-segment URL. A regex rewrite in your server config handles the whole class at once.
- Set self-referencing canonicals on the real pages so any straggler variant points home.
- Re-crawl and confirm. Run Screaming Frog again with the same regex. Zero matches means you are done. Then watch Crawl Stats settle over the next couple of weeks.
Repetitive path vs healthy path
| URL | Verdict | Why |
|---|---|---|
/blog/my-post/ | Healthy | Each segment appears once, clean hierarchy |
/blog/blog/my-post/ | Broken | Duplicate of the real page, wastes crawl budget |
/products/shoes/products/shoes/ | Broken | Repeated pair from a bad relative link |
/a/a/a/a/a/ | Spider trap | Recursive, effectively infinite URL space |
/news/news-roundup/ | Healthy | Similar words, not a repeated segment, no problem |
That last row matters. A repeated word inside a single slug, like news-roundup, is fine. The check is about repeated path segments separated by slashes, not repeated letters.
- Use root-relative or absolute URLs in templates
- Regex-test every crawl for repeated segments
- 301 the junk variants to the clean canonical
- Add self-referencing canonical tags
- Re-crawl to confirm zero matches remain
- Ship relative links without a leading slash
- Assume canonicals alone fix crawl waste
- Block the folder in robots.txt and call it done
- Ignore a growing depth count in your crawler
- Confuse a repeated word with a repeated segment
What good looks like
A clean site has one URL per page, every path segment appears once, and a full crawl returns zero repeated-segment matches. Crawl depth stays shallow and predictable. Search Console shows crawl requests concentrated on real pages, not on folder loops. When you fix a repetitive path properly, the junk stops being generated at the source, the redirects mop up what already leaked out, and your crawl budget goes back to the pages that actually earn you traffic.
FAQ
Is a repeated word in a slug the same problem?
/best-best-sellers/ is one slug with a repeated word and it is fine. The issue is a repeated path segment between slashes, like /best/best/, which usually means a broken relative link.Can I just block the pattern in robots.txt?
Why does the URL count keep growing every crawl?
Should I 301 or 410 the junk URLs?
Repetitive paths are usually the tip of a template problem. An audit finds the source link, maps the junk, and hands you the fix list.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
