Crawl Trap

A crawl trap is any part of a site that manufactures a practically infinite supply of crawlable URLs — calendars that page forward forever, filters that combine into millions of permutations, session IDs that mint a fresh URL per visit. The stakes: crawlers burn their request allowance on generated junk while the pages you actually want indexed wait in line, and on big sites the junk can outnumber real pages a thousand to one.

How a handful of templates becomes a million URLs

The classic example is an events calendar. One template, two links ("previous month," "next month"), and every month is a valid URL:

/events/2026/07/
/events/2026/08/
/events/2026/09/
...
/events/2189/11/   <- still returns 200, still links to /events/2189/12/

Nothing ever 404s, so the crawler keeps going. Real crawls of neglected calendar plugins turn up URLs decades into the future, every one fetched, every one empty.

Faceted navigation is the same failure with multiplication instead of addition. A category with 12 colors, 8 sizes, 6 brands, and 4 sort orders yields 12 × 8 × 6 × 4 = 2,304 URL variants — per category, before pagination, and before filters combine with each other in different orders (?color=red&size=m vs ?size=m&color=red are two URLs to a crawler). A 200-category store quietly offers Googlebot half a million addresses for maybe 5,000 real pages. The full containment playbook for that specific trap is in the faceted navigation guide.

Trap types and their detection signatures

Each trap type leaves a distinct fingerprint in a crawl or a log file. This is the identification table:

How to check it on your own site

1. Run a crawler with a URL cap (say 100k) and watch the discovered-vs-crawled counter. On a healthy site, discovery flattens out. If the queue keeps growing while the crawl runs, something is generating URLs faster than they can be fetched — that's the trap smell.
2. Sort the crawl by URL path and eyeball the patterns. Traps cluster: ten thousand URLs sharing a template with only parameters changing jump right out of a sorted list.
3. Grep your access logs for what Googlebot wastes time on:

   grep Googlebot access.log | awk '{print $7}' | grep -o '^[^?]*' | sort | uniq -c | sort -rn | head -20
   grep Googlebot access.log | grep -c '?'     # share of parameterized fetches

   If more than a modest slice of Googlebot's requests carry query strings you never intended to index, you have your answer.

4. Check GSC → Settings → Crawl stats for the "Crawled but not indexed" and by-file-type breakdowns, and GSC → Indexing → Pages for ballooning "Duplicate" and "Crawled – currently not indexed" buckets — traps inflate both.
5. Verify the fix holds: after adding robots rules per the robots.txt reference, re-crawl with the same cap and compare unique indexable URLs. The number should drop toward your real page count.

Common audit mistakes

• Declaring "no traps" after a capped crawl finished cleanly. The cap is exactly what hid the trap. Check whether discovery had flattened before the crawl ended, not just whether it ended.
• Fixing traps with canonical tags alone. Canonicals consolidate indexing signals, but Google has to crawl the URL to read the tag — the crawl waste continues at full speed. Traps need crawl-level blocks, not just index-level hints.
• Slapping noindex on trap URLs and calling it done. Same problem: noindex is read after the fetch. It empties the index, not the crawl queue.
• Blocking the trap in robots.txt after thousands of junk URLs got indexed. Now Google can't recrawl them to see they're gone, so they linger. Sequence matters: let them return 404/410 or noindex first, block the pattern once the index is clean.
• Worrying about crawl traps on a 200-page site. Googlebot will crawl a small site's junk and its real pages without breaking a sweat. Trap remediation earns its keep at tens of thousands of URLs and up — crawl budget is a big-site problem.

FAQ

How is a crawl trap different from ordinary duplicate content?

Scale and generation. Duplicate content is a finite set of overlapping pages. A trap is a URL generator — the set isn't finite, so no amount of one-by-one cleanup ends it. You fix the generator, not the URLs.

Won't Google figure the trap out on its own?

Eventually, partially. Googlebot does detect infinite spaces and throttles them, but "eventually" can mean months of diluted crawling first, and the detection is conservative — it keeps sampling the trap indefinitely. Don't outsource your plumbing to Google's patience.

Is robots.txt or nofollow the better fix?

Robots.txt blocks the fetch, which is what you want for a trap. Nofollow on the linking elements helps discovery but Google treats it as a hint, not a barrier. Belt and suspenders: nofollow the filter links and disallow the pattern.

Can a crawl trap hurt rankings directly?

Indirectly but genuinely: important pages get crawled and refreshed less often, new content takes longer to index, and if junk URLs make it into the index they drag down the site's aggregate quality picture. No single "trap penalty" exists; the damage is death by dilution.

What's the fastest way to spot one right now?

The log grep in step 3. Sixty seconds against yesterday's access log tells you what Googlebot actually spends your server's time on — no crawl needed.