AI Summary
This case study documents how an ecommerce team reclaimed crawl budget that was being drained by faceted navigation. The core method is to find the waste in server logs, apply the right mix of robots controls, canonical tags, and internal link pruning, then let Googlebot refocus on pages that earn revenue.
- Faceted navigation is the classic crawl budget sink: filters and sorts multiply into near infinite URLs.
- The fix is layered: robots.txt to block crawl, canonical to consolidate signals, noindex for thin variants.
- Robots.txt stops crawling but not always indexing, so combine controls deliberately.
- Measure with server logs and the Crawl Stats report before and after each change.
This SEO case study documents a successful optimization initiative, providing actionable insights for practitioners. The documented approach demonstrates how strategic SEO implementation drives measurable results.
Initial Situation
Understanding the starting point is essential context for evaluating any case study. This documentation covers the initial challenges, competitive position, and business objectives that shaped the SEO strategy.
Strategy and Approach
The strategic approach combined multiple SEO disciplines to address identified opportunities. Key decisions around prioritization and resource allocation provide a template for similar initiatives.
Implementation
Moving from strategy to execution required specific technical implementations, content development, and process changes. This case study documents the practical steps that translated strategy into action.
Results and Learnings
The outcomes demonstrate effectiveness through measurable improvements in rankings, traffic, and business metrics. Analysis of successes and challenges provides learning value for practitioners.
Case studies like this contribute to the SEO knowledge base, helping practitioners learn from documented real-world experiences.
Source: https://engineering.skroutz.gr/blog/seo-crawl-budget-optimization-2019/
The faceted navigation problem
On a large catalog, every filter you expose can be combined with every other filter. Color, size, brand, price band, sort order, and page number multiply into a combinatorial explosion of URLs, most of which show near duplicate listings. Left unchecked, Googlebot spends a large share of its fetches wandering through these permutations instead of crawling the category and product pages that actually convert. This is the single most common reason a big ecommerce site struggles to get new products discovered quickly. It is the same pattern discussed in faceted navigation strategy.
Controls that actually move the needle
There is no single switch. Each control does one job, and you combine them on purpose:
- robots.txt Disallow stops Googlebot fetching a URL pattern. It saves crawl capacity immediately, but a blocked URL can still be listed as a bare link if it is found elsewhere, because it controls crawling rather than indexing.
- rel canonical points a filtered variant at the clean category URL so ranking signals consolidate. It works only if the variant is crawlable, so do not both block and canonicalize the same URL.
- noindex, follow on a meta robots tag removes a thin variant from the index while still letting Google follow its links. Use it where you want the page gone but the crawl path kept.
- Internal link hygiene means not linking to filter combinations you do not want crawled, and avoiding endless nofollowed filter links that still create discoverable paths.
A worked robots.txt example
If your filters live in query parameters, a pattern block can cut the worst waste quickly:
User-agent: *
Disallow: /*?*color=
Disallow: /*?*sort=
Disallow: /*?*sessionid=Roll changes out in stages and watch the Crawl Stats response breakdown. If you block a pattern, confirm those URLs are not also the ones you rely on for indexing, and make sure the clean canonical version stays reachable. This is why the order of operations matters: decide what should be indexed first, then choose the control that fits.
Reading the result
Success looks like a falling share of fetches on parameter URLs, a rising share on product and category templates, faster discovery of new products, and a healthier ratio of valid to excluded pages in the Page Indexing report. For the definition behind all of this, see what crawl budget is.
Which control for which goal
| Control | Stops crawling? | Removes from index? | Best used for |
|---|---|---|---|
| robots.txt Disallow | Yes | No, may still list URL | High volume junk parameters |
| rel canonical | No | Consolidates signals | Near duplicate filtered pages |
| meta noindex, follow | No | Yes | Thin variants to remove but still crawl through |
| Internal link pruning | Indirect | Indirect | Reducing discoverable junk paths |
What has changed since this case study
Google deprecated the old URL Parameters tool in Search Console, so parameter handling is now something you control on your own site rather than declaring to Google. The rel next and rel prev pagination hints were also retired years ago, which pushes the emphasis toward strong internal linking and canonical clarity instead. Googlebot renders JavaScript, so filters implemented purely in the front end can still create crawlable URLs if they update the address bar. The measurement toolkit has improved too: the Crawl Stats and Page Indexing reports now give a clearer before and after picture than the tooling available when many of these faceted navigation projects were first written up.
Related on SEO ProCheck
Frequently asked questions
What causes crawl budget waste on ecommerce sites?
Faceted navigation is the usual culprit. Filters, sort orders, price bands, and pagination combine into a huge number of near duplicate URLs that Googlebot fetches instead of your core product and category pages.
Should I block filters in robots.txt or use noindex?
Use robots.txt to stop crawling of high volume junk parameters, and use noindex where you need a page removed from the index but still want Google to follow its links. Do not block and canonicalize the same URL, because a blocked URL cannot pass its canonical.
Does canonical fix crawl budget?
Not on its own. Canonical consolidates ranking signals, but Google still has to crawl the variant to read the tag. For pure crawl savings you need robots.txt or fewer discoverable links, then use canonical to keep signals clean.
How do I prove the optimization worked?
Compare server log fetch distribution and the Crawl Stats report before and after. Look for fewer fetches on parameter URLs, more on product and category templates, and faster discovery of new pages.
Will removing filter URLs hurt rankings?
Not if you consolidate correctly. The clean category and product pages keep their signals through canonical tags and internal links, while the thin filter permutations that never ranked are simply removed from the crawl path.
Is faceted navigation always bad for SEO?
No. Well governed facets can target real demand. The problem is uncontrolled combinations. Decide which filtered views deserve indexing, make those clean and linkable, and suppress the rest.
Take this with you
The assistant links open with a question about this page already written. Nothing is sent until you press enter.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
Recent Posts
- Most Top Sites Never Say Who They Are in Structured Data October 14, 2026
- Homepage Titles Are Mostly Too Short, Not Too Long October 8, 2026
- Who Actually Serves an llms.txt? 184 Top Sites Measured September 29, 2026