Webinar On-Demand: SEO Case Study: How To Index Millions Of Pages Through Technical SEO Improvements - Lumar
- February 16, 2023
- Content
AI Summary
This Lumar on demand webinar features Anna Uss, SEO team lead at Snyk, explaining how the developer security company closed an indexing gap on a site of roughly 2 million pages of which only about 400,000 were indexed. The fix was structural rather than editorial: directory hub pages, denser internal linking, and two way links between package pages and directories, after which roughly 1 million more pages were indexed and organic clicks more than doubled within six months.
- Presented by Anna Uss (Snyk), published by Lumar on 13 February 2023.
- Starting point: about 2 million pages, about 400,000 indexed.
- Result: roughly 1 million additional pages indexed, clicks more than doubled in six months, average position stable.
- Progress was tracked with internal crawls first, because Search Console lags on sites this size.
What the webinar covers
Lumar's on demand session is a case study presentation rather than a tool demo. Anna Uss, SEO team lead at Snyk, walks
through an indexation problem of a shape that only appears on large programmatic sites: the pages existed, the content was
legitimate, and Google simply was not fetching most of them. Snyk's site carried roughly two million pages built around
open source package data. Around four hundred thousand of those were indexed. The remaining pages were not penalised or
deindexed; they had never been meaningfully crawled.
The changes presented are structural. Snyk created directory pages that list and link to package URLs, improved internal
linking across the site, and linked those directory pages back from the individual package pages so discovery flowed in both
directions. The reported outcome was approximately one million additional pages indexed, organic clicks more than doubling
within six months, and click-through rate rising while average position stayed broadly stable. The session also stresses
that programmatic SEO should mean quality content assembled from templates and databases, not auto-generated filler.
Practitioner commentary: why this pattern works
The result here is often misread as "Snyk added pages and got traffic." They did not add pages. The two million URLs
already existed. What changed was the link graph around them, and that distinction is the whole lesson.
Google's indexing pipeline has three gates, and large sites usually fail at the first one while their owners debug the
third. Discovery decides whether Googlebot ever learns a URL exists. Crawl scheduling decides whether it is worth fetching
given everything else in the queue. Index selection decides whether the fetched document earns storage. A page reachable
only through an internal search form or a JavaScript-driven filter fails at gate one no matter how good it is. Adding it to
an XML sitemap does not reliably fix this either: a sitemap is a hint about which URLs exist, while internal links are the
signal Google uses to judge which URLs matter.
Directory pages fix gate one and gate two simultaneously. They convert an orphaned set into a linked hierarchy, and they
collapse click depth. On most large sites the correlation between click depth and crawl frequency is brutal: pages four or
five clicks from the home page get fetched a fraction as often as pages two clicks away. Cutting depth from six to three
across a million URLs changes the crawl economics of the entire domain.
How to run this diagnosis on your own site
Before copying the fix, confirm you have the same problem. Work through the gates in order.
| Gate | Symptom | Where you check it | The fix that applies |
|---|---|---|---|
| Discovery | URL is in the sitemap but your crawler cannot reach it by following links | Crawl with sitemap ingestion on, filter to orphan URLs | Directory or hub pages, category listings |
| Crawl scheduling | Googlebot fetches the URL rarely or never; deep pages have stale last-crawl dates | Server logs filtered to verified Googlebot; Crawl Stats report | Reduce click depth, remove duplicate URL variants |
| Index selection | Crawled but marked Crawled, currently not indexed | Search Console Pages report | Increase per-page unique value, consolidate near-identical templates |
| Serving | Indexed but no impressions | Search Console Performance, filtered by page | Query demand or relevance problem, not an indexing problem |
The most expensive mistake is treating a discovery failure as an index selection failure and rewriting content that Google has never fetched.
The single most useful artefact is an orphan report. Configure your crawler to read the XML sitemap and
the internal link graph, then list every sitemap URL with zero inbound internal links. That list is your directory page
backlog, ordered by how much of it clusters under a common path. Our
internal link analyzer surfaces the same
distribution for smaller sites, and the
log file analyzer tells you whether Googlebot is
actually reaching the pages after you link them.
Designing directory pages that Google keeps
A directory page that is nothing but a wall of links is itself a thin page, and thin hub pages get dropped, taking their
discovery value with them. Three design rules keep them alive:
Paginate deliberately, not defensively. Cap each directory at a few hundred links rather than tens of
thousands. Each paginated page needs a self referencing canonical, because page seven contains items page one does not.
Canonicalising pagination back to page one is the classic way to undo the entire exercise.
Give the hub its own reason to exist. Even two or three sentences of genuine orientation, plus sorting
or grouping that reflects real structure in the data, moves a directory from an index of links to a usable page. This is
also what separates it from the scaled content pattern Google penalises.
Link back up as well as down. Snyk's third change, linking directory pages from the individual package
pages, matters more than it sounds. One way top-down linking creates a funnel that dries up as soon as the hub itself drops
in crawl priority. Two way linking distributes discovery across the whole cluster.
What's changed since February 2023
The mechanics in this webinar have aged well; the risk environment around them has not stayed still. Three developments
matter if you are applying this pattern now.
Scaled content abuse became a named spam policy. In March 2024 Google updated its spam policies to
include scaled content abuse, replacing the narrower automatically-generated content wording. The policy is explicitly
agnostic about how pages are produced: what matters is whether many pages are generated primarily to manipulate rankings
without serving users. A directory built over a real proprietary dataset, as in this case, is not the target. A directory
built by templating a language model over keyword permutations is. If you are applying the Snyk pattern in 2026, the
question to answer first is whether each generated page contains information that exists nowhere else.
The helpful content system stopped being a separate layer. Also in March 2024, Google folded helpful
content signals into its core ranking systems rather than running them as a distinct sitewide classifier. Practically, this
means large-scale thin page problems now surface as gradual core update movement rather than a single identifiable hit,
which makes them harder to diagnose and slower to recover from.
Crawl budget now has AI crawlers competing for it. Since 2023 the number of automated agents fetching
pages has grown substantially, and server capacity spent on them is capacity not spent serving Googlebot. On sites of this
size it is worth segmenting logs by user agent before concluding that Googlebot's crawl rate is the constraint. Our
GEO and AI search consulting page covers how to read that split.
One caveat on the numbers: the figures above are those presented by Snyk in the webinar, which is a vendor-hosted case
study rather than an independently audited dataset. They are internally consistent and plausible for a site of that shape,
but they are self-reported. Treat them as a directional pattern to test, not a benchmark to promise a stakeholder.
Applying it in order
If your site has an indexing gap of this shape, the sequence that works is: build the orphan report, group orphans by
path pattern, ship directory hubs for the largest groups, add reciprocal links from detail pages, then watch server logs
weekly for crawl rate change before you expect anything in Search Console. Sites with a related duplication problem should
resolve that first, because crawl budget spent on URL variants is budget unavailable for the pages you are trying to get
discovered. The broader indexation workflow is covered on our
performance and indexation page, and there is a
related structural example in our
Saramin technical SEO briefing.
FAQ
Indexing is a selection process, not an entitlement. Google discovers a URL, decides whether it is worth crawling, crawls it, then decides whether it is worth storing. Pages fail at the discovery stage when nothing links to them, at the crawl stage when budget is exhausted on duplicates, and at the storage stage when the content is too thin or too similar to pages already indexed. Diagnose which stage is failing before you change anything.
A directory page is a hub that lists and links to a large set of detail URLs. It solves discovery: without one, programmatically generated detail pages are often orphaned, reachable only from an internal search box that crawlers cannot use. Adding directory pages gives every detail URL a crawlable parent and cuts its click depth from the home page, which is one of the strongest predictors of whether Google bothers to crawl it.
In the Snyk case described in this webinar, organic clicks more than doubled within six months. That is a reasonable planning horizon for large scale indexation work: crawl rate changes appear in server logs within weeks, index coverage shifts over months, and traffic follows last. Google increases crawl rate gradually as it finds the newly linked pages worth fetching.
Programmatic page generation itself has never been against Google's guidelines; generating pages at scale with little original value is. Google's March 2024 spam policy update added scaled content abuse as a named violation, which targets mass-produced pages that do not help users regardless of whether a human or a machine made them. A template populated from a genuine proprietary database, which is what Snyk had, sits on the safe side of that line.
Both, for different jobs. Search Console tells you what Google actually did, but it lags by days and samples heavily on large sites. An internal crawl tells you what Google will encounter next: orphan pages, click depth, broken internal links. Snyk's team used internal crawling specifically so they could see progress before it appeared in Search Console, which is the right pattern when your feedback loop would otherwise be weeks long.
Reducing click depth on orphaned templates. Find URLs in your sitemap that your own crawler cannot reach by following links, then give them a linked parent. It requires no new content, no new URLs and no migration, and it addresses the single most common reason large sites have millions of pages that Google has never fetched.
Sitting on millions of pages Google has never fetched?
An advanced audit maps your orphan clusters, click depth and crawl allocation, then sequences the fix.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
