How to Find Lower-Quality Content Being Excluded From Indexing
- October 5, 2021
- Cross-Industry
AI Summary
When a search engine crawls a page and then declines to index it, that decision is a quality verdict you can read and act on. Export the excluded URLs from Google Search Console and Bing Webmaster Tools, group them by cause, then improve, consolidate, or remove the genuinely weak ones.
- Crawled not indexed and discovered not indexed are value judgments, not technical glitches.
- Quality is judged at the site level, so a large pool of weak URLs can drag down strong pages.
- When both Google and Bing exclude the same URL, the quality signal is strongest.
- The goal is a lean set of pages that each earn their place, not the largest possible index.
TL;DR
When a search engine crawls a page and decides not to index it, that decision is a quality verdict you can read. Pull the excluded URLs from Google Search Console and Bing Webmaster Tools, look for patterns (thin pages, duplicates, doorway-style content), then improve, consolidate, or remove the genuinely weak ones. Treating exclusion as feedback rather than a glitch is one of the cleanest ways to raise the average quality of a site.
Most site owners watch indexing reports hoping every URL turns green. A more useful habit is the opposite: study the pages the engine refuses to index. Search engines crawl far more than they keep, and the gap between what they fetch and what they index is one of the most honest quality signals available. This methodology, popularized by Glenn Gabe, treats that gap as a map of where a site is weakest, so you can act before it drags down everything around it.
Why exclusion is a quality signal
When you see statuses such as "Crawled, currently not indexed" or "Discovered, currently not indexed," the engine is telling you something specific. It found the URL, looked at it (or chose not to spend the resources to), and judged that adding it to the index was not worth doing. That is a value judgment, not a technical error. The engine is effectively saying it sees little reason for the page to compete in results.
This matters because quality is increasingly evaluated at the site level, not only page by page. A large pool of thin, duplicative, or low-value URLs can weigh on how the whole domain is perceived. Pages the engine quietly declines to index are exactly the candidates that may be diluting your site. Listening to that signal early lets you fix the cause rather than wonder why strong pages underperform.
One caution up front: not every excluded URL is a problem, and some exclusion is completely normal. The skill is in separating expected exclusion from the exclusion that reveals real weakness.
How to find excluded content
Start with the Google Search Console Page indexing report. It groups URLs by why they are not indexed, and the buckets worth your attention include crawled-not-indexed, discovered-not-indexed, duplicate without a user-selected canonical, and alternate pages with a proper canonical. Export the affected URLs from each relevant group so you can work with the full list rather than the sample shown on screen.
Then bring in Bing Webmaster Tools. Bing's coverage and sitemap reporting often surfaces a different slice of excluded or problematic URLs, and a second engine's read on the same site is a valuable cross-check. When both engines decline to index the same pages, the quality signal is much stronger than when only one does. Comparing your submitted sitemap against what each engine actually indexes shows you the delta directly.
With both lists in hand, look for patterns instead of treating each URL alone. Common clusters include thin pages with little unique content, near-duplicate variants generated by filters or parameters, tag and archive pages that add no value, auto-generated location or doorway content, and orphaned pages nothing links to. While reviewing, also confirm these are not soft 404 errors being misread, and verify nothing valuable is being blocked upstream by checking your robots.txt configuration.
How to act on it
Once you have grouped the excluded URLs, each cluster points to one of three actions. Improve the pages that should rank but are too thin, by adding genuine depth, original information, and a clear reason to exist. Consolidate near-duplicates and overlapping pages into a single strong URL, redirecting the weaker versions so their value is combined rather than scattered. Remove or noindex the pages that have no path to quality, such as endless filtered variants or content created only to chase keywords.
This is precisely how a site recovers quality. By shrinking the pool of low-value URLs and strengthening the rest, you raise the average and give the engine fewer reasons to doubt the domain. The goal is not the largest possible index footprint; it is a lean set of pages that each earn their place.
A practical workflow
Export the excluded URLs from the Google Search Console Page indexing report and from Bing Webmaster Tools coverage. Combine them into one working sheet and tag each URL by suspected cause. Sort into the three buckets: improve, consolidate, or remove. Prioritize by where the largest clusters and the most strategically important sections sit. Make the changes in batches, then watch the indexing reports over the following weeks to confirm that improved pages get indexed and removed pages drop out cleanly. Revisit on a recurring schedule, because new exclusion patterns are an early warning that a template or content process has started producing weak pages again.
What each exclusion status is telling you
| Status in the reports | What it usually signals | Typical action |
|---|---|---|
| Crawled currently not indexed | Fetched but judged not worth indexing, often thin or duplicative | Add depth, or consolidate into a stronger page |
| Discovered currently not indexed | Known but not yet crawled, a crawl priority or budget signal | Strengthen internal links and page value |
| Duplicate without user selected canonical | Overlapping pages competing for the same intent | Consolidate and set a clear canonical |
| Alternate page with proper canonical tag | Correctly canonicalized, usually expected | No action needed |
| Soft 404 | Thin or empty page treated as not found | Add real content, or return a true 404 |
| Excluded by noindex tag | Intentionally kept out of the index | Confirm the noindex is deliberate |
FAQ
Does excluded always mean low quality?
No. Some exclusion is normal, such as correctly canonicalized alternates, intentional noindex pages, and recently published URLs still waiting to be processed. The signal worth acting on is the recurring pattern of pages that should be valuable but are repeatedly declined.
Why use Bing as well as Google?
A second engine gives you an independent read on the same pages. When both engines exclude the same URLs, you have stronger confirmation that the issue is content quality rather than one engine's quirk.
Should I just noindex everything that is excluded?
No. Decide page by cluster. Pages that should rank need improvement, overlapping pages need consolidation, and only the genuinely valueless ones should be removed or set to noindex. Blanket action can bury pages that simply needed more depth.
Which Google Search Console report shows excluded pages?
The Page indexing report groups every known URL by why it is or is not indexed. Open the buckets for crawled not indexed, discovered not indexed, duplicate without a user selected canonical, and alternate page with proper canonical tag, then export the affected URLs to work the full list.
How long after I improve a page should it get indexed?
There is no fixed timer, but you usually see movement over a few weeks as the engine recrawls and re evaluates. Watch the indexing report over the following weeks to confirm improved pages get indexed and removed pages drop out cleanly.
Is crawled not indexed the same as discovered not indexed?
No. Discovered not indexed means Google knows the URL exists but has not crawled it yet, often a crawl priority signal. Crawled not indexed means it fetched the page and still chose not to index it, which is a clearer quality verdict.
Turn indexing gaps into a quality plan
If your excluded-URL list is large or hard to interpret, an audit can sort the signal from the noise and hand you a prioritized fix list.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
