Indexing

Indexing is the step where a search engine decides to keep a page it has crawled: the content gets parsed, deduplicated, and stored in the database that queries are answered from. A page that misses this cut is invisible in search no matter how good it is — and on real sites the miss rate is brutal, with large sites routinely seeing a third or more of their crawled URLs left out of Google's index.

That gap is money. If you publish 400 product pages and 150 of them sit in "Crawled — currently not indexed," you paid for 400 pages of content and can rank with 250. Nobody refunds the difference.

Crawled is not indexed, and indexed is not ranked

The pipeline runs discovery → crawl → render → index → rank, and each arrow is a filter. After fetching a page, Google evaluates whether it deserves storage: is it a duplicate of something already indexed, is it thin, does a canonical tag point elsewhere, does a noindex directive forbid it. Only survivors become candidates for ranking, and inclusion still guarantees zero impressions — the index is the entry ticket, not the prize.

Two directives control the door from your side. In the HTML head:

<meta name="robots" content="noindex, follow">

or as an HTTP header, which also works for PDFs and other non-HTML files:

curl -sI https://www.example.com/internal-report.pdf | grep -i x-robots

x-robots-tag: noindex

One trap worth engraving somewhere: a robots.txt Disallow does not deindex anything. It prevents the fetch, which means Google can never read the noindex you put on the page. URLs blocked this way can linger in the index for months as bare links — there's a full case study on fixing exactly that.

Reading the GSC Page indexing report

Search Console's Pages report is where indexing reality lives. Each state means something specific, and half the diagnostic work is knowing which states are problems and which are the system working as designed:

A deeper walkthrough of this report, state by state, is in the Page indexing report guide.

How to check indexing on your own site

1. GSC → Indexing → Pages. Note the indexed count and read every "Why pages aren't indexed" bucket. Trend matters more than snapshot — a falling indexed line with rising "Crawled — currently not indexed" is a quality signal from Google.
2. URL Inspection on your ten most important URLs. Confirm "URL is on Google," and compare the user-declared canonical against the Google-selected one. Disagreement means your signals conflict.
3. Filter the Pages report to "All submitted pages." This scopes the report to your sitemap, so the not-indexed buckets contain only URLs you actively claimed matter — the honest measure of your indexation rate.
4. Automate it if you run more than a few thousand URLs — the URL Inspection API lets you batch-check 2,000 URLs a day per property and catch deindexing before traffic charts do.
5. Use site:example.com only as a rough smoke test. The count is an estimate and fluctuates wildly; treat it as a smell, never a metric.

Common mistakes

• Deindexing with robots.txt. Covered above, still the single most repeated error in audits. Fix: allow the crawl, serve noindex, wait for refetch, then optionally block.
• A leftover sitewide noindex from staging. One deploy of the wrong config file and your index evaporates over two weeks. Fix: automated post-deploy check that greps production HTML for noindex on key templates.
• Expecting thin doorway-style pages to index. Google's storage bar has risen every year; near-duplicate location pages or spun variants pile up in "Crawled — currently not indexed." Fix: consolidate or genuinely differentiate — there is no markup trick around a quality judgment.
• noindex plus canonical on the same page. Contradictory: one says "I don't exist," the other says "count me toward that URL." Google may ignore either. Pick one intent per page.
• Panicking over "Page with redirect." After a migration this bucket balloons and that is exactly what should happen. Investigate spikes without an accompanying release, not the bucket itself.

FAQ

How long does indexing take after publishing?

Anywhere from hours to weeks. News-y sites with high crawl demand see near-instant pickup; new or low-authority sites wait longest. Submitting via URL Inspection's "Request indexing" nudges a single URL but doesn't override the quality evaluation.

Why is my page crawled but not indexed?

Because Google fetched it and elected not to store it — usually a duplication or value judgment. Look at what already ranks for the topic; if your page adds nothing beyond it, that's the answer.

Can I pay or submit my way into the index?

No. There is no paid inclusion in Google web search, and the Indexing API is officially restricted to job posting and broadcast-event pages. For everything else it's sitemaps, internal links, and content that clears the bar.

How do I get a page OUT of the index fast?

GSC's Removals tool hides it within about a day (temporary, ~6 months), while a noindex tag or 404/410 makes it permanent once recrawled. Use both together when it's urgent — removal for speed, directive for permanence.

Does "indexed" mean the whole page is stored?

Effectively yes for ranking purposes — and Google can also rank specific passages within an indexed page independently, so a well-structured long page can surface for queries its title never mentions.