Indexing

Indexing is the process by which a search engine stores and organizes the pages it has crawled so they can be retrieved and ranked in response to a query.

Indexing happens after crawling and, for pages that depend on scripts, after rendering. Google parses the page content, evaluates signals such as canonical tags, structured data, and language, then decides whether to add the URL to its index. A page can be crawled and still never indexed if Google judges it duplicative, thin, or low value. The reverse is also true: a URL can be indexed without being crawled recently. Inclusion in the index is a prerequisite for ranking, but it does not guarantee visibility.

You influence indexing with several controls. A noindex directive keeps a page out of the index even after it is crawled. A canonical tag points consolidation toward a preferred URL. Robots.txt blocks crawling but does not by itself remove a page from the index, since a blocked URL can still be indexed from external links. To reliably exclude a page, allow crawling and serve a noindex tag, or use the URL removal tools.

In Search Console, the Pages report groups URLs as indexed or not indexed with reasons such as "Crawled - currently not indexed" or "Discovered - currently not indexed." These reasons help diagnose whether the issue is quality, crawl budget, or a blocking directive. On large sites, monitoring the ratio of submitted to indexed URLs flags index bloat and wasted crawl effort early.

Related: Indexation check, Index bloat, Noindex, Mobile-first indexing