Gemini

What Gemini is (for site owners)

Gemini is Google’s family of AI models and the assistant built on them: the Gemini app, the AI layer inside Google Search (AI Overviews and AI Mode), and the sidekick across Workspace. What makes Gemini different from ChatGPT or Perplexity for a site owner: there is no separate Gemini crawler to manage. Everything rides on Googlebot’s existing crawl, and your only Gemini-specific control is a robots.txt token that most people misunderstand.

How Gemini gets web content

Google reuses its Search infrastructure. Googlebot crawls as it always has; that one crawl then feeds multiple AI surfaces through different paths:

• Grounding: AI Overviews, AI Mode and the Gemini app can query the live Search index and cite indexed pages.
• Training: crawled content can be used to train Gemini models, unless you opt out via Google-Extended.

And here is the part to be precise about: Google-Extended is not a crawler. No user agent called Google-Extended will ever appear in your logs. It is a control token that Googlebot reads in robots.txt to decide whether your already-crawled content may be used for Gemini training and grounding in Gemini apps. Blocking it does not affect your crawling, your rankings, or your inclusion in Google Search, and per Google’s documentation, it does not remove you from AI Overviews either, because those are part of Search and grounded on the normal index.

Gemini surfaces and the path your content takes

Example: the opt-out that doesn’t do what people think

User-agent: Google-Extended
Disallow: /

After shipping this, a site keeps its rankings, keeps appearing in Google Search, and can still show up in AI Overviews. What changed is narrower: Google stops using the content to train Gemini models and for grounding in Gemini apps. If the actual goal was “keep my text out of AI answers on Google Search,” the relevant levers are snippet controls:

<meta name="robots" content="nosnippet">

That has the side effect that regular snippets disappear too. There is no scalpel that cuts AI Overviews out while leaving classic results untouched.

How to check it

1. Read your robots.txt and locate any Google-Extended group. Confirm the team knows what it does and doesn’t do before keeping or removing it.
2. Don’t grep for Google-Extended: it never appears as a UA. Instead confirm the underlying crawl is healthy:

   grep -ic "googlebot" access.log

3. Check indexation in Search Console: a page absent from the index is absent from every grounded Gemini surface, full stop.
4. Prompt-test both surfaces: ask the Gemini app and AI Mode the questions your pages answer; note whether you’re linked as a source. Repeat across sessions before concluding anything.
5. Review snippet directives (nosnippet, max-snippet): teams sometimes shipped these years ago and now wonder why AI surfaces skip them.

Common mistakes

• Adding a WAF rule for the “Google-Extended” user agent. It matches nothing, because no such UA exists. Worse, sloppy patterns can catch real Googlebot traffic.
• Blocking Google-Extended to escape AI Overviews. Wrong lever: AI Overviews are grounded on the Search index and governed by indexing and snippet controls.
• Blocking Googlebot to escape AI. That escapes Google entirely, organic traffic included. Almost never what anyone means.
• Assuming Gemini visibility is a separate optimization discipline. For grounded answers it inherits your Google SEO nearly wholesale: where the disciplines genuinely differ is mapped in SEO vs AEO vs GEO.
• Forgetting old snippet restrictions. A max-snippet:20 from a 2019 CCPA sweep quietly caps what any Google AI surface can quote from you.

FAQ

Is there a GeminiBot I should allow?

No. Gemini has no dedicated crawler; Googlebot does the fetching and Google-Extended is only a usage-policy token evaluated against robots.txt. The wider token landscape is covered in the complete robots.txt reference.

Does blocking Google-Extended hurt my rankings?

Google says no: it is not a ranking signal and does not affect crawling or indexing. It strictly governs Gemini training and grounding usage.

How do I know if Gemini cites my site?

Manual testing, mostly. Ask the Gemini app your target questions and check the source links. For AI Overviews, impressions and clicks fold into ordinary Search Console data with no separate report.

Should I opt out of Google-Extended?

For most commercial sites the honest answer is that opting out buys little: your competitors’ content still trains the models, and you give up nothing measurable by staying in. Publishers with licensing leverage are the exception. What AI-grounded search means for the bottom line is laid out in what AI search changes for your business.

Do AI Overviews use content from pages blocked via Google-Extended?

They can, yes: AI Overviews are a Search feature grounded on the regular index, and Google-Extended does not govern Search features. That is precisely why the token confuses people.