LLM (Large Language Model)

What an LLM is

A Large Language Model (LLM) is a neural network trained on enormous amounts of text to predict the next token, which turns out to be enough to write, summarize, translate, and answer questions — it's the engine inside ChatGPT, Claude, Gemini, and every AI search feature. For anyone who runs a website, the stakes are practical: LLMs are becoming a distribution layer between your content and your audience, and whether they know about you depends on decisions you make in robots.txt, your HTML, and your publishing habits.

The example that matters for SEO: training vs retrieval

Here's the distinction that untangles most confusion. When you ask ChatGPT "what's a good CRM for a small agency," it might answer two ways. From training: the model recalls patterns absorbed from its training corpus — a snapshot that ended months or years ago, with no source links and no way to trace why it named those brands. Or from retrieval: the system runs a live web search, reads current pages, and synthesizes an answer with citations. Same chat window, completely different mechanics — and completely different optimization levers. Training presence is slow, cumulative, and built from how often and how consistently the wider web mentions you. Retrieval presence is fast and depends on being crawlable, indexed, and quotable today. The AI citation entry covers the retrieval side in depth.

Major LLM families and how each one gets your content

Two consequences worth underlining. Blocking one bot doesn't block "AI" — each row is a separate decision, and blocking Googlebot-based access would take your search visibility down with it. And because much training data flows through public crawls, a site can end up in models it never explicitly allowed or blocked.

How to check LLM interaction with your own site

1. Read your robots.txt right now. List which of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Meta-ExternalAgent you allow or block. Many sites carry blanket blocks a plugin added in 2023 that nobody has revisited.
2. Grep your access logs for those user agents. Real crawl activity tells you which systems are actually reading you, and how often. Zero visits from search-side bots (OAI-SearchBot, PerplexityBot) means zero chance of live citations from those engines.
3. Interrogate the models. Ask ChatGPT, Claude, and Gemini — with browsing off, where possible — what they know about your brand. That probes training-data presence. Then repeat with search on to test retrieval presence. The gap between the two answers is your roadmap.
4. Fetch a key page as a bot would. curl it without JavaScript execution. Most AI crawlers don't render JS, so client-side content is invisible to them; what the raw HTML shows is what an LLM gets. A structured way to think about this is our practical framework for LLM consumption.
5. Decide on llms.txt with clear eyes. It's a proposed convention for pointing LLMs at your key content — adoption by major engines remains limited, so treat it as cheap insurance, not a ranking lever. Background in llms.txt explained, and our llms.txt generator builds one in minutes.

Common mistakes (and the fix)

• Talking about "the LLM" as one thing. Optimization decisions differ per family and per mode (training vs retrieval). Fix: use the table above; decide per bot.
• Blocking every AI crawler on principle, then buying "AI visibility" tooling. You can't be cited by systems you 403. Fix: align the access policy with the visibility goal — they're the same lever.
• Expecting a model to know last month's launch. Training snapshots lag reality by months at minimum. Fix: for anything time-sensitive, retrieval surfaces are the only game — publish crawlable, quotable pages.
• Trusting an LLM's claims about itself. Models routinely misstate their own cutoffs and capabilities. Fix: verify crawler behavior in your logs and engine behavior by testing, not by asking the model.
• Shipping JS-only content and wondering why AI answers ignore it. Fix: server-side render anything you want machines to read — schema and clean markup help too, per our machine-readable web standards research.

FAQ

Do LLMs crawl the web like Google does?

The chat model itself doesn't crawl anything. Companies run crawlers to collect training data (GPTBot, ClaudeBot) and separate systems for search retrieval (OAI-SearchBot, PerplexityBot). The distinction matters because you can allow one and block the other.

If I block GPTBot, will ChatGPT stop mentioning my site?

Not immediately, and maybe never entirely. Blocking GPTBot affects future training crawls; existing model knowledge persists, and ChatGPT search uses OAI-SearchBot — a different rule. Mentions of your brand on other sites feed models regardless.

What's the difference between an LLM and AI search?

The LLM is the language engine; AI search is an application that bolts retrieval onto it — fetch pages, then have the model write a grounded, cited answer. AI Overviews and Perplexity are retrieval applications; the base model alone is not a search engine.

Can I optimize content for LLM training?

Only indirectly and slowly: broad, consistent brand mentions across crawled sites, canonical definitions others copy, entity clarity. There's no submission queue and no fast feedback loop — which is why most practical GEO work targets retrieval instead.

Do LLMs "index" my site?

Training bakes patterns into model weights — nothing like an index you can query or update. Retrieval systems do keep proper indexes (OpenAI's, Perplexity's, Google's). Only the second kind gives you a feedback loop you can work with.