User-Agent

A user-agent is the HTTP header a client sends with every request to identify what it is — a browser, a search crawler, an AI bot, a monitoring script. It's the first thing your server learns about a visitor, and it drives real decisions: which robots.txt rules apply, what your bot management blocks, how your analytics segments traffic. Misread it and you block the crawler paying your bills, or welcome the scraper draining your server.

What a user-agent looks like on the wire

Every HTTP request carries the header. Here's a real request from a desktop Chrome browser:

GET /pricing/ HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36
            (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,...

That string is a fossil record. Every browser claims to be "Mozilla/5.0" because of compatibility sniffing wars from the 1990s; Chrome claims Safari's WebKit tokens; Edge appends its own on top of Chrome's. Nobody designed this — each layer was added so old sniffing code wouldn't break. Two consequences follow: parsing user-agents by substring is fragile, and the header is trivially forged. Anyone can send any string:

# Fetch your own site claiming to be Googlebot
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/

# Fetch it as GPTBot
curl -A "GPTBot/1.2" https://example.com/

That second point is why serious bot verification never stops at the UA — see the DNS verification walkthrough in our Googlebot entry.

The crawler strings that matter in 2026

Your access logs now carry three generations of bots: search crawlers, AI training crawlers, and AI assistants fetching on behalf of live users. Version numbers drift, but the identifying tokens are stable:

Which of these to allow is a business decision, not a technical default — our breakdown of which AI bots you're actually blocking walks through the tradeoffs, and the AI crawler map keeps the full census current.

How to check it on your own site

1. Census your logs. awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -30 ranks every user-agent hitting you by request count. Most site owners are surprised by what's in the top ten.
2. Isolate the AI cohort: grep -Ei "gptbot|claudebot|perplexitybot|oai-searchbot|ccbot|bytespider|chatgpt-user" access.log | wc -l — then break it down per bot and per URL to see what they're reading.
3. Test your robots.txt against specific tokens — fetch pages with curl -A using each bot's string and confirm your server and CDN rules behave as intended (robots.txt is advisory; your firewall is not).
4. Verify the heavy hitters aren't fakes. Reverse-DNS a sample of IPs claiming to be Googlebot or Bingbot; check published IP ranges for GPTBot and friends (each operator documents theirs).
5. Cross-check log reality against intent with log file analysis: if a bot you blocked in robots.txt still appears in logs, it either ignores the standard or you wrote the rule wrong — different problems, different fixes.

Common audit mistakes (and fixes)

Blocking by careless substring

A firewall rule matching bot blocks Googlebot, Bingbot, and your own uptime monitor in one stroke. I've watched exactly this take a site out of the index. Fix: match full tokens, test against the real strings above, and log-then-block rather than block-on-deploy.

Treating the user-agent as identity

UA strings are claims, not credentials. Analytics "Googlebot traffic," scraper allowlists, and cloaking logic built on unverified UAs are all built on sand. Fix: verify by DNS/IP for anything that grants privileges.

Confusing robots.txt tokens with header strings

The robots.txt User-agent: line matches the bot's short token (GPTBot), not the full header. Rules written against the full string silently match nothing. Fix: use the token column from the table; syntax details are in the robots.txt reference.

Assuming every bot obeys robots.txt

Reputable crawlers comply; plenty of scrapers and some training bots don't. Fix: robots.txt for the polite ones, rate limiting and IP-level blocks for the rest, and don't mistake publishing a rule for enforcing one.

Forgetting that browser UAs are frozen

Chrome reduced its user-agent years ago — OS versions and device details are now frozen or generic, with the detail moved to Client Hints (Sec-CH-UA headers). Device-detection logic that still parses UA minutiae misidentifies a growing share of visitors. Fix: feature-detect in JavaScript; use Client Hints server-side if you truly need device data.

FAQ

Can I trust the user-agent header?

For statistics, mostly. For security or privileges, never. It's a self-declared label with zero authentication — treat it the way you'd treat a visitor's name tag at a conference.

Should I block AI bots like GPTBot and ClaudeBot?

Depends what your content earns you. Blocking training bots keeps your content out of future models; it also keeps your brand out of the answers those models give. Blocking retrieval agents (ChatGPT-User, Perplexity's user fetcher) removes you from live AI answers where citations send actual traffic. Decide per bot class, not with one angry wildcard.

Why do all browsers say "Mozilla/5.0"?

Legacy compatibility. Servers in the 90s sent better content to Netscape ("Mozilla"), so every subsequent browser impersonated it, then impersonated each other. The prefix is meaningless today — it survives because removing it would break ancient sniffing code that still runs somewhere.

What user-agent should my own scripts send?

An honest, contactable one: MyCompanyAuditBot/1.0 (+https://example.com/bot-info). Ops teams block anonymous Python-requests defaults on sight; a UA with a URL and a purpose gets tolerated and gets you unblocked faster when something goes wrong.

Is the user-agent used for rankings?

No — it's plumbing, not a signal. Where it touches SEO is operational: serving the right content to the right crawler, keeping rendering parity between bots and users, and not accidentally locking the index out of your site with a bad bot rule.