The Complete robots.txt Reference: Precedence, Wildcards, AI Bots & Real-World Receipts

Most robots.txt files are copied from a template, pasted once, and never understood. That is how a single line silently removes a site from half the web's crawlers, or, just as often, fails to block the thing the owner thought it blocked. This is the complete reference: what every directive does, the precedence rules almost nobody reads, real configurations pulled from live sites in June 2026, and the mistakes that quietly cost traffic.

📍 What robots.txt is, and the three things it is not

robots.txt is a plain-text file at the root of a host (https://example.com/robots.txt) that tells compliant crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. That is all it does. It is not:

• Not a security control. The file is public and advisory. Listing Disallow: /admin/ tells the world where your admin lives while stopping no determined actor. Protect with authentication, not robots.txt.
• Not a way to de-index. Disallowing a URL stops crawling, but Google can still index a disallowed URL it finds via links, showing it with no snippet. To remove a page from search, use a noindex meta tag or header (and leave it crawlable so Google can see it).
• Not a guarantee. Compliance is voluntary. Mainstream crawlers honour it; scrapers and some AI bots do not (covered below).

🔧 The anatomy: every directive that matters

A robots.txt file is one or more groups. Each group starts with one or more User-agent lines and contains Allow/Disallow rules. Sitemap is independent of groups.

User-agent: Googlebot # this group applies to Googlebot only
Disallow: /cart/ # block a folder
Allow: /cart/promo.html # …but allow one file inside it
Disallow: /*?sort= # wildcard: any URL containing ?sort=
Disallow: /*.pdf$ # $ anchors the end: only URLs ending .pdf

User-agent: * # everyone else
Disallow: /private/

Sitemap: https://example.com/sitemap.xml # absolute URL, group-independent

• User-agent, names the crawler the group applies to. * matches any bot that doesn't have its own group.
• Disallow, a path prefix the crawler must not request. Empty Disallow: means "allow everything."
• Allow, carves an exception out of a broader Disallow. Supported by Google and most majors.
• * and $, * matches any sequence of characters; $ anchors the end of the URL. Google supports both; some legacy crawlers ignore them entirely.
• Sitemap, an absolute URL to your XML sitemap. It is global, not tied to any user-agent group, and you can list several.

⚖️ The precedence rules almost nobody reads

This is where well-meaning files go wrong. Two separate precedence questions decide what a crawler actually does, and they resolve differently across crawlers.

Rule 1, User-agent groups: most specific wins, and it's winner-take-all

A crawler reads only the one group whose user-agent token is the longest match for its name. It does not merge groups. So a Googlebot group makes Googlebot ignore the * group completely, even the rules you assumed were universal.

Receipt (live, June 2026): LinkedIn, Yelp and Reuters all publish User-agent: * → Disallow: /. It looks like "block the entire internet." It isn't: each of them also defines dedicated Googlebot and Bingbot groups, so those engines crawl freely while everyone else is turned away. The * block never applies to Google because Google matched a more specific group first. LinkedIn even appends a note: apply for whitelisting by email.

Rule 2, Within a group: Google uses specificity, legacy crawlers use order

Once a crawler is inside its group and a URL matches several rules, who wins? Here the standards diverge, and this is the part that surprises people:

• Googlebot (and most modern majors): the most-specific rule wins, specificity measured by the number of characters in the path. Order in the file is irrelevant. If an Allow and a Disallow are exactly equally specific, Google breaks the tie toward the least restrictive rule (the Allow).
• Legacy / strict-RFC crawlers: many older or simpler parsers use first-match-wins. With those, the order of your Allow/Disallow lines changes the outcome.

So the honest answer to "does the order of rules in robots.txt matter?" is: not for Google, but yes for a meaningful slice of other crawlers. Write your file so it produces the correct result under both interpretations, put the specific Allow before the broad Disallow, and don't rely on Google's longest-match cleverness if you care who else obeys you.

# Correct under BOTH longest-match and first-match crawlers:
User-agent: *
Allow: /downloads/whitepaper.pdf # specific exception first
Disallow: /downloads/ # broad block second

🌐 Real configurations in the wild (June 2026)

Pulled live from production robots.txt files, these are observations, not value judgements. The point is what each does.

The AI-bot blocks above are usually deliberate (licensing posture). The trap is doing the same thing by accident, see the AI section below.

🚫 The mistakes that quietly cost you

🤖 robots.txt and AI crawlers

The fastest-growing reason to touch robots.txt in 2026 is AI. The critical distinction, covered in depth in The AI Crawler Map, is that AI bots do different jobs: some train models, some index pages to answer questions, some fetch a page live for a user. Blocking a training crawler costs almost no traffic; blocking a search/answer crawler removes you from that engine's citations. The publisher blocks in the table above are deliberate licensing decisions. The danger is replicating them by pasting a "block all AI" snippet you found online and silently deleting yourself from ChatGPT Search, Perplexity and Claude. The trap is reusing a broad "block all AI" snippet without checking which bots it actually covers, so a training block quietly takes out the search and answer crawlers too.

And remember: robots.txt is a request. Independent monitoring through 2025 reported that some AI operators fetched disallowed pages anyway. If you need a hard block, enforce it at the edge (WAF, verified IP ranges), not in a text file.

🧪 How to test it

1. Read the live file. Open yourdomain.com/robots.txt in a browser, that's exactly what crawlers see.
2. Inspect a URL in Search Console. The URL Inspection tool reports whether a specific URL is blocked by robots.txt for Googlebot.
3. Use a validator. Google's open-source robots.txt parser (the basis for its real behaviour) and several third-party testers will tell you whether a given URL/user-agent pair is allowed.
4. Confirm assets render. Use the URL Inspection "view rendered page" to ensure CSS/JS aren't blocked.

❓ robots.txt FAQ