Non-ASCII URL Characters

Element Code: UR-004

What counts as a non-ASCII URL character

ASCII is the original 128 character set: unaccented a to z, A to Z, digits, and a short list of punctuation. A URL path such as /cafe/ is pure ASCII. The moment you introduce an accented letter like the e in /café/, a non Latin script like /书/, a space, or a symbol, you are outside ASCII, and the character has to be represented some other way to travel safely across the web.

RFC 3986, the URI standard, only permits a limited set of characters in a raw URL. Anything else must be percent encoded in the path and query, or converted to Punycode in the domain (that is what turns an internationalized domain name into the xn-- form). So the accented cafe path becomes /caf%C3%A9/ once the e with an acute accent is UTF-8 encoded and percent escaped. Same destination, two very different looking strings.

Why this actually bites you

Modern browsers are polite. They show you /café/ in the address bar for readability, then quietly send /caf%C3%A9/ over the wire. That politeness hides a mess underneath, and the mess is where SEO problems live.

• Duplicate URLs: if your server treats the raw and the encoded form as two valid pages, you now have two URLs for one piece of content, splitting signals and inviting canonical confusion.
• Broken inbound links: a person copying a raw character URL into an email, a CMS field, or another platform can produce a string that double encodes (%25C3%25A9) or mangles, and the link 404s.
• Inconsistent internal links: half your templates link to the raw form and half to the encoded form, so crawlers waste budget fetching both.
• Analytics fragmentation: the same page shows up under multiple path spellings in your reports, and nobody trusts the numbers.
• Homograph and trust smell: mixed script or unusual encoded paths can look sketchy to users and, in extreme cases, resemble spoofing patterns.

To be fair: Google explicitly says it can crawl and index IDNs and encoded URLs, and non Latin slugs can help users in their own language recognize the link. This is not "never use accents." It is "pick one encoded form, serve it consistently, and stop the duplicates."

Raw versus encoded, side by side

Fixing it, step by step

1. Pick your canonical form. For most sites, decide whether readable non Latin slugs matter for your audience. If yes, keep them but serve the consistently percent encoded UTF-8 version. If they add nothing, transliterate to ASCII (cafe, resume) and be done with the whole class of problems.
2. Standardize encoding to UTF-8. Percent encoding is only unambiguous if everyone agrees the underlying bytes are UTF-8. Confirm your CMS, server, and templates all emit UTF-8. Legacy Latin-1 encodings produce a different byte sequence and therefore a different, broken URL.
3. 301 redirect the variants. Map the raw form, any alternate encodings, and case variants to the single canonical encoded URL with permanent redirects. Test that a double encoded request does not resolve to a live page.
4. Set the canonical tag. Every page should self reference the exact canonical encoded URL in its rel=canonical, matching character for character what you redirect to.
5. Fix internal links at the source. Update templates, navigation, and sitemaps to output the canonical encoded form. The XML sitemap in particular must use encoded URLs per the sitemap spec.
6. Handle IDNs at the domain. If the non-ASCII is in the hostname, that is Punycode territory. Register and serve the xn-- form and make sure your certificate covers it.

Encoding quick reference

How to detect it

1. Screaming Frog: crawl and use the URL tab filter for non ASCII characters, plus the "Underscores" and "Uppercase" reports for related hygiene issues. It flags the exact offending URLs.
2. Search Console: the Pages report and URL Inspection will show whether Google indexed the encoded, the raw, or both. Look for the same content under two spellings.
3. Server logs: grep access logs for both the raw multibyte sequence and the percent encoded form to see which one real traffic and bots actually request.
4. Sitebulb: its URL hints call out non ASCII and encoding issues per template so you can fix the pattern, not just one page.

Do this / skip that

What good looks like

Every version of the page resolves to a single canonical URL in one encoding. Internal links, the sitemap, and the canonical tag all spell it the same way, byte for byte. Search Console shows one indexed URL, not a pair. Users see a readable address in the browser, and the machine sees a clean, standards compliant string underneath. If you went the transliteration route, even simpler: the whole class of encoding bugs just cannot happen.

FAQ