Fixing Unicode normalization mismatches in NFC and NFD

The Unicode standard defines multiple canonical representations for characters that can be expressed as a base character plus one or more combining marks. The letter ñ can be the single code point U+00F1 (precomposed) or the sequence U+006E + U+0303 (Latin small letter n followed by combining tilde). Both are valid Unicode and both look identical. The choice between them is a normalization form.

Unicode defines four normalization forms. NFC (Canonical Decomposition followed by Canonical Composition) produces precomposed characters where possible — it is the compact form and the standard for web content, HTTP headers, and most databases. NFD (Canonical Decomposition) decomposes all characters to their base plus combining marks — it is the form that macOS HFS+ and APFS use for filenames. NFKC and NFKD additionally apply compatibility mappings, collapsing visually similar characters like the fraction one-half to '1/2'. NFKC is used by some search engines and identifier parsers where compatibility equivalence matters more than round-trip fidelity.

The practical problem is that these forms are not equal under byte comparison. When a macOS user creates a file named 'résumé.pdf' and a Linux service tries to find it by the same NFC string, the comparison fails because macOS returned an NFD filename. Neither is wrong from a Unicode perspective, but the application has no normalization layer to reconcile them.

Text pasted from different sources is another normalization mismatch source. Text copied from a PDF may be in NFD if the PDF renderer decomposed characters. Text typed on a French keyboard on Windows is in NFC. Text entered via iOS keyboard may differ from macOS. When users paste a string into a search field that is compared against database-stored NFC values, the search finds nothing if the pasted text happens to be NFD.

The most direct diagnostic is to print the code point sequence of the string rather than the rendered text. In JavaScript: [...str].map(c => 'U+' + c.codePointAt(0).toString(16).toUpperCase().padStart(4,'0')).join(' '). An NFC é prints as U+00E9. An NFD é prints as U+0065 U+0301. If you see sequences of base character plus combining mark (U+0300 through U+036F range) where you expected precomposed characters, the string is in NFD or has been partially decomposed.

In Python 3, use import unicodedata; [unicodedata.name(c) for c in s] to print the Unicode name of each code point. This shows 'COMBINING ACUTE ACCENT' when a combining mark is present. The unicodedata.is_normalized('NFC', s) function (Python 3.8+) returns True or False without modifying the string, which is useful for logging or assertions.

For database comparison issues, run the comparison directly in SQL. In PostgreSQL with ICU: SELECT 'e\u0301' = 'é' COLLATE "und-x-icu". The und-x-icu collation performs Unicode canonical equivalence comparison. The default C and POSIX collations are binary and return false for NFD vs NFC comparisons.

For macOS vs Linux file path issues, print the hex bytes of the filename. In Bash: ls uploads/ | xxd | head. On macOS you will see decomposed byte sequences for accented characters (65 cc 81 for NFD é). On Linux you will see precomposed bytes (c3 a9 for NFC é in UTF-8). This confirms the platform-specific normalization behavior.

The correct fix is to normalize all strings to NFC at the points where external input enters your system. This means normalizing at form submission handlers, at file upload processing, at API ingestion points, and at search query preprocessing. Once data is normalized to NFC at entry, all downstream comparison and storage operations work correctly without further normalization overhead.

In JavaScript use str.normalize('NFC') which is available in all modern browsers and Node.js 0.12+. For filenames from the filesystem, normalize after readdir: files.map(f => f.normalize('NFC')). For search queries, normalize the query and also ensure stored values were normalized at write time. For user-facing input, normalize in a centralized input handler or Express middleware.

In Python 3 use unicodedata.normalize('NFC', s). For file operations normalize paths: unicodedata.normalize('NFC', filename) before constructing the full path. For Django and Flask, apply normalization in form validation or a request middleware class.

For database storage, the safest approach combines application-level NFC normalization with a database collation that performs canonical equivalence. In PostgreSQL with ICU collation, two canonically equivalent strings compare as equal even without application normalization, providing a safety net. In MySQL, utf8mb4_unicode_ci performs case-insensitive Unicode comparison but may not perform full canonical normalization in all versions. Treat application-level normalization as the primary safeguard and the database collation as a secondary check.

For the macOS and Linux cross-platform file sync problem, normalize filenames to NFC before any filesystem operations that compare against user-provided or database-stored values. When building paths from user input or database values always normalize: path.join(uploadDir, filename.normalize('NFC')).

NFKC normalization can cause unexpected data loss in some contexts. NFKC maps compatibility equivalents: the copyright symbol (U+00A9) becomes '(c)', the non-breaking space (U+00A0) becomes a regular space, the subscript 2 in chemical formulas (U+2082) becomes the digit '2', and ligatures like 'fi' (U+FB01) are split into two letters. If your application uses these characters intentionally — in a chemistry editor, a legal document tool, or a typography system — applying NFKC would corrupt the data. Use NFC for storage and NFKC only for search normalization where compatibility equivalence is desirable.

Strings that are already in NFC form do not change when normalize('NFC') is called. This makes NFC normalization idempotent and safe to apply repeatedly. However, NFC normalization is not fully symmetrical with NFD: normalizing an NFC string to NFD and back to NFC should round-trip correctly, but some code points have normalization exclusions that prevent re-composition. These affect a small number of Korean Hangul characters and certain compatibility characters documented in the Unicode standard's normalization exclusions table.

Some scripts do not have precomposed forms and are always decomposed in canonical form. Arabic and Hebrew combining vowel marks, Devanagari vowel signs, and Thai characters that combine with consonants do not have single precomposed code points. For these scripts NFC and NFD produce identical output, so normalization form differences cannot cause comparison failures. NFKC may still differ from NFC for some compatibility characters in these ranges.

Password normalization requires special care under the PRECIS framework (RFC 8264). The SASLprep profile (used in SASL and many authentication protocols) applies NFC normalization to passwords. If a user sets a password on a macOS device that produces NFD characters and then authenticates from a Linux device that produces NFC input, both must normalize to NFC before hashing to produce the same hash. Failure to normalize at both password setting and verification time causes intermittent authentication failures on cross-platform deployments.

Applying normalization at comparison time rather than at write time is the most common mistake. If normalization is only applied during lookup but not during storage, the database accumulates both NFC and NFD forms. Future comparisons that apply normalization at query time will match both forms, but direct SQL WHERE clauses without normalization will miss NFD entries. The correct approach is to normalize at the earliest opportunity: when data enters the system through API, form, or file upload.

Using the wrong normalization form for the use case is another common mistake. Using NFD for storage instead of NFC causes problems when text is processed by tools that do not normalize before comparison, because most tools default to binary or NFC comparison. NFC is the web standard: W3C recommends NFC for all web content, and most databases, HTTP headers, and JSON parsers assume NFC. NFD is only appropriate when interoperating with macOS filesystem APIs or other systems that specifically produce NFD output.

Assuming that normalize() handles all equivalence cases is incorrect. Unicode canonical equivalence handled by NFC and NFD and compatibility equivalence handled by NFKC and NFKD are the two defined Unicode equivalence relations. But there are other levels that normalize() does not address: case equivalence (A vs a), Unicode confusables (l vs I vs 1), variant selectors, and ligature decompositions. For security-sensitive comparisons like usernames, also apply case folding and confusable character detection as defined in Unicode Technical Standard 39.

Not testing with actual non-ASCII input that includes combining marks is why these bugs survive code review. Add test cases using café, résumé, naïve, Zürich, and names from Arabic, Hebrew, Thai, and Korean to your test suite. Each of these will expose normalization, length, sorting, and slicing bugs that ASCII-only tests miss entirely.

Establish a single normalization policy for your entire application and document it. The recommended policy is: normalize all text to NFC at input boundaries, store NFC in the database, compare NFC strings directly. This means every string in the system is in a known canonical form and comparisons are reliable without additional runtime normalization overhead.

Create a utility function that enforces normalization and apply it in middleware: function normalizeText(str) { return typeof str === 'string' ? str.normalize('NFC') : str; }. In Express, apply this in a body parser middleware. In Django or Flask, apply it in a form clean method or a custom middleware class. For REST APIs, apply normalization in the request deserialization layer before any business logic runs.

For search functionality, apply NFKC additionally for compatibility equivalence: query.normalize('NFKC').toLowerCase(). This handles both normalization and basic case-folding, so a search for 'cafe' finds 'café' and 'CAFÉ'. Store the normalized search form in a separate indexed column if you need high-performance lookup, and the original NFC form in the display column.

For file system operations on cross-platform codebases, normalize all paths that originate from user input or database retrieval to NFC before any filesystem operation. Do not normalize paths that come directly from fs.readdir unless you are comparing them against user-provided values. For internationalized applications, integrate with the ICU normalization library through the full-icu Node.js build which includes ICU's normalization, collation, and case folding algorithms, handling all edge cases including normalization exclusions and Hangul composition rules correctly.