Diagnosing and fixing UTF-8 vs UTF-16 encoding mismatches

UTF-8 and UTF-16 encode the same Unicode code points using fundamentally different byte representations. UTF-8 uses 1-4 bytes per code point, with ASCII characters occupying exactly one byte. UTF-16 uses 2 bytes for code points in the Basic Multilingual Plane (U+0000 to U+FFFF) and 4 bytes for code points above U+FFFF. When a program reads UTF-16 bytes and interprets them as UTF-8, the result is mojibake because the null bytes that pad ASCII characters in UTF-16 (H in UTF-16 LE is 48 00) are treated as the NUL character in UTF-8.

The Byte Order Mark (BOM) is a mechanism to identify the encoding of a file. A UTF-16 LE file begins with the bytes FF FE, UTF-16 BE with FE FF, and optionally UTF-8 with EF BB BF. When a UTF-16 file without a BOM is read, the program must guess the encoding based on the content, which may fail for short files or files containing only ASCII characters. Windows applications including Notepad, Visual Studio, and Excel frequently produce UTF-16 LE files with BOM when saving non-ASCII content.

In JavaScript, the problem is compounded by the language's internal string representation. JavaScript strings are sequences of 16-bit code units (UTF-16). The length property counts code units, not Unicode code points or visible characters. The charCodeAt method returns a 16-bit code unit value, which is the high surrogate value (in the range 0xD800-0xDBFF) for the first half of a surrogate pair, not the actual code point. codePointAt was introduced in ES6 to provide the correct code point value by combining surrogate pairs.

In Node.js, the Buffer object's encoding argument controls how bytes are interpreted. Buffer.from(str, 'utf8') and Buffer.from(str, 'utf16le') produce completely different byte sequences for the same string. Using the wrong encoding to decode a Buffer produces garbled output. The 'binary' encoding (alias for 'latin1') treats each byte as a single character, which produces incorrect results for any character above U+00FF and should not be used for Unicode text.

The most reliable way to detect encoding is to inspect the BOM bytes. In Node.js: const raw = fs.readFileSync(path); console.log(raw.slice(0, 4).toString('hex')). Compare the first bytes: ef bb bf is a UTF-8 BOM, ff fe is UTF-16 LE, fe ff is UTF-16 BE. If there is no BOM, look at the pattern of bytes for text content: a UTF-16 LE ASCII text file will show alternating content bytes and null bytes (48 00 65 00 = 'He'), while UTF-8 will show only content bytes without nulls.

In Python, the chardet library provides encoding detection: import chardet; result = chardet.detect(open('file', 'rb').read(10000)); print(result). For short files chardet can be wrong, so for production code where the encoding source is known (such as files from a specific Windows application) it is better to document and hard-code the encoding assumption rather than auto-detecting.

For JavaScript string inspection, print the code units: Array.from({length: str.length}, (_, i) => str.charCodeAt(i).toString(16)).join(' '). If you see values in the range d800-dbff followed by dc00-dfff, those are surrogate pairs for non-BMP characters. If you see many 0000 values, the string may have been constructed from UTF-16 bytes with null padding.

For HTTP responses, the encoding is declared in the Content-Type header: Content-Type: text/html; charset=UTF-16. The fetch API's response.text() method reads the charset from the Content-Type header and decodes accordingly. If the header is wrong or missing, response.text() defaults to UTF-8. Axios similarly reads charset but may handle edge cases differently. Always check that the Content-Type of the response matches your decoding assumption.

In Node.js, always pass the explicit encoding to Buffer and fs methods. Use fs.readFileSync(path, 'utf8') for UTF-8 files. For UTF-16 LE files, read as a Buffer first and convert: Buffer.from(rawBuffer.buffer, rawBuffer.byteOffset, rawBuffer.byteLength).toString('utf16le'). The iconv-lite package provides broader encoding support including UTF-16 BE, which Node.js's built-in 'utf16le' does not handle: require('iconv-lite').decode(rawBuffer, 'UTF-16BE').

In Python 3, always pass the encoding parameter to open(): open('file.txt', 'r', encoding='utf-8'). For files that may have a BOM, use the special encoding name 'utf-8-sig' which automatically strips the UTF-8 BOM if present. For UTF-16 files, use encoding='utf-16' which auto-detects the byte order from the BOM, or 'utf-16-le' or 'utf-16-be' for explicit byte order. The io.open function (which is the same as the built-in open in Python 3) supports these encoding names.

For JavaScript string operations on non-BMP characters, replace charCodeAt with codePointAt and use the for...of loop or Array.from for iteration. When building a Buffer from a JavaScript string that may contain emoji or other non-BMP characters, use Buffer.from(str, 'utf8') rather than manually iterating with charCodeAt.

For converting a UTF-16 file to UTF-8 on the command line, use iconv: iconv -f utf-16le -t utf-8 input.txt > output.txt. On macOS, iconv is built in. On Linux it is available via libiconv. For batch conversion, find all UTF-16 files and convert: find . -name '*.txt' | xargs -I{} sh -c 'iconv -f utf-16le -t utf-8 "{}" > "{}.utf8" && mv "{}.utf8" "{}".

The UTF-8 BOM (EF BB BF) is a frequently misunderstood edge case. While it is technically valid Unicode, it is not recommended for UTF-8 by the Unicode standard because UTF-8 has no byte order ambiguity. However, many Windows tools insert it, and many Unix tools do not expect it. A JSON file with a UTF-8 BOM will fail to parse in Python with json.load(open('file.json')) because the JSON parser sees a non-whitespace character before the first { or [. The fix is to use encoding='utf-8-sig' in the open() call, or to explicitly strip the BOM: text.lstrip('\ufeff').

Unpaired surrogates are another edge case. A valid UTF-16 string must pair each high surrogate (U+D800-U+DBFF) with a following low surrogate (U+DC00-U+DFFF). However, JavaScript allows strings to contain unpaired surrogates, which are called lone surrogates. Functions like encodeURIComponent throw a URIError when called with a string containing a lone surrogate. TextEncoder throws for lone surrogates in some environments. JSON.stringify produces a malformed JSON string for a string with lone surrogates. If your code receives strings from external sources, validate that they contain no lone surrogates before processing.

Windows Registry exports (.reg files) are saved as UTF-16 LE by default. When parsing .reg files programmatically in Node.js or Python, always specify the UTF-16 LE encoding. The same applies to Windows event log exports (.evtx files), Windows INF driver files, and many configuration files from Windows applications. These file formats predate the UTF-8 ubiquity and assume a UTF-16 reading environment.

Node.js's http.IncomingMessage (for incoming requests) and the fetch API's Response object both expose the response body as UTF-8 via toString() and text() respectively. If the server returns UTF-16 encoded content without a Content-Type charset declaration, the client will decode it as UTF-8 and produce garbage. The solution is to use the arrayBuffer() method to get the raw bytes and then decode manually with TextDecoder: new TextDecoder('utf-16le').decode(await response.arrayBuffer()).

Using Buffer.from(str, 'binary') for text encoding is a common mistake that produces silent corruption. The 'binary' encoding (same as 'latin1') treats each character code as a single byte. For characters with code points above 255, it truncates to the lower 8 bits: the character é (code point 233) round-trips correctly, but a Chinese character (code point 20013 for 中) would be truncated to 0xED (109), losing data. Use 'utf8' for Unicode text and 'binary' only for true binary data that was originally constructed as a binary string.

Not handling the case where response.text() decodes as UTF-8 when the server sends UTF-16 is a persistent API integration mistake. The fetch spec says response.text() uses the encoding from the Content-Type charset. If the server sets Content-Type: application/json; charset=UTF-16 and the client calls response.text(), the browser handles the charset correctly. But if the server omits the charset, the client defaults to UTF-8 and garbles the content. Always verify the Content-Type in the Network DevTools tab when debugging encoding issues with API responses.

Assuming that string.length equals the number of visible characters is the most common mistake in JavaScript Unicode handling. A string containing one emoji has length 2 (two UTF-16 code units in a surrogate pair). A string containing a flag emoji (two regional indicator characters) has length 4. A string containing a family emoji (four emoji joined by ZWJ) has length up to 11. Use Intl.Segmenter for accurate visible character counting, not length.

Passing non-UTF-8 bytes to TextDecoder without specifying the correct label causes silent or fatal failures depending on the fatal flag. new TextDecoder() uses UTF-8 with fatal: false by default, replacing invalid sequences with U+FFFD. new TextDecoder('utf-16le') decodes as UTF-16 LE. Always specify the encoding label explicitly and use fatal: true in contexts where invalid sequences indicate a data integrity problem.

Standardize on UTF-8 for all new code and convert legacy UTF-16 files during migration. UTF-8 is the dominant encoding on the web, in Linux systems, and in cloud environments. When receiving data from Windows systems or processing legacy files, add an explicit detection and conversion step at the boundary rather than spreading encoding awareness throughout the application.

In Python 3 applications, set the default encoding at process start: import sys; sys.stdin.reconfigure(encoding='utf-8'); sys.stdout.reconfigure(encoding='utf-8'). For production services on Linux, this is usually unnecessary because the locale default is UTF-8, but on Windows servers running Python the default encoding may be cp1252 or another Windows encoding. Setting it explicitly prevents the inconsistency.

For JavaScript applications that process text from multiple sources, create an encoding utility module that wraps TextDecoder and TextEncoder with appropriate error handling. Expose functions like decodeUtf8(buffer), decodeUtf16LE(buffer), encodeUtf8(str) that handle BOM detection and removal, lone surrogate validation, and consistent error reporting. This centralizes encoding logic and prevents it from being reimplemented inconsistently in different parts of the codebase.

When writing to files or network sockets that may be read by Windows applications, consider whether to include a UTF-8 BOM. Most modern Windows applications including Visual Studio Code handle UTF-8 without BOM correctly. Older applications may require the BOM to recognize the encoding. If you must interoperate with older Windows software, write the BOM explicitly: fs.writeFileSync(path, Buffer.concat([Buffer.from([0xEF, 0xBB, 0xBF]), Buffer.from(content, 'utf8')])).