Binary to Text Encoding: How Base64, Hex, and MIME Work

Modern computer systems store and process data as bytes — values from 0 to 255. However, many of the protocols and file formats that move data between systems were designed for text, not binary. SMTP email, JSON, XML, HTML attributes, HTTP headers, and many database text column types can only reliably handle printable ASCII characters (0x20 to 0x7E, approximately characters 32-126) or a well-defined Unicode encoding like UTF-8. Bytes outside these ranges may be silently dropped, modified, or rejected entirely.

The problem is particularly acute for SMTP email. The original SMTP specification (RFC 821) required all message content to be 7-bit ASCII. Although the 8BITMIME extension (RFC 6152) has been widely supported since the 1990s, not every mail relay, spam filter, or legacy mail server in a message's path guarantees that 8-bit bytes will pass through unchanged. Line-ending normalization, dot-stuffing, and maximum line-length enforcement can all corrupt binary data that was not pre-encoded.

JSON has a different constraint: string values must contain valid Unicode sequences, not arbitrary bytes. A JPEG image file contains many byte sequences that are not valid UTF-8. Attempting to embed raw image bytes in a JSON string causes the JSON serializer to emit Unicode replacement characters (U+FFFD) for invalid sequences, corrupting the data silently. The receiver decodes what appears to be a valid JSON string but gets a different byte sequence than the original image.

HTTP headers are restricted to ISO 8859-1 characters with further restrictions on control characters. HTTP/2 headers use HPACK compression and require valid UTF-8. Neither supports arbitrary binary. This means authentication tokens, encryption keys, and other binary values that need to appear in HTTP headers must be encoded as base64 or hex before placement.

When a file arrives corrupted after email transmission, the most reliable diagnostic is to compare the SHA-256 hash of the attachment as sent with the hash of the file as received. If the hashes differ, the binary data was modified in transit. Open the raw MIME source of the email (most email clients have a Show Original option) and look for the Content-Transfer-Encoding header of the attachment part. If it is missing, or set to '7bit' or '8bit' instead of 'base64', the binary data was sent without encoding and was subject to modification.

For corrupted binary data in API responses, log the raw HTTP response body before any JSON parsing. Check whether the Content-Type response header is 'application/json' or 'application/octet-stream'. If JSON is expected but the body contains binary bytes, inspect the byte at offset where corruption begins. If you see EF BF BD (the UTF-8 encoding of U+FFFD), the server performed an incorrect UTF-8 conversion on binary data before serializing to JSON.

For data URIs not rendering images, verify the MIME type prefix matches the actual file format. data:image/jpeg;base64,... must contain valid base64-encoded JPEG bytes. A common mistake is using data:image/png for a JPEG file, which some browsers refuse to render. Also check for whitespace or newline characters in the base64 string — data URI parsers in some browsers reject base64 with embedded newlines, even though MIME base64 requires them.

To quickly validate a base64 string, check three conditions: the length is a multiple of 4 (including = padding), all characters are in the set A-Za-z0-9+/=, and the = padding appears only at the end as 0, 1, or 2 characters. For base64url, the valid characters are A-Za-z0-9-_ with no padding. Any deviation from these rules means the string is malformed.

For MIME email attachments in Node.js, use the nodemailer library which handles Content-Transfer-Encoding: base64 and the 76-character line wrapping automatically when you provide an attachment as a buffer or file path. If you are constructing raw MIME manually, encode with buffer.toString('base64') and then wrap the result: base64Str.match(/.{1,76}/g).join('\r\n'). Always use CRLF (\r\n) as the line separator in MIME, not LF alone, per RFC 2045.

For embedding binary data in JSON, use buffer.toString('base64') on the Node.js side and Buffer.from(str, 'base64') to decode. Include an 'encoding' field in your JSON schema that declares the encoding used, so that future maintainers do not need to reverse-engineer whether a field is hex, base64, or UTF-8. A clear contract looks like: { "data": "...", "encoding": "base64", "mimeType": "image/png" }.

For data URIs in HTML and CSS, construct the URI as: 'data:' + mimeType + ';base64,' + buffer.toString('base64'). Data URIs have practical size limits in browsers — Chrome has approximately a 2 MB limit for img src data URIs. For larger images, use object URLs created with URL.createObjectURL(blob) instead. Reserve data URIs for small icons and inline SVGs that are used on every page load, saving one HTTP request per resource.

For UUencoding (the historical predecessor to MIME base64, used in USENET and early email), modern systems should not generate new UUencoded content. If you receive UUencoded files, most Unix systems include the uudecode utility, and Node.js has npm packages for legacy UUencode support. The UUencode alphabet maps 6-bit values to characters starting at ASCII 32 (space), producing output that is visually similar to random text and harder to debug than base64.

MIME base64 (RFC 2045) requires output to be split into lines of at most 76 characters. This is distinct from standard base64 encoding, which produces no line breaks. Some MIME parsers reject base64 with lines longer than 76 characters. The safe approach is always to include line breaks when generating MIME base64, and to strip all whitespace when decoding any base64 string that might have come from a MIME source. Node.js Buffer.from(str, 'base64') already ignores whitespace; browser atob() does not and will throw on strings containing newlines.

When streaming large binary data through a base64 encoder, you must buffer input in multiples of 3 bytes to avoid incorrect padding mid-stream. Encoding a 1024-byte chunk produces a 1368-byte base64 string with a trailing = because 1024 mod 3 = 2. If the next chunk starts immediately after this =, the concatenated base64 is invalid because padding should only appear at the end of the complete base64 string, not at chunk boundaries. The correct approach is to accumulate input bytes and only encode when you have complete 3-byte groups, flushing any remainder only when the stream ends.

Data URIs are cached by the browser as part of the document, not as separate cacheable resources. The same image embedded as a data URI on 100 different pages will be decoded and stored 100 times in memory, whereas a linked image URL would be cached once via the HTTP cache. For performance, prefer linked image URLs for any image used more than once, and reserve data URIs for single-use icons or dynamically generated images.

URL length limits also affect base64 in URLs. While the HTTP specification does not define a URL length limit, practical limits exist: many web servers reject URLs longer than 8192 characters, and AWS CloudFront has a 20,480-character URL limit. A 1 KB binary value encoded as base64url produces approximately 1368 characters, safely within most limits. Anything larger should use POST request bodies instead of URL parameters.

Treating base64 encoding as encryption is a common and dangerous misunderstanding. Base64 is a lossless encoding, not encryption. Anyone who receives a base64 string can decode it instantly — base64 provides no confidentiality. A common mistake is encoding a user password in base64 before storing it in a database, believing it provides some protection. It does not. Use bcrypt or Argon2id for password storage.

Using quoted-printable encoding for binary attachments is another mistake. Quoted-printable is designed for text content that is mostly ASCII with occasional non-ASCII characters. It represents non-ASCII bytes as =XX hex escapes. For binary data, almost every byte triggers an escape, producing output that is larger than base64 and barely readable. Always use base64 for binary attachments in MIME email.

Not handling the 76-character MIME line limit is a frequently encountered bug in custom email generation. A developer generates correct base64 output but does not add line breaks. The resulting email works with Gmail and Outlook (lenient decoders) but corrupts attachments when relayed through older or stricter SMTP servers. Always add 76-character line wrapping to base64 content in MIME messages.

Double-encoding is a common bug in multi-layer systems. If middleware base64-encodes an already-base64-encoded value, the output looks like valid base64 but decodes to another base64 string rather than the original data. Debug by checking the length: a 32-byte input should produce 44 base64 chars after one encoding. If you see 60 or more chars, it has likely been double-encoded. Establish a clear data contract where encoding is applied exactly once at the serialization boundary.

Choose the right encoding for each channel and document it explicitly. For HTTP APIs: base64 in JSON body fields, with an explicit 'encoding' metadata field. For URLs and HTTP headers: base64url with no +, /, or = characters. For MIME email: base64 with 76-character line wrapping and CRLF. For human-readable checksums: hex. For binary columns in PostgreSQL: no encoding needed — use BYTEA and let the ORM or database driver handle serialization transparently.

Never apply base64 encoding more than once to the same data. Double encoding is a common bug in systems where multiple layers each independently decide to safely encode data. Establish a clear data contract: encoding is applied exactly once at the serialization boundary (when writing to disk, network, or database) and removed exactly once at the deserialization boundary.

For large binary files such as images, videos, and compressed archives, avoid base64 entirely when possible. Use multipart/form-data for file uploads, which sends binary data in its native form with MIME boundaries and no encoding overhead. Use pre-signed URL patterns (AWS S3 pre-signed URLs, Google Cloud Storage signed URLs) for large file downloads to avoid proxying binary data through your application server. Reserve base64 for values that are small enough (under 64 KB) that the 33% overhead is acceptable.

Test your encoding implementation with edge cases: zero-length input (should produce an empty string), inputs of exactly 1, 2, and 3 bytes (tests all three base64 padding scenarios), and inputs containing all 256 possible byte values (stress-tests the character mapping). For MIME, also test with input lengths that are not multiples of 57 bytes, since 57 bytes produce exactly 76 base64 characters — the standard MIME line length.