Unicode, Encodings and Real-World Text

Decode, Normalise and Compare Text Without Data Loss

• Decode UTF-8 deliberately and report invalid byte sequences instead of silently discarding them.
• Compare visually similar strings after Unicode normalisation and case folding.
• Explain why a failed decode is data-quality evidence rather than an inconvenience to hide.
• Compare the raw bytes, decoded string and normalised form before deciding whether a record is safe to keep.

Characters, bytes and normal forms

The letter a user sees may be stored as one code point or as a base letter plus a combining mark. Those sequences can look identical while failing equality checks.

NFC is a sensible comparison form for many applications, but search, legal names and security identifiers need their own written policy.

A safe ingestion boundary

Decode once at the boundary, keep the original text, and derive a normalised comparison field. Never repeatedly encode and decode values throughout a pipeline.

Choose a comparison policy, not a universal cleanup recipe

Casefolding is broader than lowercasing and is useful for caseless matching. It can still remove distinctions that matter to display or identity.

Compatibility forms such as NFKC can collapse formatting variants. That may help catalogue search but can be too aggressive for evidence, passwords or source quotations.

Questions before changing text

Which system produced the bytes? Which characters must remain distinguishable? Must the original wording be reconstructed later?

Store both raw and derived forms when the answer is uncertain.

Build the Unicode, Encodings and Real-World Text example

The example creates a file with a decomposed accented character, so the encoding and normalisation problem is reproducible on any machine.

Use strict decoding during an audit because it points to the exact record that violates the expected encoding. A production importer can quarantine that record rather than lose its bytes.

from pathlib import Path
import unicodedata

path = Path('sample.txt')
path.write_bytes('Café, Straße, نص'.encode('utf-8'))
raw = path.read_text(encoding='utf-8', errors='strict')
normal = unicodedata.normalize('NFC', raw)
print(raw == normal)
print(normal.casefold())
print([hex(ord(ch)) for ch in raw[:5]])

Expected Decode and Normalisation Evidence

False
café, strasse, نص
['0x43', '0x61', '0x66', '0x65', '0x301']

The equality result is false before normalisation because the visible accent uses a separate combining code point. NFC composes it.

Casefolding changes Straße to strasse for comparison. Keep the original text for display.

Diagnose failures in Unicode, Encodings and Real-World Text

Start with the smallest failing byte or string. Print code points and the encoding assumption before changing the data.

Check your Unicode, Encodings and Real-World Text reasoning

Test your Unicode decisions on byte decoding, normal forms and comparison policy before revealing the explanations.

Primary references for Unicode, Encodings and Real-World Text

• Unicode Standard Annex #29: standard rules for Unicode text boundaries.
• Python Unicode HOWTO: official guidance for strings, encodings and error handling.

Recheck this lesson when Python changes its Unicode database or codec behaviour. Save the Python and Unicode versions with the multilingual audit.