Text Normalization

September 2026 · 14 min read

Text normalization chooses which differences a task should ignore. That might mean treating two Unicode encodings of “café” alike, accepting “IN_TRANSIT” as a known status, or matching text without regard to case. Each choice has consequences for what can still be distinguished afterward.

Keep the original text and derive a separate normalized field or matching key. We will first make invisible representation differences visible, then inspect operations that deliberately merge different strings.

Identical-looking text can have different stored characters

The final “é” in “café” can be stored as one Unicode code point, U+00E9, or as “e” followed by a combining acute accent, U+0065 and U+0301. A code point is a numbered element of Unicode text. A visible character can involve more than one code point.

The two words normally look alike, but their raw strings compare unequal. Their lengths also differ: Python counts four versus five code points; UTF-8 uses five versus six bytes. These counts measure the representation, not how many characters a reader perceives.

Choose the equivalence you actually need

Canonical normalization reconciles representations of the same abstract text. NFC composes where possible; NFD decomposes. Neither operation removes the accent from “café”.

Compatibility normalization also folds distinctions such as fullwidth letters, circled digits and superscripts. NFKC turns “ＭＬ” into “ML”, but it also turns “x²” into “x2”. Whether that is appropriate depends on the field and task.

The Unicode normalization specification defines these transformations. They are not spelling correction or a test that two records refer to the same entity.

See which distinctions each operation removes

Case folding creates keys for caseless matching. Python’s casefold() is more thorough than lowercasing: “Straße” becomes “strasse”, allowing comparison with “STRASSE”. It also makes “mW” and “MW” equal, although those unit symbols have different meanings.

Removing accents is a separate operation. The example below decomposes the text, removes characters in Unicode category Mn (nonspacing marks), then recomposes. This merges “café” and “cafe”. Nonspacing marks include more than Latin accents, so this rule can remove meaningful information in other scripts or presentation selectors too.

Call this a collision when different original strings map to the same key. Collisions can be intentional, but the key cannot reconstruct which original you started with. A normalized name is therefore evidence for record linkage, not a unique identity guarantee.

Python: keep the operations separate and audit their outputs

The functions below reproduce the four options in the comparison. The case-folding option applies NFC before and after folding so its output also has a consistent canonical form. The mark-removal function makes its broad character-category rule explicit.

import unicodedata as ud

NORMALIZERS = {
    "NFC": lambda text: ud.normalize("NFC", text),
    "NFKC": lambda text: ud.normalize("NFKC", text),
    "casefold": lambda text: ud.normalize("NFC", ud.normalize("NFC", text).casefold()),
}

def remove_nonspacing_marks(text):
    decomposed = ud.normalize("NFD", text)
    kept = "".join(c for c in decomposed if ud.category(c) != "Mn")
    return ud.normalize("NFC", kept)

NORMALIZERS["marks"] = remove_nonspacing_marks
pairs = [("café", "cafe"), ("Straße", "STRASSE"), ("①", "1"),
         ("ＭＬ", "ML"), ("mW", "MW"), ("x²", "x2")]
for name, normalize in NORMALIZERS.items():
    merged = [(a, b) for a, b in pairs if normalize(a) == normalize(b)]
    print(name, merged)

raw = ["caf\u00e9", "cafe\u0301"]
print("raw equal:", raw[0] == raw[1])
print("NFC equal:", ud.normalize("NFC", raw[0]) == ud.normalize("NFC", raw[1]))

NFC []
NFKC [('①', '1'), ('ＭＬ', 'ML'), ('x²', 'x2')]
casefold [('Straße', 'STRASSE'), ('mW', 'MW')]
marks [('café', 'cafe')]
raw equal: False
NFC equal: True

For a dataset audit, group distinct raw strings by their normalized key and inspect groups with more than one raw form. Record the normalization policy and Unicode-library version so training, indexing and serving apply compatible rules.

Whitespace and punctuation follow the field contract

For a controlled shipping-status field, “ IN TRANSIT ” might legitimately map to in_transit. For source code, indentation can carry meaning. For natural language, deleting punctuation can merge “can't” with “cant”. A cleanup rule that helps one field can damage another.

The next helper is deliberately for a small ASCII-only status vocabulary. It folds ASCII letter case, collapses ASCII whitespace and resolves two documented aliases. Unknown statuses remain errors to review. This is independent of Unicode normalization.

import re

ALIASES = {"in transit": "in_transit", "in_transit": "in_transit",
           "pending": "pending", "delivered": "delivered"}

def normalize_status_ascii(raw):
    if not raw.isascii():
        raise ValueError("this field accepts ASCII text only")
    key = re.sub(r"[ \t\r\n\f\v]+", " ", raw).strip(" ").lower()
    if key not in ALIASES:
        raise ValueError("unknown status")
    return ALIASES[key]

for raw in ["  IN\tTRANSIT ", "IN_TRANSIT", " Delivered "]:
    print(normalize_status_ascii(raw))

in_transit
in_transit
delivered

Optional C++: implement the same bounded status rule

This byte-oriented helper rejects non-ASCII input. Full Unicode normalization requires Unicode-aware algorithms and tables; use a maintained implementation such as ICU for that work.

#include <iostream>
#include <map>
#include <stdexcept>
#include <string>
#include <string_view>

std::string normalize_status_ascii(std::string_view raw) {
    std::string key;
    bool pending_space=false;
    for (unsigned char c:raw) {
        if (c>127) throw std::invalid_argument("ASCII text required");
        bool whitespace=c==' ' || c=='\t' || c=='\r' || c=='\n' || c=='\f' || c=='\v';
        if (whitespace) { pending_space=!key.empty(); continue; }
        if (pending_space) { key+=' '; pending_space=false; }
        key+=static_cast<char>(c>='A' && c<='Z'?c-'A'+'a':c);
    }
    static const std::map<std::string,std::string> aliases={
        {"in transit","in_transit"},{"in_transit","in_transit"},
        {"pending","pending"},{"delivered","delivered"}
    };
    auto found=aliases.find(key);
    if (found==aliases.end()) throw std::invalid_argument("unknown status");
    return found->second;
}
int main() {
    const char* samples[]={"  IN\tTRANSIT ","IN_TRANSIT"," Delivered "};
    for (auto raw:samples) std::cout << normalize_status_ascii(raw) << "\n";
}

in_transit
in_transit
delivered

Test both the intended matches and the distinctions to preserve

Normalization does not repair text decoded with the wrong character encoding. Fix decoding first and preserve the source needed to investigate it. Nor does normalization make all visually similar letters equal; Latin “a” and Cyrillic “а” remain different under NFC and NFKC.

For model input, follow the intended tokenizer and preprocessing contract, and evaluate any additional transformation on the actual task. For deduplication, treat the normalized field as a comparison feature whose collisions still require an identity decision.

References and further reading

• Unicode Standard Annex #15: Normalization Forms — canonical versus compatibility equivalence and the four normalization forms.
• Python: unicodedata — normalization, Unicode categories and the underlying database version.
• Python: str.casefold — caseless matching and the difference between case folding and lowercasing.
• ICU: normalization — Unicode-aware normalization implementations and normalization boundaries.