Data Cleaning

September 2026 · 9 min read

Data cleaning makes the meaning and quality of each observation explicit. Before calculating a mean or fitting a model, establish what a row represents, what its fields mean and which transformations are supported by the source. The result should include usable records and an explanation of the unresolved ones.

This series follows five common decisions: parsing fields, reconciling units, resolving duplicate records, normalizing text and investigating unusual values. The chapters use small examples, diagrams and optional Python/C++ implementations so you can understand the decision before studying the code.

Start with the unit of observation

Suppose the prediction task concerns individual shipment events. A customer may have many shipments, and a shipment may appear in several source exports. State that one model observation represents one event before choosing a duplicate key or counting how many observations you have.

Also define the meaning of each column. The string “0017” can be a customer identifier whose zeros matter. A length needs a unit. A date needs a format and, for timestamps, a time-zone interpretation. A shipping status needs a documented set of accepted values.

Follow two source records toward one comparable representation

Here two exports claim the same event E17. Assume their event IDs are reliable and all other required business fields agree. Their raw length and status fields differ in representation:

The canonical values do not replace the source evidence. Retain both imported record IDs and map them to the chosen event record. If the sources disagree after conversion, preserve the conflict for investigation. A third row for event E18 should remain a separate observation even if it belongs to customer “0017”.

This example explains why cleaning steps have dependencies. A unit mismatch can look like a numeric conflict; a text-encoding difference can prevent a valid match. Conversely, overly broad normalization can make genuinely different records look identical.

Read by the decision you need to make

Read parsing and units first if you are building a new numeric dataset. Read text normalization before constructing text-based matching keys. For statistical scoring details after the final chapter, continue to the separate outlier-method series.

Distinguish a format rule from a correction and a modeling choice

A format rule follows the source contract: “these lengths are centimeters,” or “this field uses YYYY-MM-DD.” Applying it consistently establishes the representation. It does not require fitting a statistic to the dataset.

A correction uses evidence that a recorded value is wrong and supplies a supported replacement. Keep the original, the corrected value and the reason. Changing an extreme value to the average would not establish what was actually measured.

A modeling treatment changes how the algorithm handles data, such as imputing a missing feature or capping a valid extreme. These choices belong in the evaluation of the complete model. The missing-value chapters and Remove vs Keep examine them in detail.

Preserve evidence while keeping learned choices inside training

Use group splits when evaluating new entities and time-based splits when evaluating future events. Copies of the same observation must not provide apparent independence across the boundary. The pipeline chapter shows how to keep fitted transformations attached to the model.

Measure what cleaning changes about the population

Record counts before and after each operation, using a clear unit: imported rows, unique events or unique entities. These are different denominators. Also track failure reasons; a missing field, an invalid field and a confirmed correction should remain distinguishable.

Suppose 1,000 distinct incoming events produce 960 usable predictions and 40 rejected or unresolved cases. Coverage is 96%. Report that coverage alongside model performance on the accepted events. Check whether particular sources or subgroups account for a disproportionate share of the excluded observations.

A sudden rise in parse failures may signal a changed source format. A rise in apparent duplicates may signal a join problem. A shift in unusual-value flags may reflect changed operating conditions. Preserve enough provenance to investigate those possibilities before changing the cleaning policy.

Each chapter’s worked examples test the transformations and their limits. For your dataset, add cases that should be accepted, cases that should remain distinct and cases that must stay unresolved. That makes the intended behavior reviewable as the data evolves.

References and further reading

• Pandas: read_csv — explicit source-format, dtype and missing-marker controls.
• Unicode Standard Annex #15 — the equivalences and information changes defined by Unicode normalization.
• NIST: detection of outliers — distinguishing unusual observations, confirmed errors and appropriate treatment.
• Scikit-learn: common pitfalls — consistent preprocessing and the boundary between fitting and evaluation.