← All Posts
Deep Learning · Data Preparation and Preprocessing · Data Cleaning

Units and Consistency

A measurement needs a number, a unit and a defined quantity. The number 170 might describe a length in centimeters, a duration in milliseconds or something else entirely. Successful numeric parsing establishes none of those meanings by itself.

We will reconcile three length readings, then use a square and a temperature example to understand why some conversions need more than one multiplication. The aim is a consistent feature representation that can be traced back to the original record.

Choose one unit for each quantity

Suppose three sources report the same length using different units. Choose meters as the canonical unit: the representation every valid length row will use downstream.

Source A: 1.7 mMultiply by 1.
Canonical record: 1.7 m.
Source B: 170 cmMultiply by 0.01.
Canonical record: 1.7 m.
Source C: 1700 mmMultiply by 0.001.
Canonical record: 1.7 m.
Three numerical representations of the same length. Comparing the raw numbers would invent a difference between these observations. After conversion, the feature values agree.

Keep the raw value and source unit alongside the converted value and its new unit. A column name such as length_m can make the storage contract visible, but it does not replace source metadata and validation.

Do this before feature scaling, distance calculations or summaries. Subtracting a mean and dividing by a standard deviation cannot repair a mixture in which some rows mean meters and others mean centimeters; it rescales the inconsistency.

Check that the units describe compatible quantities

Meters and centimeters both measure length. Kilograms measure mass. Converting a kilogram value into meters requires additional relationships and information; it is not a choice of a conversion factor from a unit table.

Even a shared unit does not guarantee a shared quantity. Net mass and gross mass may both be stored in kilograms while referring to different things. Likewise, a lifetime count and a daily count can be integers with different time coverage. Document the field’s meaning before merging sources.

Unit symbols are case-sensitive. mW means milliwatts; MW means megawatts. Lowercasing both would collapse a billion-fold difference. Recognize documented aliases explicitly, and leave unknown or missing units unresolved rather than guessing from a plausible-looking value.

A squared unit needs a squared conversion factor

A square with 10 cm sides has area 100 cm². Each side is 0.1 m, so the same square has area 0.01 m². We converted two lengths before multiplying them.

Change the unit; keep the square10 cm10 cm100 cm²Same physical square10 cm × 10 cm = 100 cm². Each small cell represents 1 cm².
Switch to meters to see both side lengths change together. Each grid cell remains the same physical area: 1 cm², or 0.0001 m².

The factor for one square centimeter follows from the same reasoning:

$$1\,\mathrm{cm}^2=(0.01\,\mathrm m)^2=0.0001\,\mathrm m^2.$$

Dividing 100 cm² by only 100 would incorrectly give 1 m², a hundred times the correct area. Volumes require the cube of the length factor. Rates require converting both numerator and denominator; 36 km/h becomes 36 × 1000 / 3600 = 10 m/s.

An offset changes readings differently from differences

On the Fahrenheit scale, 32 °F corresponds to 0 °C, and 212 °F corresponds to 100 °C. The scales have different zero points as well as different step sizes. Calling the numerical readings $t_F$ and $t_C$, convert a Fahrenheit reading with

$$t_C=\frac{t_F-32}{1.8}.$$

For example, 68 °F becomes (68 − 32) / 1.8 = 20 °C. A change of 18 °F instead becomes a change of 10 °C. Both endpoints carry the same offset, so that offset cancels when taking their difference.

What the field representsFahrenheit valueCelsius value
Temperature reading68 °F20 °C
Temperature increase18 °F10 °C
No temperature change0 °F difference0 °C difference

A generic “multiply everything by a factor” function is insufficient for temperature readings. The schema needs to distinguish a reading from a difference. The NIST conversion table lists them separately for this reason.

Convert the value and the unit together

This small converter accepts already-parsed, finite, nonnegative lengths in exactly three supported units. It returns both the numerical value and the canonical unit. Unsupported units require a source decision, not a default factor.

Python: convert lengths and distinguish temperature readings from changes
import math

LENGTH_FACTORS = {"m": 1.0, "cm": 0.01, "mm": 0.001}

def to_metres(value, unit):
    if unit not in LENGTH_FACTORS:
        raise ValueError("unsupported length unit")
    if not math.isfinite(value) or value < 0:
        raise ValueError("finite nonnegative length required")
    return value * LENGTH_FACTORS[unit], "m"

rows = [("A", 1.7, "m"), ("B", 170.0, "cm"), ("C", 1700.0, "mm")]
for source, value, unit in rows:
    converted = to_metres(value, unit)
    print(f"{source}: {converted[0]:.3f} {converted[1]}")
    assert to_metres(*converted) == converted

def fahrenheit_to_celsius(value, *, interval=False):
    if not math.isfinite(value):
        raise ValueError("finite temperature required")
    return (value - (0 if interval else 32)) / 1.8

print(f"reading: {fahrenheit_to_celsius(68):.1f} C")
print(f"change: {fahrenheit_to_celsius(18, interval=True):.1f} C")
A: 1.700 m
B: 1.700 m
C: 1.700 m
reading: 20.0 C
change: 10.0 C

The repeat-conversion assertion checks idempotence: applying canonicalization again leaves an already-canonical record unchanged. If you changed 170 to 1.7 but accidentally kept the unit “cm”, a second pass would produce 0.017 m. Updating the value and unit together avoids that mistake.

The printed decimal places are display formatting. Unit conversion does not create additional measurement precision, and binary floating-point arithmetic may introduce small representation errors. Retain the source precision or uncertainty when it matters to the task.

Check agreement after the representations agree

Once units and meanings match, check relationships between fields. Examples include an end time following its start time, component measurements agreeing with a reported total, and two sources describing the same object consistently. A failed check is evidence to investigate; it does not tell you which source is wrong.

For measured or computed quantities, choose a tolerance in the canonical unit. An absolute tolerance of 0.001 m means a difference of up to one millimeter. Whether that is acceptable depends on the measurement process. Relative tolerances instead scale with magnitude.

Python: compare converted readings with a declared absolute tolerance
a, _ = to_metres(170, "cm")
b, _ = to_metres(1700.5, "mm")
print(math.isclose(a, b, rel_tol=0.0, abs_tol=0.001))
print(math.isclose(a, b, rel_tol=0.0, abs_tol=0.0001))
True
False

Here the readings differ by 0.0005 m, or half a millimeter. The first policy accepts that difference; the second flags it. Stating the tolerance makes the outcome interpretable.

Category consistency also needs a field-specific rule

Suppose a shipping-status schema says “in transit” and “IN_TRANSIT” both mean in_transit. An explicit alias map can reconcile those spellings. It should leave an unfamiliar status for review and preserve the raw text.

This differs from assuming every text field should be lowercased or stripped of punctuation. Those operations may erase distinctions in identifiers, unit symbols or natural language. The text-normalization chapter handles those choices separately.

Optional C++: keep a quantity’s unit attached

The struct makes the value–unit pair explicit. The converter supports lengths only and returns meters; extending it to other dimensions or offset units requires a corresponding contract.

Compile the length converter
#include <cmath>
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <string_view>

struct Length { double value; std::string_view unit; };
Length to_metres(Length input) {
    double factor;
    if (input.unit=="m") factor=1;
    else if (input.unit=="cm") factor=0.01;
    else if (input.unit=="mm") factor=0.001;
    else throw std::invalid_argument("unsupported length unit");
    if (!std::isfinite(input.value) || input.value<0)
        throw std::invalid_argument("finite nonnegative length required");
    return {input.value*factor,"m"};
}
int main() {
    const Length values[]={{1.7,"m"},{170,"cm"},{1700,"mm"}};
    std::cout << std::fixed << std::setprecision(3);
    for (Length value:values) {
        auto result=to_metres(value);
        std::cout << result.value << " " << result.unit << "\n";
    }
}
1.700 m
1.700 m
1.700 m

Record the conversion as part of the dataset definition

01Retain the source pairKeep the raw field, parsed value, source unit and quantity definition.
02Apply a reviewed conversionValidate the dimension, convert both value and unit, and record the conversion-table version.
03Check the resultApply domain and cross-field checks in canonical units. Preserve failures and the reason each was flagged.
An auditable canonical record. A fixed conversion from a documented unit definition is not a statistic fitted from the dataset. Use the same reviewed rule in training and serving.

Rules inferred from data, including learned scaling parameters or imputation values, still belong inside the training boundary. Keep them separate from unit conversion so a source-format change can be diagnosed without mistaking it for a model failure.

References and further reading