Data Types and Parsing

September 2026 · 14 min read

Parsing turns recorded text into values with a defined meaning. A file may contain only strings, but its columns can represent identifiers, counts, dates, categories and measurements. Choosing the right type begins with what a column means.

This chapter follows one count field from raw text to a checked value. We keep the raw entry and a reason for every failure, so a malformed value does not silently become an ordinary missing observation.

The same characters can mean different things

Consider the text 0017. As a customer identifier, its leading zeros may be part of the identity. As a count, it can represent the integer 17. A parser cannot decide that distinction from the characters alone.

A schema records the expected columns and types. A useful parsing contract also states accepted syntax, missing markers and value constraints. For a timestamp, include its format and time-zone meaning. For a measurement, include its unit; unit conversion is a separate step from recognizing a number.

Separate absent, malformed and out-of-range values

Our illustrative count contract permits ASCII digits, including leading zeros. It removes surrounding spaces, tabs and line breaks. A blank entry is missing. Decimal points, commas, signs and units are not accepted syntax. The final count must be at most 10,000.

The maximum comes from this example’s field specification. It is not an outlier threshold learned from a sample. A different count field needs its own domain rule.

Zero is a recorded count. Missing means no count was provided. Invalid means the entry failed the agreed rules. Keeping those distinctions lets you decide later whether to request a correction, isolate a row or use a missing-value treatment.

Preserve the recorded fields before converting them

CSV readers can infer types and recognize missing markers automatically. That convenience can erase leading zeros or treat a legitimate category such as “NA” as missing. In the example below, we import every field as text and disable default missing-string recognition. The original file remains the source of truth; a parsed field is not a byte-for-byte archive of CSV quoting and encoding.

Keep a row identifier, the raw field, the parsed value and the status. Pandas’ nullable integer dtype Int64 can represent both whole-number values and missing entries without converting the column into floating point. The status column explains why a value is absent.

import re
from io import StringIO
import pandas as pd

def parse_count(raw):
    text = raw.strip(" \t\r\n")
    if text == "":
        return None, "missing"
    if re.fullmatch(r"[0-9]+", text) is None:
        return None, "invalid_format"
    digits = text.lstrip("0") or "0"
    # Compare bounded digit strings before converting a huge integer.
    if len(digits) > 5 or (len(digits) == 5 and digits > "10000"):
        return None, "invalid_range"
    return int(digits), "ok"

csv_text = """customer_id,item_count
0001,17
0002," 17 "
0003,
0004,17kg
0005,3.0
0006,"1,234"
0007,10001
0008,0
0009,00017
"""
df = pd.read_csv(StringIO(csv_text), dtype="string", keep_default_na=False)
results = [parse_count(raw) for raw in df["item_count"]]
df["count"] = pd.array([value for value, status in results], dtype="Int64")
df["status"] = [status for value, status in results]
print(df["customer_id"].tolist())
print(df["status"].value_counts().sort_index().to_dict())

['0001', '0002', '0003', '0004', '0005', '0006', '0007', '0008', '0009']
{'invalid_format': 3, 'invalid_range': 1, 'missing': 1, 'ok': 4}

pd.to_numeric(..., errors="coerce") is useful when conversion failures should become missing values, but it does not preserve the reason by itself. Record which inputs were originally absent and which failed conversion before those cases merge into the same null representation.

Dates need both a syntax check and a calendar check

For our date field, “2026-04-03” means April 3. “03/04/2026” is rejected because it does not use the declared format. “2026-02-30” has the right shape but names a nonexistent date. These are different failures.

import re
from datetime import datetime, timezone

def parse_date(raw):
    text = raw.strip(" \t\r\n")
    if text == "":
        return None, "missing"
    if re.fullmatch(r"[0-9]{4}-[0-9]{2}-[0-9]{2}", text) is None:
        return None, "invalid_format"
    try:
        return datetime.strptime(text, "%Y-%m-%d").date(), "ok"
    except ValueError:
        return None, "invalid_calendar"

for raw in ["2026-04-03", "03/04/2026", "2026-02-30"]:
    value, status = parse_date(raw)
    print(raw, status, value)

instant = datetime.fromisoformat("2026-09-14T09:00:00+05:30")
print(instant.astimezone(timezone.utc).isoformat())

2026-04-03 ok 2026-04-03
03/04/2026 invalid_format None
2026-02-30 invalid_calendar None
2026-09-14T03:30:00+00:00

The final line converts an offset-aware timestamp to UTC while preserving the instant. A bare local time needs a declared time zone and rules for ambiguous or nonexistent clock times. Assigning UTC to a local timestamp is not the same operation as converting it to UTC.

Likewise, “1,234” needs a numeric locale: a comma may be a grouping mark or a decimal separator. Configure the documented source format, or isolate ambiguous entries. Automatically removing every comma can turn a valid value into a different valid value.

Optional C++: reject partial numeric parses

std::from_chars reports both a conversion error and the first unconsumed character. A successful numeric prefix is insufficient: “17kg” must not become 17. This helper enforces the same count contract and returns a value only for the ok status.

#include <charconv>
#include <iostream>
#include <optional>
#include <string_view>
#include <system_error>

struct Result { std::optional<int> value; std::string_view status; };
Result parse_count(std::string_view text) {
    auto space=[](char c) { return c==' ' || c=='\t' || c=='\r' || c=='\n'; };
    while (!text.empty() && space(text.front())) text.remove_prefix(1);
    while (!text.empty() && space(text.back())) text.remove_suffix(1);
    if (text.empty()) return {std::nullopt,"missing"};
    for (char c:text)
        if (c<'0' || c>'9') return {std::nullopt,"invalid_format"};
    unsigned long long value=0;
    auto result=std::from_chars(text.data(),text.data()+text.size(),value);
    if (result.ec==std::errc::result_out_of_range)
        return {std::nullopt,"invalid_range"};
    if (result.ec!=std::errc{} || result.ptr!=text.data()+text.size())
        return {std::nullopt,"invalid_format"};
    if (value>10000) return {std::nullopt,"invalid_range"};
    return {static_cast<int>(value),"ok"};
}
int main() {
    for (auto raw:{"17","","17kg","10001","0"}) {
        const auto result=parse_count(raw);
        std::cout << '\"' << raw << "\" -> " << result.status;
        if (result.value) std::cout << " " << *result.value;
        std::cout << "\n";
    }
}

"17" -> ok 17
"" -> missing
"17kg" -> invalid_format
"10001" -> invalid_range
"0" -> ok 0

Make parsing an observable part of the data pipeline

A fixed format rule can be applied consistently to training and serving data. Statistics learned from data, such as replacement medians or category vocabularies, still belong inside the training boundary. Keep the parser contract stable across splits, and measure which observations are excluded by any failure policy.

Continue with units and consistency: a perfectly parsed number can still be wrong to compare with another column if their units or meanings differ.

References and further reading

• Pandas: read_csv — explicit text dtypes, missing-marker policies and source-format controls.
• Pandas: to_numeric — conversion behavior and errors="coerce".
• Pandas: nullable integer data type — whole-number storage with an explicit missing representation.
• Python: datetime — calendar parsing, aware versus naive timestamps and time-zone conversion.
• C++ working draft: from_chars — conversion results, unconsumed characters and out-of-range errors.