Handling Missing Data

September 2026 · 8 min read

A missing value is a statement about what was recorded, not a number waiting for a universally correct replacement. Before choosing a method, ask what the absence means and what the finished analysis needs to do.

This series separates three tasks: making predictions from incomplete inputs, constructing features from a time history, and estimating population quantities with uncertainty. They can use similar operations but need different evidence that those operations are appropriate.

Start with the meaning of the gap

Suppose a customer record contains tenure = 12 months and monthly usage = missing. An unavailable usage log does not establish zero usage. A complete log with no recorded activity may support zero, depending on the collection contract. A product the customer never had may make the field not applicable.

Preserve these distinctions when the source supports them. Standardize the actual missing markers during parsing; do not automatically interpret every zero, empty string, or category named “NA” as absence. Keep the raw data available after building the model’s numerical representation.

The missingness mechanism describes what gets hidden

MCAR means missingness is independent of the data values. MAR allows it to depend on observed information, but not additionally on the hidden value after conditioning on that information. MNAR allows a dependence on hidden values that the observed information does not explain.

These are assumptions about the recording process, not labels an imputer can discover simply by fitting well. Associations with recorded features can supply clues, but observed data generally cannot establish MAR instead of MNAR. The mechanisms chapter makes the distinction concrete with explicit hiding rules.

Choose a route based on the question

Deleting incomplete rows is another option, with a cost in sample retention or prediction coverage. Its validity depends on the intended estimate and selection process; “MAR always makes deletion invalid” is too broad. Check which columns actually need to be complete for the analysis.

Some estimators handle NaNs directly, so filling every feature is not a universal requirement. Native missing-value handling is still a modeling choice: compare its performance and behavior on the missing patterns expected at serving time.

For prediction, evaluate the entire fitted procedure

Choose an evaluation split before learning fill values, donor pools, or per-column regression models. Fit them on training rows and apply them to validation rows. During cross-validation, repeat that fitting inside each training fold; the tuning chapter explains how a pipeline enforces the boundary.

Compare alternatives on the same validation design. Check the task metric, performance on incomplete inputs, coverage, feature-schema stability, and cost. If one method drops difficult rows while another predicts for everyone, a single accuracy number does not make them comparable.

For time histories, reproducing information availability matters as well as fitting boundaries. Valid past observations may carry into a later validation period, but later-arriving records cannot inform an earlier prediction. For statistical inference on an existing study, a train/test workflow may not be relevant; the imputation and analysis assumptions take its place.

Read the chapter that answers your next question

For a first pass, read the mechanisms, simple-imputation, and indicator chapters. Then follow the route that matches your data. Each chapter owns its worked examples; this overview is the map.

Understand the absence and establish a baseline

• Missingness mechanisms — what MCAR, MAR, and MNAR say, and what the recorded data can reveal.
• Mean, median, and mode — fit a reproducible baseline, inspect what it changes, and handle empty columns.
• Missingness indicators — distinguish an observation from a replacement while preserving the feature schema.
• Listwise and pairwise deletion — track retained cases, changing populations, and incompatible pairwise statistics.

Choose a richer representation when the task supports it

• Constants and sentinels — separate meaningful defaults from special encodings and inspect their numerical effects.
• KNN imputation — choose usable donors through shared observed features and specify fallback behavior.
• Iterative imputation — follow column updates, retain the original mask, and transform new rows with fitted models.

Use the time history deliberately

• Forward versus backward fill — compare pass order, source provenance, interpolation, and entity boundaries.
• Forward fill — carry available past readings with explicit freshness limits.
• Backward fill — understand later sources, reconstruction assumptions, and lookahead limits.

Carry uncertainty into an analysis

• Multiple imputation — generate plausible completions, analyze each, and pool the estimates and uncertainty.

What a finished missing-data decision should explain

Be able to trace a prediction’s filled feature back to its rule and permitted information. State what remains unknown, what happens when a feature is entirely missing, and how the method behaves when the recording process changes.

The outcome is a defensible procedure for the task, not merely a matrix without NaNs. Keep the raw missingness pattern, record the fitted policy with the model, and revisit the choice when new data no longer resembles the conditions used to validate it.

References and further reading

• Scikit-learn: imputation guide — simple and multivariate imputers, indicators, and estimators that accept missing values.
• Van Buuren: missingness mechanisms — the assumptions behind MCAR, MAR, and MNAR.
• Van Buuren: simple solutions — deletion and single imputation, including their statistical limitations.
• Feast: point-in-time joins — matching feature history to the time of a decision.