Missingness Indicators

September 2026 · 9 min read

After replacing missing values with zero, an observed zero and a gap both look like zero. A missingness indicator preserves that distinction. It records whether the original entry was absent, while an imputer supplies a numerical representation the model can use.

The flag is not a confidence score and does not reveal the missing value. It tells the model which kind of input it received.

Preserve the original gap before filling it

Let $M=1$ mean the raw value was missing and $M=0$ mean it was observed. Using a zero fill, three inputs become:

If you calculate isnan only after filling, the original gaps have already disappeared. For multiple input columns, keep a documented order for the value columns and their corresponding flags.

What can a model do with the extra bit?

A tree can split on $M$ to separate observed and imputed cases. A linear model can learn an offset for missing inputs. Write $\tilde x$ for the filled value and $\beta_0$, $\beta_x$, and $\beta_M$ for coefficients learned from training data:

For an observed zero, the prediction is $\beta_0$. For a missing value filled with zero, it is $\beta_0+\beta_M$. The model can therefore give the two cases different predictions. This simple additive form does not automatically model every interaction between missingness and other inputs.

The fill value remains part of the representation. If a missing value is filled with a constant $c$, its prediction becomes $\beta_0+\beta_xc+\beta_M$. Changing the fill at serving time without retraining changes what the fitted coefficients receive.

An indicator can help even when gaps are MCAR

Suppose a binary feature $X$ is equally likely to be 0 or 1, and the target we want to predict is $Y=X$. Hide $X$ with probability one half, independently of its value. That is MCAR.

With zero filling, a filled zero has two explanations. It may be an observed zero, whose target is definitely 0, or a missing value, whose target is equally likely to be 0 or 1. The flag lets a squared-error predictor use the conditional mean 0 for the first case and 0.5 for the second.

Missingness itself is independent of the target here. The indicator helps because it distinguishes an observed input from its replacement after imputation. It does not reconstruct the missing 0 or 1.

import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression

# Four equally likely population cases: X=0/1, independently missing/present.
raw = np.array([[0.], [1.], [np.nan], [np.nan]])
y = np.array([0., 1., 0., 1.])  # The target equals the original X.
encoded = SimpleImputer(
    strategy="constant", fill_value=0, add_indicator=True,
).fit_transform(raw)

plain = LinearRegression().fit(encoded[:, :1], y)
flagged = LinearRegression().fit(encoded, y)
assert np.allclose(plain.predict(encoded[:, :1]), [1/3, 1, 1/3, 1/3])
assert np.allclose(flagged.predict(encoded), [0, 1, 0.5, 0.5])
print("Without flag:", np.round(plain.predict(encoded[:, :1]), 4))
print("With flag:", np.round(flagged.predict(encoded), 4))

Decide which indicator columns exist

SimpleImputer(add_indicator=True) appends indicators for features that were missing during fitting. It does not add a new indicator column merely because a previously complete feature becomes missing at transform time.

For example, training rows [[1, 10], [missing, 20], [3, 30]] contain gaps only in column 0. Median imputation learns values 2 and 20. A test row [4, missing] becomes [4, 20, 0]: two filled values plus the existing flag for column 0. There is no flag for column 1.

If the feature contract needs a flag for every input column, use MissingIndicator(features="all") in a separate branch that sees the original input:

import numpy as np
from sklearn.impute import SimpleImputer, MissingIndicator
from sklearn.pipeline import FeatureUnion

train = np.array([[1., 10.], [np.nan, 20.], [3., 30.]])
test = np.array([[4., np.nan]])

# Adds a flag only for column 0, which had training-time gaps.
compact = SimpleImputer(strategy="median", add_indicator=True).fit(train)
assert compact.transform(test).tolist() == [[4., 20., 0.]]

# Both branches receive the SAME raw input.
features = FeatureUnion([
    ("values", SimpleImputer(strategy="median", keep_empty_features=True)),
    ("mask", MissingIndicator(features="all")),
]).fit(train)
assert features.transform(test).tolist() == [[4., 20., 0., 1.]]
print(features.transform(test))

A stable schema does not supply experience the model never saw. If column 1 was always present in training, its missingness flag was always zero; the learner has no training examples showing what should happen when it becomes one. Monitor such new patterns and assess the resulting predictions.

The keep_empty_features=True option preserves the value-branch width even for an entirely missing training column. Its fallback-value semantics still need an explicit decision, as explained in the all-missing-column discussion.

Evaluate the representation under the collection process

Indicators can help distinguish input quality, optional fields, or different measurement processes. They can also let a model rely on an association that disappears after a form redesign, sensor repair, or data-provider change.

Compare imputation with and without flags using the same appropriate validation splits. Break results down by missingness pattern and relevant groups or periods. A rare pattern may have too little support for a reliable conclusion even when the overall metric looks stable.

Construct features from information available at prediction time. A flag saying that a measurement was requested after the outcome can leak future information just as an ordinary input can.

Finally, indicators are a predictive representation, not an automatic correction for biased population estimates under MNAR. The mechanism chapter explains why preserving the fact of absence does not identify the distribution of the unseen values.

C++: encode one raw value as a pair

The implementation reads the missingness state once, then returns both the replacement value and the original flag. The fill is assumed to have been chosen or fitted during training.

#include <cmath>
#include <iostream>
#include <limits>
#include <vector>

struct Encoded { double value; int wasMissing; };
Encoded encode(double raw, double fittedFill) {
    const bool missing = std::isnan(raw);
    return {missing ? fittedFill : raw, missing ? 1 : 0};
}

int main() {
    const double NA = std::numeric_limits<double>::quiet_NaN();
    const double fittedFill = 0;
    for (double raw : std::vector<double>{0, NA, 7}) {
        const auto result = encode(raw, fittedFill);
        std::cout << "value=" << result.value
                  << " was_missing=" << result.wasMissing << "\n";
    }
}

value=0 was_missing=0
value=0 was_missing=1
value=7 was_missing=0

References and further reading

• Scikit-learn: MissingIndicator — all-feature masks, training-selected masks, and behavior for newly missing features.
• Scikit-learn: SimpleImputer — which indicator columns add_indicator creates and how empty features are handled.
• Scikit-learn: marking imputed values — combining missingness masks with imputed values in a predictive pipeline.