Backward Fill

September 2026 · 9 min read

Backward fill copies the next observed value into an earlier gap. The algorithm is useful when a later observation is an acceptable source for the task. Its name describes the direction of copying, not evidence that the value really held earlier.

The distinction from forward fill is the source: an earlier observation versus a later one. We will keep the mechanics short and focus on what that later source means for reconstruction and prediction.

Trace each replacement to its next observation

Consider [missing, 3, missing, missing, 5, missing] at minutes 0 through 5. Backward fill returns [3, 3, 5, 5, 5, missing]. The first gap borrows from minute 1; the two interior gaps borrow from minute 4. The final gap has no later observation.

Scanning the array from right to left makes this easy to implement: remember the next real observation, then copy it into earlier gaps. An all-missing series stays missing, as does any trailing stretch after the last observation.

A reconstruction needs a reason to extend values backward

Suppose a device’s setting is 3 at minute 1 and 5 at minute 4. Those two readings alone do not tell us when it changed. It might have changed at minute 2, at minute 4, or moved through other values between the observations.

An explicit validity interval is stronger evidence. A record received at 09:10 might state that a setting has been valid since 09:00. A retrospective state table can use that interval, but should not extend it to 08:55 merely because an earlier cell is empty. Prefer interval-aware joins when that is the source contract.

For a descriptive plot, a backward-filled step function may be an acceptable visual convention. Mark the filled regions so readers can distinguish observations from reconstruction. Using the final dataset offline does not make arbitrary fills appropriate for estimating trends or uncertainty.

The cutoff is when the prediction was made

If a prediction at minute 2 uses a value first measured at minute 4, its feature contains future information. This can inflate a backtest even when both timestamps fall inside the training period. A train/test split does not repair information leaking between rows within a split.

The relevant question is whether the source was available at the decision cutoff. A schedule published yesterday for an event tomorrow can be a legitimate feature today; an event’s realized value measured tomorrow cannot. A future effective date and future availability are different things. The forward-fill chapter’s arrival-time example covers the complementary late-record case.

If the task deliberately waits until minute 4 to estimate what happened at minute 2, then future observations are available to that delayed reconstruction. Evaluate it as such, with the same delay in validation and deployment. Do not describe it as an immediate minute-2 forecast.

Bound the lookahead and keep groups separate

Even for retrospective work, a distant successor can be a poor substitute. A maximum lookahead rejects sources too far after the gap. Pandas’ bfill(limit=1) limits the number of consecutive missing rows filled before an observation; it does not impose one minute of lookahead on irregular timestamps.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "device": ["A"] * 6 + ["B"] * 2,
    "minute": [0, 1, 2, 3, 4, 5, 0, 1],
    "value": [np.nan, 3, np.nan, np.nan, 5, np.nan, np.nan, np.nan],
}).sort_values(["device", "minute"])
assert not df.duplicated(["device", "minute"]).any()

observed_at = df["minute"].where(df["value"].notna())
source_time = observed_at.groupby(df["device"]).bfill()
df["lookahead"] = source_time - df["minute"]
candidate = df.groupby("device")["value"].bfill()
# Offline reconstruction policy: permit a source at most one minute later.
df["filled"] = candidate.where(df["lookahead"] <= 1)
assert np.allclose(df.loc[df.device == "A", "filled"],
                   [3, 3, np.nan, 5, 5, np.nan], equal_nan=True)
assert df.loc[df.device == "B", "filled"].isna().all()
print(df.loc[df.device == "A", "filled"].tolist())

[3.0, 3.0, nan, 5.0, 5.0, nan]

Sort within entities and resolve duplicate timestamps before filling. Applying a global fill after a grouped operation can accidentally cross entity boundaries; keep each directional pass grouped. The combined-fill chapter shows why the order of those passes also matters.

Leave a gap when the source does not justify a fill

Backward fill is a reasonable candidate for a deliberately retrospective representation when the next value is a defensible stand-in. Compare it with preserving gaps or using an interpolation model when the quantity is continuous and both endpoints are available.

It is not a general solution for the leading gaps left by forward fill. In a live predictor, use available history, an explicit missing-input policy, or a delayed decision if the application permits it. In a statistical analysis, keep track of how much of a reported pattern depends on carried values.

C++: the reverse scan

This version demonstrates unlimited backward fill on one already ordered series of finite values or NaNs. It returns a copy, leaving the raw data unchanged. The reverse loop safely handles an empty vector; each element is visited once, so the scan takes linear time.

#include <cmath>
#include <iostream>
#include <limits>
#include <vector>

std::vector<double> backwardFill(const std::vector<double>& raw) {
    auto filled = raw;
    double next = std::numeric_limits<double>::quiet_NaN();
    // This reverse-loop form also handles an empty vector.
    for (std::size_t i = raw.size(); i > 0;) {
        --i;
        if (!std::isnan(raw[i])) next = raw[i];
        else if (!std::isnan(next)) filled[i] = next;
    }
    return filled;
}

int main() {
    const double NA = std::numeric_limits<double>::quiet_NaN();
    const auto filled = backwardFill({NA, 3, NA, NA, 5, NA});
    for (std::size_t i = 0; i < filled.size(); ++i) {
        if (i) std::cout << " ";
        if (std::isnan(filled[i])) std::cout << "NA";
        else std::cout << filled[i];
    }
    std::cout << "\n";
}

3 3 5 5 5 NA

References and further reading

• pandas: backward fill — copying the next valid observation and limiting consecutive rows.
• pandas: as-of joins — matching preceding or subsequent observations with explicit tolerances.
• Feast: point-in-time joins — reconstructing features at a specified historical timestamp.