Forward and Backward Fill

September 2026 · 9 min read

Forward fill borrows an earlier observation; backward fill borrows a later one. Combining them is an ordered policy: the first pass decides most of the values, and the second fills whatever remains.

This chapter compares those policies. For source-age limits and historical availability, see forward fill; for the assumptions behind using later observations, see backward fill.

Compare the source as well as the filled value

Use one series at times 0 through 5: [missing, 3, missing, missing, 5, missing]. There are four gaps and two observations. Switch the policy below to see which real observation supplies each replacement.

The first pass has priority

Forward then backward first fills every gap with an available predecessor. The second pass leaves those replacements alone and fills the remaining leading gaps from the first observation. Its output is [3, 3, 3, 3, 5, 5].

Backward then forward does the opposite: successors determine the interior gaps, then the final forward pass patches the trailing edge. Its output is [3, 3, 5, 5, 5, 5]. Both contain no gaps, yet they encode different histories.

With unlimited passes and at least one observation in an entity, either combination fills every position. An entirely missing entity remains missing. With row or time limits, gaps may remain even when the entity has observations.

Combining the passes does not make later information available sooner. In the forward-first result, time 0 still borrows from time 1. If the reading first became available at time 1, it could not have supported a time-0 prediction.

Two-way fill is not nearest-neighbor selection or interpolation

Look at time 3. Its next observation, 5 at time 4, is closer than its previous observation, 3 at time 1. Forward-first filling still chooses 3. The algorithm follows its pass priority; it never compares the two time distances.

Interpolation assumes a path between endpoints; carrying assumes a constant value over the gap. Choose according to the variable’s meaning. A category such as an operating mode cannot be meaningfully averaged like a temperature. On irregular timestamps, interpolation must use elapsed time rather than assume all rows are equally spaced.

Nearest matching and interpolation can also use later observations. Their suitability for retrospective plots does not establish suitability for immediate prediction features.

Keep both passes inside the entity

A common pandas trap is to group the first operation but not the second. The result of a grouped ffill() is an ordinary Series or DataFrame. Calling bfill() on that result performs an ungrouped fill and can copy a value from the next entity.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "device": ["A"] * 6 + ["B"] * 2 + ["C"] * 2,
    "time": [0, 1, 2, 3, 4, 5, 0, 1, 0, 1],
    "value": [np.nan, 3, np.nan, np.nan, 5, np.nan,
              np.nan, np.nan, np.nan, 9],
}).sort_values(["device", "time"])
assert not df.duplicated(["device", "time"]).any()

# Both passes happen inside each group.
df["filled"] = df.groupby("device")["value"].transform(
    lambda s: s.ffill().bfill()
)
assert df.loc[df.device == "A", "filled"].tolist() == [3, 3, 3, 3, 5, 5]
assert df.loc[df.device == "B", "filled"].isna().all()
assert df.loc[df.device == "C", "filled"].tolist() == [9, 9]

# The second call here is ungrouped: it incorrectly gives B the value 9.
wrong = df.groupby("device")["value"].ffill().bfill()
assert wrong.loc[df.device == "B"].tolist() == [9, 9]

Preserve a raw missingness mask before either pass. When you need an audit trail or elapsed-time tolerance, preserve the original source timestamp too. A value copied into an intermediate row must not become a new observation with a fresh timestamp.

Decide how to handle the boundaries first

For an online feature, use eligible past history and an explicit policy for leading gaps or stale values. For a retrospective series, decide whether you accept a successor, a predecessor, a time-based interpolation, or no reconstruction at each type of gap.

Interior gaps have two possible anchors; edge gaps have only one. Extending an endpoint beyond the observed interval makes a different assumption from filling between two observations. An all-missing series supplies neither anchor. Treat these cases separately when reviewing outputs.

Evaluate the filled representation for the intended task, including error during long gaps and the proportion of rows that remain unsupported. A reduction in the number of NaNs measures coverage, not accuracy. Retain visible gaps in a plot when a continuous-looking line would imply evidence you do not have.

C++: carry the source index through both passes

The following teaching implementation returns each value together with the index of the original observation that supplied it. It supports either pass order on one ordered series with unlimited fills. It assumes the series is small enough for an integer source index, and values are finite numbers or NaNs.

#include <cmath>
#include <iostream>
#include <limits>
#include <vector>

struct Cell { double value; int source; };

std::vector<Cell> fillBoth(const std::vector<double>& raw, bool forwardFirst) {
    std::vector<Cell> out;
    for (std::size_t i = 0; i < raw.size(); ++i)
        out.push_back({raw[i], std::isnan(raw[i]) ? -1 : static_cast<int>(i)});
    auto forward = [&]() {
        Cell last{std::numeric_limits<double>::quiet_NaN(), -1};
        for (auto& cell : out) {
            if (cell.source >= 0) last = cell;
            else if (last.source >= 0) cell = last;
        }
    };
    auto backward = [&]() {
        Cell next{std::numeric_limits<double>::quiet_NaN(), -1};
        for (std::size_t i = out.size(); i > 0;) {
            --i;
            if (out[i].source >= 0) next = out[i];
            else if (next.source >= 0) out[i] = next;
        }
    };
    if (forwardFirst) { forward(); backward(); }
    else { backward(); forward(); }
    return out;
}

int main() {
    const double NA = std::numeric_limits<double>::quiet_NaN();
    const std::vector<double> raw = {NA,3,NA,NA,5,NA};
    for (bool forwardFirst : {true,false}) {
        const auto result = fillBoth(raw,forwardFirst);
        std::cout << (forwardFirst ? "forward then backward: " : "backward then forward: ");
        for (const auto& cell : result)
            std::cout << cell.value << "@" << cell.source << " ";
        std::cout << "\n";
    }
}

forward then backward: 3@1 3@1 3@1 3@1 5@4 5@4
backward then forward: 3@1 3@1 5@4 5@4 5@4 5@4

Read 3@1 as “value 3, originally observed at index 1.” Two scans take linear time, with storage for the output. Source index −1 represents an unresolved gap.

References and further reading

• pandas: forward fill — the last-observation rule and limits.
• pandas: backward fill — the next-observation rule and limits.
• pandas: grouped fill — propagation inside groups and the resulting Series or DataFrame.
• pandas: interpolation — linear versus time-aware interpolation methods.
• pandas: as-of joins — directional and nearest time matches.