Time-Series Cross-Validation

September 2026 · 8 min read

Time-series cross-validation replays a sequence of forecasting decisions. Each round moves the prediction date forward, fits or updates the forecasting procedure using eligible history, and evaluates later outcomes. The forecast schedule determines what information is available in each round.

The time-splitting chapter covers date boundaries, delayed labels, and gaps. Here we focus on rolling forecast origins, multi-step predictions, and scoring the exact horizon we intend to deploy.

Define the origin and the horizon

A forecast origin is the point when the forecast is issued. The horizon is how far ahead it predicts. If we issue a forecast just after observing day 5, horizons 1, 2, and 3 refer to days 6, 7, and 8. All three predictions must use information available at that same origin.

For a concrete example, use the following daily demand sequence. These are constructed values for checking the arithmetic:

Day:     0   1   2   3   4   5   6   7   8   9  10  11  12  13  14
Demand: 10  12  11  15  14  16  17  18  22  20  21  25  24  23  27

Our baseline predicts the last observed value at every future horizon. After day 5, the last observation is 16, so the forecasts for days 6–8 are all 16. This is the naïve forecasting method. It isolates the forecast clock without requiring a complicated model fit.

Issue the forecast, then reveal the future

Keep the horizon in the score

Repeat the three-step forecast after days 5, 8, and 11. We measure absolute error: the magnitude of the difference between actual demand and its forecast. The table contains the errors produced by those three origins:

Mean absolute error (MAE) averages those magnitudes. Average down each horizon’s column to answer how well the method predicts that many days ahead:

The overall average is $20/9\approx2.22$ demand units. It is a useful summary only if giving equal weight to these nine forecast errors matches the application. If the business decision is made three days ahead, the third-horizon result deserves particular attention. Forecast error need not increase monotonically with horizon on every dataset; these values describe this example.

A daily update answers a different question

Suppose we instead wait for day 6’s observed demand of 17, then issue a new one-day forecast for day 7. The naïve forecast becomes 17. At the original day-5 issue time, that value was not yet known, so the two-day-ahead forecast for day 7 was 16.

Using 17 while scoring the original day-5 forecast would give the model extra information. Using 17 for a new forecast issued after day 6 is valid. The distinction is the issue time attached to the prediction, not which part of the spreadsheet stores the actual value.

For an autoregressive model that needs previous target values as inputs, a multi-step forecast may have to feed its own earlier predictions into later steps. Supplying future observed targets instead changes the task. Specify the forecasting procedure before comparing scores.

Choose how much history each fit may use

For models that learn parameters, an expanding window retains all eligible history as the origin advances. A rolling window retains only a recent fixed-length history. For a four-day rolling window in our example:

Expanding windows keep more examples. Rolling windows can reduce the influence of obsolete behavior but leave less data to fit. Select window length using chronological development results. The naïve baseline above only uses the latest observation, so it does not measure the difference between those training-window choices.

TimeSeriesSplit can generate expanding training indices, limit their size with max_train_size, and exclude recent rows with gap. These are row counts, not calendar durations. Irregular events and delayed labels need explicit timestamp checks, as in the worked availability example.

Reproduce every forecast and error

This code deliberately keeps prediction construction separate from later scoring. The forecast matrix uses only the observations at the three origins; the target matrix reads the future values afterward.

import numpy as np

demand = np.array([10, 12, 11, 15, 14, 16, 17, 18,
                   22, 20, 21, 25, 24, 23, 27])
origins = np.array([5, 8, 11])
horizons = np.array([1, 2, 3])

# Freeze the forecasts at their issue times.
forecast = np.repeat(demand[origins, None], 3, axis=1)

# Later, reveal and score the corresponding outcomes.
actual = demand[origins[:, None] + horizons[None, :]]
absolute_error = np.abs(actual - forecast)
mae_by_horizon = absolute_error.mean(axis=0)

assert absolute_error.tolist() == [[1, 2, 6], [2, 1, 3], [1, 2, 2]]
assert np.allclose(mae_by_horizon, [4/3, 5/3, 11/3])
print("MAE by horizon:", mae_by_horizon)
print("Overall MAE:", absolute_error.mean())

Keep the evaluation faithful as it grows

These three origins are a development backtest. After choosing the recipe and window policy, retain a later untouched period for final assessment when that is the agreed evaluation design. Fit every learned transform using eligible history at its origin; do not normalize the entire series once before backtesting.

If origins are close enough that target windows overlap, the same actual observation can be scored at several horizons. That can be intentional: they are forecasts issued with different information. Keep their origin and horizon labels, and do not treat the resulting errors as independent observations when estimating uncertainty.

Even our non-overlapping target blocks can share seasonal conditions, and their training histories overlap. Report the date range, horizons, update schedule, baseline, and aggregation rule. The nested-CV chapter explains how to evaluate a procedure that chooses among candidate forecasting recipes while maintaining an outer boundary.

References and further reading

• Hyndman and Athanasopoulos: time-series cross-validation — rolling forecast origins and horizon-specific forecast evaluation.
• Forecasting: Principles and Practice — simple methods — the naïve forecast used as a transparent baseline.
• Scikit-learn: TimeSeriesSplit — expanding windows, maximum training size, and row-count gaps.