Time-Series Splitting

September 2026 · 9 min read

To judge a forecast, reconstruct the moment it would have been made. At that moment, which measurements had arrived? Which outcomes were known? Which model version was available? A random row split usually cannot answer those questions because it mixes earlier and later information.

A time split trains on the past and evaluates on later predictions. It must also respect feature and label availability. This chapter builds a daily example with delayed labels, then turns it into a reproducible evaluation schedule.

Start with the prediction clock

Suppose a model predicts demand at the beginning of each day. Yesterday’s completed sales can be an input; today’s final sales cannot. A field named “yesterday’s sales” may still be unavailable if the data arrives two days late. The relevant timestamp is when a value became usable, not just the date the value describes.

Write down the prediction time, forecast horizon, data arrival delays, and retraining schedule before choosing dates for the split. A one-day-ahead forecast retrained every morning and a 30-day forecast issued once a month are different evaluation tasks.

Reserve a later period for the final assessment

A basic design uses earlier dates for development and a later block for test. Within development, choose models using earlier-to-later validation splits. Keep the final period outside those choices, just as in the three-set workflow.

Choose blocks that reflect the use case. A one-week test window may miss monthly patterns; a holiday-only window may answer a narrower question than everyday operation. If you evaluate several periods, show their dates and scores rather than hiding a steadily worsening result inside one average.

Chronology does not promise an easy or perfectly representative test. A future period may contain changed behavior, equipment, or demand. Measuring that difficulty is often the point. Forecasting: Principles and Practice describes evaluation with a moving forecast origin and explains why the horizon should match the intended forecasts.

A gap can keep unfinished outcomes out of training

Use integer day numbers for a small example. A prediction made at the start of day $t$ is evaluated on the next three complete days: $t+1$, $t+2$, and $t+3$. Its label is known only after day $t+3$ ends. Assume there is no additional reporting delay.

If a model is fitted just before day 9 begins, the prediction row from day 5 has a usable label: its outcome window ends on day 8. The row from day 6 does not: its outcome includes day 9, which has not happened yet.

For this exact convention, exclude prediction rows 6, 7, and 8 from training before evaluating days 9–11. This is a three-row gap. The labels of the validation rows are revealed later for scoring; they are not available to the model when those predictions are issued.

Move the fitting date forward

One validation block can be unusually easy or hard. Repeat the experiment at later dates, fitting a fresh model using only the eligible history each time. In an expanding window, old eligible rows remain available as training grows:

A rolling window instead drops older training rows and keeps a fixed amount of recent history. That can help when old patterns become misleading, but it uses less data. Choose the window length using chronological development results. Do not choose it by repeatedly comparing final test scores.

Make the schedule executable

The example below reserves prediction days 18–20 for the final test and uses days 0–17 for development. TimeSeriesSplit produces the three validation rounds above. It calls each held-out block “test”; during model selection, those blocks serve the validation role.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

day = np.arange(21)
dev, test = day[:18], day[18:]

# A day-t target needs the complete days t+1, t+2, t+3.
# Fit before a cutoff day begins: last_outcome_day must be smaller.
last_outcome_day = day + 3

cv = TimeSeriesSplit(n_splits=3, test_size=3, gap=3)
for train_pos, val_pos in cv.split(dev):
    train, val = dev[train_pos], dev[val_pos]
    cutoff = val.min()
    assert last_outcome_day[train].max() < cutoff
    print(train.tolist(), "→", val.tolist())

# Refit the chosen recipe before the final test period begins.
final_train = dev[last_outcome_day[dev] < test.min()]
assert final_train.tolist() == list(range(15))  # Days 0–14
assert set(final_train).isdisjoint(test)

The final fit cannot use development prediction rows 15–17 yet: their outcome windows have not all ended before day 18. To score the complete final test, the offline dataset must eventually contain outcomes through the end of day 23. This example checks the schedule; it does not train a predictor.

TimeSeriesSplit operates on row positions and does not sort timestamps for you. Its gap and test_size are counts of rows. Sort and validate the input order first; for irregularly spaced events or many rows per timestamp, build date-based boundaries and check availability explicitly.

Feature construction must follow the same clock

A correct split cannot repair a feature that already contains the future. Consider a moving average of demand: a trailing average of completed days may be usable; a centered average includes later observations and is not available for a live forecast. Shift the series before computing a lagged statistic when the current value is still unknown at prediction time.

Fit learned scaling, imputers, feature selectors, and model parameters on the eligible training portion of each round. Fixed causal lag construction can use earlier raw observations as context, including observations before the validation boundary. A historical value appearing in a validation feature is not automatically leakage; the question is whether it was available when that prediction was made. Scikit-learn’s lagged-feature example illustrates causal features and contrasts temporal evaluation with a shuffled split.

Report the time boundary, not just the score

Alongside the metric, record training and evaluation dates, the prediction horizon, retraining frequency, label delay, and number of evaluated forecasts. Show scores by relevant horizon or period when they reveal materially different behavior. Compare with an appropriate simple forecast, such as the last observed value or a seasonal counterpart.

Forecast errors can share events or seasonal conditions, so treating every error as an independent observation can overstate precision. Overlapping target windows especially deserve care. Chronological validation gives a more faithful experiment; it does not remove dependence between its errors.

If the system must handle both future periods and unseen entities, combine time and group constraints. The next chapter brings these boundaries together as a practical data-leakage audit.

References and further reading

• Hyndman and Athanasopoulos: time-series cross-validation — rolling forecast origins, training on prior observations, and matching the forecast horizon.
• Scikit-learn: TimeSeriesSplit — expanding training sets, row-count gaps, test sizes, and equal-spacing assumptions.
• Scikit-learn: lagged features for forecasting — a worked forecasting pipeline with causal lagged inputs and temporal evaluation.