Repeated K-Fold

September 2026 · 8 min read

One K-fold run depends on which examples were grouped together. Repeated K-fold performs the complete K-fold process again with a different random partition, then combines the results. It helps reveal how sensitive an estimate is to that partition choice.

The dataset stays the same. Repeating the split provides more views of these examples; it does not collect new independent examples from the population. That distinction matters when interpreting the resulting spread of scores.

Repeat the partition, not just the model fit

In ordinary K-fold CV, every example is validated once and trains the other fold models. In repeated K-fold, that complete coverage happens once per repeat.

Use 12 independent examples labeled A–L. Four folds leave three examples for validation and nine for training in each fit. Three repeats require 4 × 3 = 12 fits. Here are the partitions generated by RepeatedKFold(n_splits=4, n_repeats=3, random_state=19):

Follow example A. It is validated in fold 2 of repeat 1, fold 3 of repeat 2, and fold 1 of repeat 3. Its training companions changed, so the model predicting A can change too. Within any one repeat, A is never used to fit the model that validates it.

First summarize each complete repeat

For a reproducible illustration, fit a ridge regression model with fixed regularization strength 1. This is a linear predictor whose fit penalizes large coefficients. The code supplies one numerical feature and 12 constructed target values; the feature is not a time index. We use a deterministic fit to isolate variation from the partitions.

Each fit receives nine training examples and predicts three held-out targets. Compute mean absolute error (MAE), the average magnitude of the prediction mistakes. The resulting fold errors are:

The three repeat means are approximately 0.7184, 0.8738, and 0.8108. Averaging them gives 0.8010 target units. Because every fold has three validation examples, averaging all 12 fold MAEs gives the same number.

We average complete repeats so the result is easy to interpret: each repeat covers every example once. With unequal fold sizes, decide whether the desired score weights folds equally or gives every example equal weight. The K-fold aggregation explanation covers that distinction.

More repeats answer a narrower question

There are two sources of uncertainty to separate. One is the partition of the dataset you already have. Another is which dataset you happened to collect from the population. Repeating CV explores the first while holding the second fixed.

Averaging over additional randomized partitions can make the reported mean less sensitive to a particular partition. It need not move monotonically upward or downward as you add repeats. It also cannot reveal a customer type or failure case absent from the dataset.

Twelve fold scores are not twelve independent experiments. Training sets overlap, and the same target values appear in validation across repeats. Dividing the fold-score standard deviation by the square root of 12 does not automatically produce a valid standard error for population performance. Bengio and Grandvalet explain why correlations make CV variance estimation difficult.

Report the repeat means as a descriptive measure of split sensitivity, alongside the dataset size and task. Repeating until one run looks especially good and reporting only that run defeats the purpose.

Keep one prediction per row per repeat

The code stores a matrix with three rows, one per repeat, and 12 columns, one per example. This makes coverage visible and prevents later predictions from silently overwriting earlier ones.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import RepeatedKFold

# Independent examples; x is a numerical feature, not a timestamp.
X = np.arange(12, dtype=float).reshape(-1, 1)
y = np.array([0, 1, 1, 3, 2, 5, 4, 7, 6, 7, 8, 10], dtype=float)
ids = np.array(list("ABCDEFGHIJKL"))
K, R = 4, 3
splitter = RepeatedKFold(n_splits=K, n_repeats=R, random_state=19)
splits = list(splitter.split(X))
errors = np.empty((R, K))
predictions = np.full((R, len(y)), np.nan)
seen = np.zeros((R, len(y)), dtype=int)

for i, (train, valid) in enumerate(splits):
    repeat, fold = divmod(i, K)
    model = Ridge(alpha=1).fit(X[train], y[train])
    pred = model.predict(X[valid])
    errors[repeat, fold] = mean_absolute_error(y[valid], pred)
    predictions[repeat, valid] = pred
    seen[repeat, valid] += 1
    print(repeat + 1, fold + 1, ids[valid].tolist())

assert len(splits) == R * K
assert np.all(seen == 1)  # Once per row IN EACH repeat.
assert np.isfinite(predictions).all()
repeat_mae = errors.mean(axis=1)
print("MAE by repeat:", repeat_mae)
print("Overall mean MAE:", repeat_mae.mean())

# Equal fold sizes make these two averages agree.
assert np.allclose(repeat_mae, np.abs(predictions - y).mean(axis=1))

The splitter produces all four folds of one repeat before the next, so divmod(i, K) recovers the repeat and fold numbers. seen == 1 checks each repeat separately. Across all repeats, each example is validated three times.

Make the repetitions reproducible without making them identical

A fixed integer seed gives a reproducible sequence of randomized partitions. RepeatedKFold advances its random generator internally as it makes the repeats.

By contrast, recreating KFold(shuffle=True, random_state=19) inside a manual loop restarts the generator each time. It yields the same partition repeatedly. If the model fit is deterministic too, those extra runs add no new information about split sensitivity.

For a randomized model, training randomness is another moving part. State whether it is held fixed or varied, and keep comparisons paired on the same split list. The seed and saved-split guide explains how to preserve assignments beyond an in-memory experiment.

Use repeats where the base split makes sense

For classification with rare labels, RepeatedStratifiedKFold retains the class-balancing constraints in each repeat. This preserves coverage; it does not manufacture more positive examples.

For grouped observations, use a design that repeats suitable group partitions. Ordinary RepeatedKFold does not enforce group boundaries. Forecasting still needs chronological evaluation; repeatedly shuffling dates would answer the wrong question.

Choose repeats according to the question and available fit budget. More repeats cost more while leaving each training-fold size unchanged. Increasing the number of folds instead changes the training fraction; that is a separate decision covered in choosing k.

If repeated CV is used to select hyperparameters, its winning score is still a selection score. Preserve an outer assessment or final test as described in nested CV. After selection, fit the chosen recipe on the intended training data; repeated CV does not automatically create or deploy an ensemble.

References and further reading

• Scikit-learn: RepeatedKFold — randomized repetitions, deterministic seeds, and the total number of splits.
• Scikit-learn: RepeatedStratifiedKFold — repeated partitions with class-wise stratification.
• Bengio and Grandvalet (2004): variance of K-fold CV — why overlapping training and test sets invalidate naive independence assumptions.