K-Fold Cross-Validation

September 2026 · 8 min read

K-fold cross-validation gives every development example a turn as held-out data. The important detail is that every turn needs a fresh fit. We are not moving one trained model between validation sets; we are evaluating the same training recipe on several different training subsets.

The hold-out comparison explains when that extra work is useful. Here we follow the mechanics completely, using a baseline simple enough to fit by hand.

One recipe, several fitted models

Suppose we have six delivery times, measured in hours. Call the rows A through F, with times 1, 3, 2, 4, 5, 3. Our baseline predicts the mean delivery time in its training data. There are no input features to learn from; the point is to see exactly which rows determine a prediction.

Use three folds: A–B, C–D, and E–F. For each round, fit the mean on the other four rows, predict the two held-out rows, and measure their errors. These fixed tiny folds make the arithmetic visible; they are not a recommendation about validation sample size.

Turn held-out predictions into a score

We will use mean squared error (MSE): subtract a prediction from the observed delivery time, square the difference, then average those squared errors. Squaring makes large mistakes count more, and the result has units of hours squared. Lower is better.

In the first round, the prediction is 3.5 hours for both held-out rows. Their observed times are 1 and 3 hours, so:

Let $K$ be the number of folds and $e_k$ the MSE in fold $k$. The usual unweighted cross-validation summary averages the fold scores:

This 2.5 describes the held-out errors of three models trained on four rows each. It is not the training error of the model fitted on all six rows, nor is it a guarantee about its next delivery.

Choose a fold and inspect the fit

Keep each prediction with its original row

After all rounds, the held-out predictions in original row order are [3.5, 3.5, 3, 3, 2.5, 2.5]. These are called out-of-fold predictions: each row is predicted by the model fitted without its fold.

Because our folds have equal size and squared error averages over rows, averaging the six squared errors also gives 2.5. With unequal fold sizes, an unweighted mean of fold MSEs gives small folds more influence per row. To give every row equal weight, weight each fold MSE by its number of validation rows—or compute MSE from the complete out-of-fold prediction array.

This equivalence depends on the metric. Averaging fold F1 scores or AUCs is not generally the same as computing the metric on pooled predictions. Preserve which fit produced each prediction and state how the results are aggregated.

Run the complete loop

The following code reproduces the table and checks that every row is validated exactly once. clone creates an unfitted copy of the recipe for each round. DummyRegressor learns the mean of the labels passed to its own fit call.

import numpy as np
from sklearn.base import clone
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold

X = np.zeros((6, 1))
y = np.array([1., 3., 2., 4., 5., 3.])
cv = KFold(n_splits=3, shuffle=False)
recipe = DummyRegressor(strategy="mean")
oof = np.full(len(y), np.nan)
seen = np.zeros(len(y), dtype=int)
errors = []

for train, val in cv.split(X):
    model = clone(recipe).fit(X[train], y[train])
    oof[val] = model.predict(X[val])
    seen[val] += 1
    errors.append(mean_squared_error(y[val], oof[val]))

assert np.all(seen == 1)
assert np.isfinite(oof).all()
assert np.allclose(errors, [3.25, 1.0, 3.25])
assert np.isclose(mean_squared_error(y, oof), 2.5)
print("Mean fold MSE:", np.mean(errors))

The unshuffled folds are intentional for this hand calculation. KFold defaults to no shuffling. For independent rows whose order has no intended evaluation meaning, consider shuffle=True with a recorded seed. For grouped or time-dependent data, change the splitter instead of assuming shuffling makes the rows independent.

The fitted object is the whole pipeline

For a real model, replace the dummy recipe with a pipeline containing every learned step. Each fold must fit its own imputer, scaler, feature selector, and estimator on that fold’s training portion. Calling fit_transform once on the full dataset before the loop defeats the held-out design.

Training data in different rounds overlap, so the fitted models and their scores are related. The spread of three or five fold scores is a useful description, but it is not automatically the standard error of the final model’s performance. The choice-of-K chapter explains the distinction without treating folds as independent repeats.

What model do we use after validation?

If cross-validation helps select a recipe, fit that selected recipe on the full development set, then evaluate it using the independent final test or the agreed outer evaluation. In our toy example, a final mean fit on all six rows predicts 3 hours. It is a new fit with more training data than any validation fit.

The cross-validation loop does not automatically define an ensemble of its fold models. You can deliberately design such an ensemble, but that is a separate prediction procedure requiring its own evaluation. Continue to stratified K-fold for classification data where rare classes can otherwise disappear from a fold.

References and further reading

• Scikit-learn: KFold — fold sizes, shuffling, and the generated train/validation indices.
• Scikit-learn: evaluating estimator performance — fitting transforms inside folds and interpreting out-of-fold predictions.
• Scikit-learn: DummyRegressor — the mean-prediction baseline used in the reproducible example.