Cross-Validation

September 2026 · 8 min read

Cross-validation evaluates a learning procedure through several held-out fits. Each round trains a fresh model on one subset and measures its predictions on another. This lets us compare recipes and inspect performance across different held-out examples.

The first design decision is still what must be unseen at prediction time: a new row, a new customer, or a future period. Cross-validation repeats that boundary. It cannot repair a boundary that answers the wrong deployment question.

This guide gives the common structure and a route through ten focused chapters. Start with the fold rotation below, then choose the chapters that match your data and evaluation task.

One recipe, five separate fits

Suppose we have 1,000 independent development examples. Divide them into five blocks of 200. In round 1, fit the complete recipe on blocks 2–5 and validate on block 1. In round 2, create a fresh fit using blocks 1, 3, 4, and 5, then validate on block 2. Continue until each block has been held out once.

Every model sees 800 training examples. Every example receives one prediction from a model fitted without its block. The result is five fitted models and five validation scores, rather than one model that continues training through all five rounds.

Accuracy is the fraction of validation examples predicted correctly. Here the five fold accuracies are 90%, 87%, 91%, 84%, and 88%. Their equally weighted mean is

Because the folds have equal size, this is also 880 correct held-out predictions out of 1,000. That equivalence needs care with unequal fold sizes or metrics such as F1 and AUC. The K-fold chapter explains the aggregation rule and follows a fully executable six-example fit.

Read the estimate at the right level

The 88% summarizes this recipe’s held-out performance when each model trains on 800 examples. A later model fitted on all 1,000 uses more training data; the CV number is not a measurement of that exact fitted model.

The illustrated fold scores range from 84% to 91%. That spread describes these five evaluations. It is not a guarantee that future accuracy lies in that range, and it is not automatically a confidence interval. Training sets overlap, and the original dataset is still one sample from the population.

Using several splits can reduce sensitivity to one particular split. How variable the overall estimate is depends on the data, learner, fold count, and repetition design. The fold-count chapter explains these tradeoffs without assuming that more folds always produce a more reliable estimate.

Choose the split to match the prediction

These constraints can combine. A group split may also need usable class counts; a customer forecast may require both group and time boundaries. Start from the intended prediction and inspect the generated indices rather than relying only on the splitter’s name.

Fit the whole recipe within each training fold

The fitted object includes preprocessing. If a scaler learns a mean, an imputer learns replacement values, or feature selection uses labels, that learning must happen within the training portion of the current fit. Apply the resulting transform to its validation examples without refitting it there.

A pipeline makes this boundary easier to maintain. The leakage experiment demonstrates the effect on randomly generated labels, while the tuning chapter shows a complete pipeline search. Merely placing an already-transformed dataset inside a CV loop does not undo information learned from the validation rows.

Model selection needs another boundary. If these CV scores choose the best recipe, the winning score also participated in the choice. Evaluate the frozen choice on an untouched final test set, or use nested CV to put the entire search inside each outer-training fold. Both are useful designs when their data boundaries match the task.

Read the chapters by the question you need to answer

1. K-fold: what is actually fitted? Select a fold, recompute its training mean, and collect every held-out prediction.
2. Stratified folds: where do the positives go? Inspect exact class allocations when the counts do not divide evenly.
3. Group folds: what is the unseen unit? Allocate unequal device groups and distinguish row-weighted from group-weighted evaluation.
4. Forecast origins: what was known then? Issue three horizons from one origin and score each horizon separately.
5. Nested CV: who evaluates the search? Follow 900 examples through inner selection, outer assessment, and a final refit.
6. Tuning: how do we read a candidate comparison? Run six recipes on shared folds and interpret negative error scores.
7. Repeated K-fold: what changes across repetitions? Track complete coverage and separate split sensitivity from new-data uncertainty.
8. Random holdouts: which rows were never validated? Audit repeated appearances and calculate their coverage probabilities.
9. Leave-one-out: what happens when one point is removed? Refit a line interactively and examine metric and cost limitations.
10. Choosing k: which tradeoff matters here? Compare training fractions, rare-class support, and the complete search budget.

Finish with an interpretable report and a separate refit

Record the recipe, split assignments, fit sizes, metric, fold scores, and aggregation rule. State whether the scores selected the recipe or evaluated a previously fixed procedure. That context is more useful than a CV mean without its experimental design.

Once selection is finished, refit the chosen recipe on the intended training data. Cross-validation does not automatically average its fold models into the deployed predictor. If you want an ensemble, define and evaluate that procedure explicitly.

References and further reading

• Scikit-learn: cross-validation guide — splitter choices, fitting transformations within folds, and out-of-fold predictions.
• Cawley and Talbot (2010): selection bias — why the best selection score is not independent evidence about the selected recipe.
• Bengio and Grandvalet (2004): CV variance — the dependence between fold errors and the limits of naive uncertainty estimates.