Hold-Out vs Cross-Validation
You have enough data to train two candidate models, but only one small validation set. Model A wins today. Would it still win if a few different examples had been reserved for validation? That is the practical reason to consider cross-validation.
A hold-out evaluates candidates on one fixed reserved subset. Cross-validation repeats fitting and evaluation with several training/validation partitions. The choice changes how development data are used and how many fits are needed. It does not change the need for a suitable boundary or an independent final assessment.
Separate model selection from final evaluation first
Suppose we have 1,000 independent examples. Reserve 100 for a final test and use 900 for development. The decision in this chapter concerns those 900 development rows. We can divide them once, or rotate which development rows validate each fit. The final 100 stay outside both procedures.
Cross-validation libraries often call a fold’s held-out rows “test.” If we use their scores to choose a model, those rows serve the validation role. The three-set chapter explains why a name in an API does not determine whether an assessment is independent.
One hold-out versus three rotating folds
Divide the development data into three blocks of 300 rows, A, B, and C. A single hold-out might train on B and C and validate on A. Three-fold cross-validation adds the other two possible held-out blocks:
A validation row may be training data in a different round. That is expected. What matters is that its own validation prediction comes from a fit that did not use it. Carrying a fitted scaler or classifier from one round into the next would break this separation.
The ranking can depend on the held-out block
These illustrative accuracies show why looking beyond one split can help:
| Candidate | Validate A | Validate B | Validate C | Mean |
|---|---|---|---|---|
| Model A | 82% | 78% | 80% | 80% |
| Model B | 80% | 82% | 81% | 81% |
On the first hold-out, Model A wins by two percentage points. Across the three rounds, Model B has the higher mean. Its calculation is simply $(80+82+81)/3=81\%$. The mean describes performance across these validation fits; it is not the accuracy of a single model trained on all 900 rows.
Every candidate must use the same folds for this comparison. Otherwise, a difference in scores might reflect a difference in evaluation examples. A one-point average advantage also does not establish that Model B is reliably better. Consider the size and consistency of differences, evaluation sample size, training cost, and the needs of the application.
Should we average fold scores or pool predictions?
For accuracy with equally sized folds, the two agree: averaging the three fractions correct is equivalent to counting correct held-out predictions across all 900 rows. With unequal fold sizes, an unweighted fold mean gives small folds more influence per row.
For nonlinear metrics such as F1 or AUC, averaging fold scores need not equal computing a metric on pooled predictions. AUC comparisons across predictions from different fitted models can introduce additional interpretation problems. State the aggregation rule and choose it for the metric and evaluation unit.
Compare the cost at the same training size
In this example, one hold-out needs one 600-row fit per candidate. Three-fold CV needs three 600-row fits per candidate. For two candidates, that is two versus six development fits. After selection, fitting the chosen recipe on all 900 development rows adds one more fit.
A larger number of folds uses a larger fraction of the development data in each fit, but also requires more fits. It is not a free route to a more reliable result. For expensive models or large datasets, one well-designed validation block may already provide enough evidence. For small datasets, rotating folds can make better use of limited examples.
| Situation | Reasonable starting design | Reason |
|---|---|---|
| Large development set; expensive fits | One substantial validation hold-out | Enough evaluation examples may be available without repeated fitting. |
| Limited independent examples; affordable fits | A small number of cross-validation folds | Every example can help train some fits and validate another. |
| Unseen users, devices, or documents | Group hold-out or group CV | The unfamiliar unit must remain separated in every fit. |
| Future predictions | Later validation blocks or rolling-origin evaluation | Training must respect chronology and label availability. |
Decide between group, time, and ordinary row boundaries before choosing the number of folds. Repeating an invalid split can make the wrong answer look more stable.
Keep preprocessing inside the search
The runnable example below uses a separate synthetic classification dataset. It evaluates three regularization strengths for logistic regression with three-fold validation. Smaller C means stronger regularization; the point here is the evaluation structure, not a claim about which value is best in general.
from sklearn.datasets import make_classification
from sklearn.model_selection import (
GridSearchCV, StratifiedKFold, train_test_split
)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=1000, random_state=9)
X_dev, X_test, y_dev, y_test = train_test_split(
X, y, test_size=100, random_state=42, stratify=y
)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=43)
pipeline = make_pipeline(
StandardScaler(), LogisticRegression(max_iter=1000)
)
search = GridSearchCV(
pipeline, {"logisticregression__C": [0.1, 1.0, 10.0]},
cv=cv, scoring="accuracy", refit=True
)
search.fit(X_dev, y_dev)
print("Chosen setting:", search.best_params_)
print("Development CV accuracy:", search.best_score_)
# Final evaluation after the search and refit are complete.
print("Final test accuracy:", search.score(X_test, y_test))There are nine validation fits—three settings times three folds—plus one refit on all 900 development rows. Each fit learns its own scaler from its training portion. The chosen refitted pipeline is what search.score evaluates on the final 100 rows. GridSearchCV’s documentation describes these search and refit roles.
Cross-validation still participates in selection
The best CV score was chosen because it was the largest among the candidates. It can therefore be optimistic about the selected procedure, especially after a large or adaptive search. Cross-validation does not make the winning development score independent of model selection.
Use the reserved final test to assess the chosen refitted system. When the aim is to assess an entire tuning procedure without relying on one final hold-out, nested cross-validation uses an outer held-out loop and performs the full search inside each outer training partition. The nested-CV example illustrates the distinction.
Do not read overlapping fold scores as independent experiments: their training sets share data. Report the folds and aggregation honestly. If the comparison remains inconclusive, better evaluation data can be more valuable than adding more repeats of nearly the same data.
Continue to the cross-validation series for the detailed fold algorithms. This splitting series’s overview brings the design choices together before the next preprocessing stage.
References and further reading
- Scikit-learn: cross-validation — fold construction, paired evaluation, preprocessing, and metric aggregation.
- Scikit-learn: GridSearchCV — candidate evaluation and refitting the selected pipeline.
- Scikit-learn: nested versus non-nested CV — the difference between selecting a model and evaluating the selection procedure.