Nested Cross-Validation

September 2026 · 9 min read

You try several model settings, compare their cross-validation scores, and keep the best one. Can that winning score also serve as your final performance estimate? It helped choose the winner, so it is no longer an independent check of the choice.

Nested cross-validation evaluates the whole selection procedure. An inner loop chooses the settings. An outer loop tests the result on examples that were excluded from that choice. If rotating folds is unfamiliar, start with the six-example K-fold walkthrough; this chapter adds a second boundary around the same operation.

The score can be overfitted too

A validation score reflects both the quality of a candidate and the particular examples in the validation set. When we choose the largest score among many candidates, we can favor a candidate that benefited from a fortunate set of examples. This can happen even when every individual model was fitted without its validation rows.

For instance, suppose three equally capable recipes would each average 80% on new data. On one development dataset, their estimated scores happen to be 78%, 81%, and 83%. Choosing the third recipe does not establish that its future accuracy is 83%. These are illustrative numbers, not results from the experiment below. The problem is selecting on a noisy estimate and then reporting that selected estimate as fresh evidence.

Cawley and Talbot study this form of overfitting in model selection. Nested CV supplies an evaluation boundary outside the selection process; it does not make validation noise disappear.

Follow one outer round with 900 examples

Assume 900 independent labeled examples from the population we want to predict. Use three outer folds. In the first round, set aside 300 examples as the outer test fold. The other 600 are the entire dataset available to that round’s search.

Split those 600 into three inner folds of 200. Each candidate is fitted three times: on 400 examples, then scored on the remaining 200. Average its three inner scores, select a candidate, and refit it on all 600. Only then score it on the 300 outer-test examples.

The names are relative: an inner validation fold helps select settings; an outer test fold evaluates that selection. Both are held out from the fit they score, but only the outer fold is held out from the whole search in its round.

What the outer scores measure

Our executable example uses logistic regression with three values of C: 0.1, 1, and 10. C controls inverse regularization strength; smaller values apply a stronger penalty to large coefficients. Each candidate includes a scaler fitted within its training fold. Accuracy is the declared selection and evaluation metric.

Running the code below with scikit-learn 1.9.1 gives these results on its synthetic dataset:

The mean outer accuracy is 75.44%: 679 correct predictions across 900 held-out examples. Equal fold sizes make the mean of fold accuracies equal to this pooled accuracy. Different settings can win in different rounds because the available training data changed.

The evaluated procedure is “scale, compare these three settings using this inner splitter and metric, then refit the winner.” It is not a claim that one particular value of C achieves 75.44%. Each outer model was trained after a search on 600 examples; the eventual model fitted using 900 may perform differently.

Keep the search inside the outer loop

GridSearchCV.fit receives only the outer-training slice. It therefore generates inner indices relative to those 600 rows. Its refit=True option then fits the winning pipeline on those same 600. Calling score on the outer-test slice evaluates that refitted model.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(
    n_samples=900, n_features=12, n_informative=6,
    n_redundant=2, flip_y=0.08, class_sep=0.8, random_state=12,
)
pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=2000)),
])
grid = {"model__C": [0.1, 1, 10]}
outer = StratifiedKFold(3, shuffle=True, random_state=7)
inner = StratifiedKFold(3, shuffle=True, random_state=17)

def new_search():
    return GridSearchCV(
        pipeline, grid, cv=inner, scoring="accuracy",
        refit=True, error_score="raise",
    )

scores, selected = [], []
seen = np.zeros(len(y), dtype=int)
for fold, (train, test) in enumerate(outer.split(X, y), start=1):
    assert len(train) == 600 and len(test) == 300
    assert not np.intersect1d(train, test).size
    search = new_search()
    search.fit(X[train], y[train])  # Inner search + refit on these 600.
    score = search.score(X[test], y[test])  # Only now use outer test.
    scores.append(score)
    selected.append(search.best_params_["model__C"])
    seen[test] += 1
    print(fold, search.best_params_, round(score, 4))

assert np.all(seen == 1)
print("Outer mean accuracy:", np.mean(scores))

# After assessment, train the deployment model using the same search.
final_search = new_search().fit(X, y)
final_model = final_search.best_estimator_
print("Final choice:", final_search.best_params_)

The seen check confirms that every row receives one outer evaluation. The pipeline ensures the scaler learns fresh means and scales inside each fit. Scaling all 900 rows before entering the loops would break that boundary.

Count fits before choosing the search budget

There are three candidates and three inner folds: nine fits per outer round. Refitting the chosen candidate adds one. Across three outer rounds, assessment needs 3 × (3 × 3 + 1) = 30 fits.

The final search on all 900 examples adds nine inner fits and one refit, bringing this example to 40 fits altogether. These counts describe model fits, not equal units of runtime: fitting on 900 examples can cost more than fitting on 400. Larger grids multiply this cost; choose the search budget before looking at outer results.

How to get the model you will actually use

After assessing the procedure, rerun the same search on all available development data and refit its winner. The code does this in final_search; here it chooses C=0.1. Do not automatically deploy the outer-fold model with the highest test score or choose settings by a majority vote over outer winners. Those are different selection procedures.

A separate untouched test set is another valid design: tune with CV on development data, freeze the procedure, then evaluate once on that test set. Nested CV is useful when you want several outer assessments while reusing a limited dataset for training. The holdout-versus-CV chapter compares these arrangements.

What the extra loop does not fix

Respect the prediction task at both levels. Repeated customers need appropriate group boundaries; forecasting needs chronological origins and eligible information. Nesting two unsuitable row splitters does not produce a realistic deployment estimate.

The protected selection process includes feature choices, preprocessing, thresholds, and early-stopping rules whenever they are learned from the data. If you inspect outer results, change the feature set or search space, and rerun until satisfied, the outer scores have entered your development process too.

Finally, the outer scores share overlapping training data. Their spread describes variation across these folds; it is not automatically a confidence interval for deployment performance. Keep the dataset, splitting rules, complete search space, metric, outer scores, and final refit procedure in the report.

References and further reading

• Cawley and Talbot (2010): overfitting in model selection — why selecting on a noisy estimate can bias subsequent performance reporting.
• Scikit-learn: nested versus non-nested cross-validation — an independent executable example separating search from outer evaluation.
• Scikit-learn: GridSearchCV — candidate search, scoring, and refitting on the dataset passed to fit.