Cross-Validation for Hyperparameter Tuning

September 2026 · 9 min read

A hyperparameter is a choice you supply to a learning procedure: a tree’s maximum depth, a regularization strength, or how many neighbors to consult. Cross-validation compares complete candidate recipes using held-out predictions, then selects one according to a declared rule.

This chapter follows one small search from its candidate grid to its final refit. The nested-CV chapter explains how to evaluate a search independently; here the emphasis is on running the search correctly and understanding the numbers it returns.

Define the recipe before pressing fit

We will predict a numerical target using nearest-neighbor regression. For a new point, the model finds nearby training examples and combines their target values. With uniform weighting it takes their ordinary average; with distance weighting, nearer examples count more.

Our grid tries 3, 5, or 15 neighbors and both weighting rules. That is 3 × 2 = 6 candidates. The number of neighbors belongs to the model; the number of CV folds controls evaluation. They are unrelated choices even though both are sometimes called “k.”

Distance also depends on feature scales. Every candidate includes a StandardScaler followed by the regressor. In the synthetic dataset below, the first feature carries a sine-shaped signal and the second is irrelevant noise on a much larger scale. Scaling prevents its units alone from dominating distance; it does not make an irrelevant feature useful.

Before searching, choose the metric and the data boundary. We use mean absolute error (MAE): for each held-out prediction, take the absolute difference from its true target, then average those differences. An MAE of 0.2 means an average miss of 0.2 target units. Smaller is better.

Compare candidates on the same examples

Reserve 24 of the 120 examples for final assessment. The other 96 are development data. Four-fold CV repeatedly fits a candidate on 72 development examples and validates it on 24. Reuse the same four partitions for all six candidates.

This lets us compare candidates on matching validation examples. If each candidate got unrelated folds, a score difference could partly reflect an easier draw of validation cases. Shared folds do not eliminate uncertainty, but they avoid that unnecessary source of variation.

Each of the 24 candidate–fold combinations fits its own pipeline. The scaler must learn from that fit’s 72 rows. It can then transform the 24 validation rows using those learned values. The leakage experiment shows what can go wrong when preprocessing is learned before the split.

Read the grid as six complete recipes

These results were produced by the code below with NumPy 2.5.3 and scikit-learn 1.9.1. Each bar averages four fold MAEs:

The selected recipe uses 5 neighbors with distance weighting, with mean CV MAE 0.2229. This is the smallest error among the six candidates we tried. It does not establish that these settings are optimal among all possible models or datasets.

Why does the library store negative error?

Scikit-learn’s search interface treats larger scores as better. Error metrics point in the opposite direction, so neg_mean_absolute_error returns the negative of MAE. An error of 0.22 becomes a score of −0.22, which ranks above −0.29.

GridSearchCV maximizes that negative score. Negate it once when reporting a positive error: mae = -search.best_score_. Do not choose the most negative score yourself; that would select the largest error.

The word test in cv_results_["mean_test_score"] refers to the held-out part of each CV split. In this example those are development validation folds. The separate 24-example final test set has not been passed to the search.

Look behind the winning average

The winning candidate’s four fold MAEs are 0.1928, 0.2483, 0.1817, 0.2687. Their average is 0.2229. Retaining these individual results makes it possible to check whether a candidate struggles on one particular fold.

std_test_score summarizes the spread of its fold scores. It is not automatically a standard error or confidence interval: folds share overlapping training data. A small difference between two mean scores is a selection result, not by itself strong evidence that one recipe will consistently win.

For near ties, a predeclared preference for lower prediction cost or simpler operation can be useful. Apply such a rule to development results and record it. Here we keep the ordinary minimum-MAE rule so the example remains reproducible.

Run the complete search

The double underscore in model__n_neighbors targets the regressor named model inside the pipeline. Materializing folds as a list makes the shared partitions explicit. Their indices are relative to the 96 development rows.

import numpy as np
from sklearn.model_selection import GridSearchCV, KFold, train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(17)
X = np.column_stack([
    rng.uniform(-3, 3, 120),
    100 * rng.normal(size=120),
])
y = np.sin(X[:, 0]) + 0.2 * rng.normal(size=120)
dev, test = train_test_split(np.arange(120), test_size=24, random_state=5)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", KNeighborsRegressor()),
])
grid = {
    "model__n_neighbors": [3, 5, 15],
    "model__weights": ["uniform", "distance"],
}
folds = list(KFold(4, shuffle=True, random_state=7).split(X[dev]))
search = GridSearchCV(
    pipeline, grid, cv=folds, scoring="neg_mean_absolute_error",
    refit=True, return_train_score=True, error_score="raise",
)
search.fit(X[dev], y[dev])

results = search.cv_results_
for i, params in enumerate(results["params"]):
    print(params, "CV MAE:", -results["mean_test_score"][i])

best = search.best_index_
fold_mae = [-results[f"split{f}_test_score"][best] for f in range(4)]
print("Selected:", search.best_params_)
print("Selected candidate's fold MAEs:", fold_mae)
print("Selected CV MAE:", -search.best_score_)

# The refit scaler used all 96 development rows, not the final test set.
assert np.allclose(search.best_estimator_["scale"].mean_, X[dev].mean(axis=0))
# Freeze the choice before this one final assessment.
test_mae = -search.score(X[test], y[test])
print("Final test MAE:", test_mae)

What is fitted when the search finishes?

With refit=True, the search takes the winning settings and fits a fresh pipeline using all 96 development rows. That pipeline is best_estimator_. It is neither an average of the four fold models nor the single fold model with the smallest error.

The search uses 6 × 4 = 24 fits to compare candidates, plus one refit: 25 fits in total. The final test call scores the already-refitted model; it does not fit another model.

After freezing the choice, the code obtains a final-test MAE of 0.1805 on the 24 reserved examples. It happens to be lower than the development CV error in this run. A final test score need not be worse: the evaluated cases differ, and the final model uses 96 training rows instead of 72. This small synthetic test is an illustration, not a precision benchmark.

Notice that search.score uses the negative-MAE scorer we supplied. Calling best_estimator_.score directly would instead use the regressor’s default R² metric. Choose the intended metric explicitly when evaluating outside the search object.

Keep a larger search manageable

A grid evaluates the Cartesian product of its value lists. Adding five values for another parameter would turn our six candidates into 30, with 120 CV fits before refitting. A randomized search can instead sample a fixed number of candidates from a declared space; the official search guide explains that option.

Use group or chronological splits when those match deployment; shuffling is appropriate here because the constructed examples are independent. Choose the scorer explicitly, inspect failed fits, and store the grid, split assignments, metric, library versions, and selected settings together. In the code, error_score="raise" makes a failed candidate visible while debugging.

Continue refining the search using development data if needed, but preserve the final evaluation boundary. If the final test result sends you back to change the recipe, that test has become development feedback. Return to the nested-CV design when you need to evaluate the full selection process across several outer splits.

References and further reading

• Scikit-learn: tuning estimator hyperparameters — grid construction, randomized search, and the components of a search procedure.
• Scikit-learn: GridSearchCV — cv_results_, selected parameters, scorer behavior, and the final refit.
• Scikit-learn: scoring conventions — why loss-based scorers carry a negative sign.
• Scikit-learn: KNeighborsRegressor — neighbor weighting and the estimator’s default R² score.