Choosing k: Bias and Variance

September 2026 · 8 min read

The number of folds changes three things at once: how much data each model sees, how many validation examples support each score, and how many fits you must run. Choose it after deciding what must be held out and which metric matters.

Five or ten folds are common starting points for independent examples. They are not universal optima. A dataset with few positive cases, repeated customers, or strict forecasting dates can make another design more informative.

Start with counts you can inspect

Suppose there are 1,000 independent examples, including 20 positives, and a search over 12 fixed candidate recipes. With five stratified folds, each fit uses 800 training rows and validates on 200, including four positives. Ten folds increase training size to 900, but leave only two positives per validation fold.

The change from five to ten folds adds 100 training rows per fit and doubles the candidate-fit count. The change from ten to twenty adds only 50 training rows per fit and doubles it again. Whether that extra data is useful depends on the learning procedure.

For equally sized folds, write $N$ for the dataset size and $K$ for the number of folds. One fold holds $N/K$ examples, leaving the rest for training:

If $N$ is not divisible by $K$, ordinary K-fold sizes differ by at most one. Use the actual counts when checking class support and fit cost rather than relying only on the nominal fractions.

Which bias are we talking about?

A CV score evaluates models trained on the training-fold size. Usually, the model we intend to use afterward is trained on all available development data. These are different training sizes.

If the learning procedure improves with additional training examples, CV on a smaller training fraction can overestimate the expected error of the final, larger-data fit. Increasing $K$ reduces that training-size mismatch. Leave-one-out takes it to the limit: each fit uses $N-1$ examples.

This is about the bias of a performance estimate relative to its intended target. It is not the same as a model underfitting because its function class is too simple. Nor does increasing $K$ remove the selection bias from trying many recipes and reporting the winning score. The nested-CV chapter handles that separate issue.

Do not assume every additional training example improves every realized fit. Training-size reasoning helps explain a tendency; it is not an exact correction you can subtract from a measured CV score.

Why more fold scores do not settle uncertainty

Increasing $K$ makes each validation fold smaller. An individual score then rests on fewer outcomes. At the same time, the training sets overlap more strongly. With ten folds, two training sets share eight of the ten blocks; their fitted models and errors can be closely related.

Averaging more scores can offset some variation, but these are dependent scores whose properties also change with $K$. That is why there is no general rule that the CV estimate’s variance must decrease, or must increase, whenever $K$ increases.

Three quantities are often confused: variation among the fold scores on this dataset, variation under new random partitions of this dataset, and variation if we collected a new dataset from the population. They answer different questions. Repeated K-fold explores partition sensitivity; Bengio and Grandvalet explain why CV uncertainty is more difficult than treating folds as independent measurements.

Let the metric impose a practical limit

Return to the 20-positive example. Recall is the fraction of actual positive cases detected. With four validation positives, one additional detection changes recall by 25 percentage points. With two positives it changes recall by 50 points; with one, recall is either 0% or 100%.

Twenty folds can put a positive in every fold, yet each fold’s recall remains extremely coarse. The minimum class count bounds how many stratified folds can all contain that class; merely satisfying that bound does not make the metric precise.

If you need more positive cases per fold, fewer folds may be more useful than a larger training fraction. Repeating the split does not create additional positive observations. Consider the stratified-CV limitations before treating a stable-looking average as strong evidence.

Other metrics impose different requirements. AUC needs both classes, and R² cannot be computed on singleton validation sets. The leave-one-out metric discussion shows why the most extreme fold count can be unsuitable even when fitting is inexpensive.

Count work at the level of the complete procedure

For $M$ candidates and $K$ folds, a direct grid search performs $MK$ candidate fits, then normally one refit of the winner. Repeating each CV run $R$ times changes the candidate count to $MKR$. A nested search adds its outer rounds and refits too.

These counts are planning tools, not runtime estimates. A fit on 950 examples may cost more than one on 500. Preprocessing, memory, parallelism, and estimator-specific shortcuts can dominate the difference. Time a representative fit before committing to an expensive search.

The following audit reproduces every count in the figure. It checks split sizes and positive counts without fitting any models:

import numpy as np
from sklearn.model_selection import StratifiedKFold

X = np.zeros((1000, 1))
y = np.array([1] * 20 + [0] * 980)
candidates = 12

for k in [2, 5, 10, 20]:
    cv = StratifiedKFold(k, shuffle=True, random_state=7)
    sizes = []
    for train, valid in cv.split(X, y):
        sizes.append((len(train), len(valid), int(y[valid].sum())))
    assert set(sizes) == {(1000 - 1000 // k, 1000 // k, 20 // k)}
    print(k, "folds:", sizes[0], "candidate fits:", candidates * k)

Some constraints come before k

Groups: with 12 independent customers, fifteen group folds are impossible, no matter how many measurement rows exist. Unequal customer sizes can also produce very different validation-row counts. Decide whether the target is performance per row or per customer; the group-CV chapter separates those averages.

Time: forecasting folds represent issue dates and future evaluation windows. The forecast horizon, label availability, seasonality, and minimum history matter more than matching a conventional fold count. Use the forecast-origin design before adjusting how many origins to evaluate.

Large datasets: a well-designed holdout may already contain ample validation cases, while many large fits add substantial cost. The holdout comparison explains when that simpler assessment is useful.

A practical decision sequence

1. Choose the exclusion unit. Identify which rows, people, or future periods must be absent from the fit.
2. Check validation support. Count the cases needed by the metric, including rare classes or meaningful groups.
3. Check the training regime. Make sure each fit has enough eligible data to represent the intended learning procedure.
4. Set a fit budget. Count candidates, folds, repeats, and any outer loop before running the search.
5. Inspect sensitivity when it matters. Compare plausible designs using development data, preserving the final evaluation boundary.

Choose $K$ for a useful evaluation design, rather than picking whichever fold count gives the most flattering score. Report the choice, actual fold sizes, metric, and any repeats so the estimate can be interpreted in context.

References and further reading

• Scikit-learn: cross-validation guide — five- and ten-fold conventions, leave-one-out tradeoffs, and structured-data alternatives.
• Scikit-learn: KFold — the actual sizes of folds when the sample count is not divisible by k.
• Scikit-learn: StratifiedKFold — class allocation within folds.
• Bengio and Grandvalet (2004): variance of K-fold CV — the dependence problem behind naive CV variance estimates.