Stratified K-Fold

September 2026 · 7 min read

K-fold validation rotates the held-out rows, but ordinary K-fold does not look at their labels. If positive examples are uncommon, a validation fold can contain few or none. Some metrics then become extremely coarse or undefined, and some training folds may fail to contain every class.

Stratified K-fold adds a class-allocation rule to the rotation: distribute each class across the validation folds as evenly as the integer counts allow. The basic splitting intuition is in stratified splitting. This chapter focuses on what changes when every row must belong to exactly one of several folds.

Divide examples, not fractional people

Suppose a binary classification dataset has 23 independent examples: five positives and 18 negatives. We want four validation folds. Five positives cannot be divided into four identical integer counts, so one fold receives two positives and the other three receive one each.

A valid allocation has fold sizes 6, 6, 6, and 5. The following diagram shows the class counts, not particular row identities:

The first fold has a 33.3% positive rate and two others have 16.7%. Those differences do not mean stratification failed. With only five positives, the integer allocation limits how closely proportions can match. “Preserve class ratios” means approximate preservation, not exact equality.

How one complete rotation is constructed

First collect the row IDs for each class. If shuffling is enabled, randomize the order within each class. Allocate the class IDs across folds, coordinating the allocations so total fold sizes also stay nearly equal. A fold’s validation rows are its assigned IDs; its training rows are the complement.

Once the partition is formed, the usual fresh-fit loop applies. Each row validates exactly one fit. Stratification changes the partition, not the rule that preprocessing and training must be fitted without the current validation fold.

Scikit-learn’s StratifiedKFold also aims to keep the largest and smallest validation folds within one row of each other. Without shuffling, it preserves ordering relationships within the class allocation; it does not convert a time-ordered dataset into a forecasting evaluation.

Inspect the actual counts

This runnable example produces the diagram’s counts. The seed selects which identities go into the folds; the label counts determine how many of each class can be allocated.

import numpy as np
from sklearn.model_selection import StratifiedKFold

y = np.array([1] * 5 + [0] * 18)
X = np.zeros((len(y), 1))
cv = StratifiedKFold(n_splits=4, shuffle=True, random_state=7)
seen = np.zeros(len(y), dtype=int)
counts = []

for train, val in cv.split(X, y):
    negative, positive = np.bincount(y[val], minlength=2)
    counts.append((int(positive), int(negative)))
    seen[val] += 1
    assert set(train).isdisjoint(val)
    assert set(y[train]) == {0, 1}
    print("Validation:", len(val), "rows;",
          positive, "positive;", negative, "negative")

assert counts == [(2, 4), (1, 5), (1, 5), (1, 4)]
assert np.all(seen == 1)

Pass labels for the rows being split. If a final test set has already been reserved, call split(X_dev, y_dev) on the remaining development data. Do not accidentally use the full dataset’s label array with a subset of its features.

Check rare-class counts before choosing K

Let $n_c$ be the number of examples of class $c$, and let $K$ be the number of folds. If every validation fold must contain at least one example of every class, there must be enough examples to distribute:

This is a necessary counting condition, not a promise of reliable metrics. In our example, four folds are possible because the rare class has five rows. But three validation folds still contain only one positive each. Their recall is either 0% or 100%, depending on that single prediction. The two-positive fold has recall values 0%, 50%, or 100%.

If the rare class has only three examples, five validation folds cannot all contain it. Libraries may warn, fail, or return folds missing a class depending on the exact configuration. Check the counts explicitly instead of treating a successfully created splitter as proof that the evaluation is informative.

Reducing K can put more positives into each validation fold, at the cost of fewer rows for each training fit. It cannot create more total positive examples. Use the choice-of-K chapter to weigh that trade-off, and report when a class has too little evidence for a useful assessment.

Do not confuse stable proportions with small uncertainty

Stratification removes some variation in class counts by design. Fold scores may look less variable as a result. That does not establish that a deployed classifier’s performance is equally stable, especially if deployment prevalence changes or the positive examples are diverse.

Metric choice still matters. A majority-class classifier can have high accuracy in every stratified fold while catching no positives. Examine the metric that answers the task—such as recall at an agreed precision requirement—and include the underlying class counts. If you tune a threshold, that tuning must also respect the evaluation boundary.

The scikit-learn guide explicitly cautions that making folds more homogeneous can hide variability associated with rare classes. Use fold spread descriptively, not as an automatic confidence interval.

Know when label stratification is insufficient

Do not oversample the full dataset before constructing folds. Copies or synthetic examples derived from the same source can cross the validation boundary. Resampling for training belongs inside each training fold; the leakage chapter explains the distinction.

For grouped classification, group K-fold and StratifiedGroupKFold offer different balancing choices. A dataset where every positive belongs to one group cannot satisfy both “every fold has positives” and “each group belongs to only one fold.” The counting diagram is useful precisely because it makes such constraints visible before training.

References and further reading

• Scikit-learn: StratifiedKFold — the class and total-size allocation constraints, shuffling, and supported target types.
• Scikit-learn: stratification and rare classes — why class-preserving folds address missing-class problems without settling statistical uncertainty.
• Scikit-learn: StratifiedGroupKFold — the interaction between label balance and group separation.