Group K-Fold

September 2026 · 8 min read

Group K-fold asks whether a training recipe transfers to groups it did not see. Each validation round holds out whole people, devices, documents, or other chosen units. Every group takes a turn in validation, and all of its rows take that turn together.

The group-splitting chapter explains identity shortcuts and how to choose the group definition. Here we focus on constructing several group-safe folds, balancing unequal groups, and passing the same boundary through model selection.

Count groups before counting folds

Suppose six devices provide 33 readings. Devices A through F contribute 8, 7, 6, 5, 4, and 3 readings respectively. We want three validation rounds. The groups must remain indivisible even though they have different sizes.

A group partition can put A with F, B with E, and C with D. Every validation fold then contains 11 readings from two devices. In each round, the training set contains the other four devices and 22 readings.

At least three distinct groups are needed for three nonempty group folds. Thousands of rows cannot make up for having only two independent devices. More generally, K cannot exceed the number of distinct groups.

How unequal groups can be assigned

One practical strategy places the largest groups first. Keep a running row count for each fold; assign the next group to the currently lightest fold. This is the unshuffled GroupKFold strategy in the scikit-learn implementation used here.

This greedy assignment is a useful heuristic, not an optimal-partition guarantee. A large group can dominate a fold, and some group-size combinations simply cannot produce equal row counts. Preserve the group boundary and report the actual fold composition.

Also distinguish equal row counts from equal group counts. In current scikit-learn, GroupKFold(shuffle=False) tries to balance rows, while shuffle=True shuffles groups and partitions them without accounting for their sizes. The user guide describes this distinction. A shuffled fold with two large groups can have many more rows than one with two small groups.

Check the boundary in executable code

The group vector has one entry per row. Rows from the same device receive the same label in that vector. Group labels guide the splitter; they need not be model features.

import numpy as np
from sklearn.model_selection import GroupKFold

sizes = [8, 7, 6, 5, 4, 3]
groups = np.repeat(np.array(list("ABCDEF")), sizes)
X = np.zeros((len(groups), 1))
cv = GroupKFold(n_splits=3, shuffle=False)
seen = np.zeros(len(groups), dtype=int)
held_out = []

for train, val in cv.split(X, groups=groups):
    train_groups = set(groups[train])
    val_groups = set(groups[val])
    assert train_groups.isdisjoint(val_groups)
    seen[val] += 1
    held_out.append("".join(sorted(val_groups)))
    assert len(train) == 22 and len(val) == 11
    print("Validation devices:", sorted(val_groups),
          "rows:", len(val))

assert held_out == ["AF", "BE", "CD"]
assert np.all(seen == 1)

This reproduces the diagram with scikit-learn 1.9.1. In a real dataset, also inspect per-fold label counts, dates, and source coverage. A group-safe split can still contain a one-class validation fold, which makes some classification metrics undefined.

After constructing the indices, use the ordinary fresh-fit loop with this splitter. Clone and fit the whole pipeline on each training portion, predict the held-out rows, and keep predictions associated with their group IDs.

Class balance is a separate constraint

GroupKFold does not use labels to preserve class proportions. If labels are concentrated in particular devices or people, fold class balances can differ substantially even when row counts match. StratifiedGroupKFold attempts to improve that balance while keeping groups intact.

It is still a heuristic. Exact class balance may be impossible, and the algorithm need not find the best possible allocation even when a better one exists. Inspect the output. Never move individual positive rows out of a held-out group merely to make the class percentages match.

If one group contains every positive example, the fold holding it out has no positive training examples. That is a property of the available evidence, not something the splitter can repair. More groups containing the rare outcome, or a revised evaluation question, may be needed.

Use group-aware validation during tuning too

When selecting hyperparameters, pass the development groups to the search alongside the development features and labels. Every candidate should be evaluated on the same group partitions. With the usual scikit-learn metadata-routing configuration, the structure is:

# pipeline and param_grid describe the candidate training recipes.
search = GridSearchCV(
    pipeline, param_grid, cv=GroupKFold(n_splits=3),
    scoring="accuracy"
)
search.fit(X_dev, y_dev, groups=groups_dev)

This snippet shows where group labels enter the search; the complete search setup is in CV for tuning. If you enable metadata routing, check the API’s group-routing requirements rather than silently dropping the group information.

An untouched outer group test can still provide an independent final assessment even if development used ordinary row folds. However, those inner folds would select recipes for the easier familiar-group task. Align the inner design with the actual transfer question so model selection rewards the behavior you need.

Balanced folds do not mean equal group influence

Our three folds have equal row counts. Averaging their accuracies gives every row equal influence across the 33 held-out predictions. Device A still contributes eight readings and device F only three, so the devices are not weighted equally.

If the target is an average new device, compute a score per device from its held-out predictions and average device scores using the intended weights. If the target is an average reading from the represented traffic, pooling rows may be appropriate. The worked weighting example shows how those summaries can disagree.

Retain group-level results to see transfer failures hidden by an aggregate. Six devices provide six group units of evidence, not 33 independent device-transfer experiments. The training sets also overlap across folds, so the fold scores are dependent.

For predictions that must be made later in time, use time-series CV as well. Holding out devices does not by itself prevent training on information from the future.

References and further reading

• Scikit-learn: grouped cross-validation — whole-group validation, row-count versus group-count balancing, and leave-one-group-out.
• Scikit-learn: GroupKFold — the group-index API and shuffling options.
• Scikit-learn: StratifiedGroupKFold — class-aware group allocation and its limitations.