Group Splitting

September 2026 · 8 min read

You have 12 sensor readings from four devices. Every row has a unique ID, and no ID appears in both train and test. Yet the model can still recognize devices it has already seen. If deployment means working on a new device, a row-level split may answer the wrong question.

Group splitting keeps all rows from the chosen unit together. The unit might be a person, device, document, household, or experiment. Choose it according to what must be unfamiliar at evaluation time—not merely according to which ID column is easiest to use.

A model can memorize the group instead of the task

Consider four devices, A through D, with three readings each. In this deliberately simple example, every reading from A and C has label 0; every reading from B and D has label 1. A lookup rule stores the label of each device encountered in training. For an unseen device, it defaults to label 0.

If two readings from every device are in training and the third is in test, the rule gets all four test readings correct. It has achieved 100% by recognizing identities. It has learned nothing about how to classify an unfamiliar device.

With A and B in training and C and D in test, both test devices are unfamiliar. The lookup rule predicts 0 for all six test readings: it gets C’s three readings right and D’s three wrong, for 50% accuracy. The lower score exposes the rule’s limitation. These constructed numbers illustrate identity memorization; they do not predict how much a real model’s score will drop.

Removing the explicit device ID does not necessarily remove the shortcut. Calibration offsets, writing style, repeated background images, or other stable properties can reveal identity indirectly. Scikit-learn’s group-validation discussion uses this same concern to motivate evaluating on subjects absent from training.

Choose a group that matches the deployment question

The last row is deliberately different. Reusing known users can be legitimate when that is the intended use. Holding every user out would instead test new-user performance. There is no universally correct group column; there is a specific claim you want the evaluation to support.

Build groups before splitting derived examples. If one document is chunked into 30 passages, assigning each passage independently can place neighboring text on both sides. Give all passages the document’s group assignment. Use a broader duplicate-family ID when several documents are copies of the same underlying source.

Split the groups, then retrieve their rows

GroupShuffleSplit chooses whole groups and returns row indices. In this runnable example, the six groups have very different sizes. Reserving one third of the groups holds out two groups; it does not promise one third of the rows.

import numpy as np
from sklearn.model_selection import GroupShuffleSplit

sizes = [2, 3, 4, 5, 6, 10]
groups = np.repeat(np.arange(6), sizes)
row_id = np.arange(len(groups))
X = row_id.reshape(-1, 1)

splitter = GroupShuffleSplit(
    n_splits=1, test_size=1/3, random_state=42
)
train, test = next(splitter.split(X, groups=groups))

train_groups = set(groups[train])
test_groups = set(groups[test])
assert train_groups.isdisjoint(test_groups)
assert len(test_groups) == 2
assert set(train).isdisjoint(test)
assert set(np.concatenate((train, test))) == set(row_id)

print("Held-out groups:", sorted(map(int, test_groups)))
print("Held-out rows:", len(test), "of", len(row_id))

This seed holds out groups 0 and 1: five of the 30 rows, despite reserving two of six groups. The API defines test size in groups. The selected row fraction depends on which groups are drawn. Inspect group counts, row counts, and labels together. A test set with many rows from one large group still contains little evidence about variation between groups.

For train/validation/test, first hold out test groups. Apply a second group split to the remaining development rows, passing their corresponding group labels. When the second splitter returns local row positions, map them back through the development indices. Every development comparison—including tuning and feature selection—must preserve the intended group boundary too.

Decide whether rows or groups should count equally

Suppose two held-out devices supply very different amounts of data:

Pooling rows gives 90 correct out of 100, or 90%. Averaging the two device accuracies equally gives 50%. The first describes a uniformly chosen reading from this collection; the second describes a uniformly chosen device followed by a reading from that device. Neither calculation is universally preferable.

Choose the weighting that matches the use case and show per-group results when feasible. If uncertainty is estimated by resampling, preserve the dependence within each group: resampling individual readings as if they were independent can make the evidence look much stronger than it is.

Group boundaries can conflict with convenient proportions

A rare class may occur in only one group. You cannot place that group in both train and test while claiming the test group is unseen. Similarly, large groups can make equal row counts impossible. StratifiedGroupKFold tries to combine group separation with class balance, but it cannot satisfy incompatible constraints.

Keep the independence boundary and report the imbalance, revise the evaluation question, or gather additional groups. Moving a few rows across the boundary solely to make percentages match defeats the purpose of grouping.

Save the group definition along with the split IDs. Check that no held-out group occurs in training, verify derived examples retain their parent’s assignment, and evaluate enough distinct groups to support the claim. Next, time-series splitting adds a different boundary: what the model could have known at the moment of prediction.

References and further reading

• Scikit-learn: group cross-validation — why repeated subjects require a different validation design when generalizing to new subjects.
• Scikit-learn: GroupShuffleSplit — whole-group hold-outs and the meaning of test_size.
• Scikit-learn: StratifiedGroupKFold — the trade-off between preserving groups and approximating class proportions.