The Hold-Out Split

September 2026 · 8 min read

A hold-out split reserves examples for evaluation before fitting the system. Its purpose is simple: ask the model to predict answers that did not help build it. The difficult part is making sure “held out” means more than assigning a different filename.

This chapter constructs a split, checks it, and interprets its score. The train/validation/test chapter explains which decisions each subset may influence. Here we assume independent rows from the same population; the group and time chapters change that assumption.

Split row identifiers, keeping each example intact

Think of a row as a package: its stable identifier, features, label, and any sample weight. Shuffle the identifiers once, choose which belong to each subset, then use that assignment for every part of the package. Shuffling feature rows and labels separately destroys the learning problem.

For 10 rows, one possible shuffled order is 7, 2, 9, 0, 4, 1, 8, 5, 3, 6. An 80/20 split can assign the first eight identifiers to training and the last two to test:

Keeping stable IDs also helps after joins, filtering, or database reloads. A positional index is only stable while the underlying row order stays the same. Store the assignment against an identifier that survives those operations.

Choose counts that answer the evaluation question

An 80/20 split is a convenient example, not a law. More training rows can improve the fitted model; more test rows can make the performance estimate more precise. Those are different benefits. A 1% test set can be large in a million-row dataset and useless in a 100-row dataset.

Start from the important outcomes and populations. If only 1% of events are positive, a 200-row random test set contains just two positives on average. That may be far too little evidence about recall. Stratification can stabilize the allocation, but it cannot create additional positive examples.

For a fixed fraction, integer counts need rounding. In scikit-learn’s splitter, a fractional test size is rounded up: 20% of 11 rows gives three test rows and, when training size is left unspecified, eight training rows. Inspect the resulting counts instead of assuming the fraction is exact.

A three-way split needs two carefully chosen fractions

Suppose we want 600 training, 200 validation, and 200 test rows from 1,000 examples. First reserve 20% of all rows for test. There are now 800 development rows left. Validation needs 200 of those 800 rows, so the second split must reserve 25% of the remainder.

For other ratios, let $f_{\mathrm{test}}$ and $f_{\mathrm{val}}$ be the desired fractions of the original dataset. The remaining fraction is $1-f_{\mathrm{test}}$. Dividing the desired validation share by that remainder gives the fraction for the second call:

In our example, this is $0.20/0.80=0.25$. This is just a change of denominator; it is not a statistical adjustment.

Construct and check a reproducible split

The following runnable example uses 1,000 synthetic rows. It splits integer IDs, then indexes features and labels with those IDs. The labels are included only to demonstrate alignment; no model is trained here.

import numpy as np
from sklearn.model_selection import train_test_split

row_id = np.arange(1000)
X = np.column_stack((row_id, row_id % 7))
y = (row_id % 10 == 0).astype(int)

dev, test = train_test_split(
    row_id, test_size=0.20, random_state=42
)
train, val = train_test_split(
    dev, test_size=0.25, random_state=43
)

assert (len(train), len(val), len(test)) == (600, 200, 200)
assert set(train).isdisjoint(val)
assert set(train).isdisjoint(test)
assert set(val).isdisjoint(test)
assert set(np.concatenate((train, val, test))) == set(row_id)

X_train, y_train = X[train], y[train]
X_val, y_val = X[val], y[val]
X_test, y_test = X[test], y[test]

The integer seeds reproduce this assignment for the same inputs and software behavior. They do not make it representative or statistically independent. Changing the input order can change which real examples receive those IDs. Save the actual ID lists as well as the seeds. For a classification split that preserves label proportions, add the appropriate stratify argument as shown in the stratification chapter.

The assertions check the partition, not every kind of leakage. Two different IDs can still refer to copies of the same document or observations of the same person. That needs a content or group-level check.

A hold-out score is a sample estimate

Suppose a fixed classifier gets 180 of 200 independent test examples correct. Its accuracy is 90%. On a different sample of 200 examples, the count may change even if the classifier and population stay the same. The observed percentage is an estimate of the classifier’s accuracy on that population.

To express the rough size of this variation, write $n$ for the number of independent test examples and $\hat p$ for their observed fraction correct. For binary correctness outcomes, a common large-sample estimate of the standard error is:

With $\hat p=0.90$ and $n=200$, this is about 0.021, or 2.1 percentage points. It describes sampling variability, not model-training variability. Multiplying by roughly two gives a quick sense of scale, but a proper binomial interval is preferable when reporting uncertainty, especially for small samples or proportions near zero or one.

What a random hold-out does not establish

A clean random split supports an estimate for new independent examples from the same distribution. It does not establish performance on a new hospital, a new language, a different sensor, or next year’s changing traffic. Those deployment questions need corresponding held-out units or time periods.

Likewise, a single split does not reveal how sensitive model selection is to the development sample. Repeated validation or cross-validation can help study that variation. Searching for a random seed with a better test score merely converts test-set luck into a design choice.

Once the partition is defined, fit learned transformations on training rows, compare choices on validation, and reserve test outcomes for the final assessment. For the low-level shuffle and partition mechanics, see the existing C++ implementation.

References and further reading

• Scikit-learn: train_test_split — aligned-array splitting, size arguments, seeds, and stratification.
• Scikit-learn: evaluation and cross-validation — the assumptions and limitations of a single random hold-out.
• NIST: confidence intervals for a proportion — binomial proportion uncertainty and alternatives to a rough normal approximation.