Shuffle-Split (Monte Carlo CV)

September 2026 · 8 min read

Shuffle-split validation, also called repeated random holdout or Monte Carlo cross-validation, repeatedly draws a training subset and a validation subset. You choose the validation size and the number of draws separately.

That flexibility is its main attraction. You can test an 80/20 training–validation arrangement 5 times or 50 times without changing the training fraction. Unlike K-fold CV, the draws do not partition the dataset into validation folds that collectively cover each row exactly once.

What one draw does

Start with ten independent examples, A–J. Randomly select two for validation and use the other eight for training. Within that draw, an example appears at most once and cannot be in both subsets.

For the next draw, start from all ten examples again and select a new two-example validation subset. Previously validated examples are eligible again. Reuse occurs across draws; there are no duplicated rows inside an individual subset.

Count appearances before interpreting the result

With five draws and two validation examples each, there are ten validation appearances. It is tempting to assume that this covers all ten examples. Here is what ShuffleSplit(n_splits=5, test_size=2, random_state=23) actually generates:

Example A is validated twice; B is never validated. This does not mean B was discarded: it belongs to the training subset in all five draws. It means this run provides no held-out prediction for B.

Across runs with fresh randomization, every example has the same chance of selection in each draw. Coverage can still be uneven in a particular finite run. Repeated K-fold has a different guarantee: each complete repeat validates every example once.

How can an example miss every validation draw?

In our ten-example dataset, any particular example has probability 2/10 = 0.2 of entering a two-example validation subset. Its probability of staying out is 0.8. Assuming independent uniform draws, it must stay out five times to remain unvalidated:

That is about a 32.8% chance for any one example. Five draws therefore do not make complete coverage likely merely because the total number of validation appearances equals the dataset size.

Same training size, different coverage rules

For ten examples, these three choices all fit on eight and validate on two:

In shuffle-split, n_splits=5 means five random draws; it does not mean “divide the dataset into five disjoint validation folds.” Use repeated K-fold when complete coverage within each repeat is useful. Use shuffle-split when controlling validation size independently from the number of fits is more useful.

Audit the generated indices

This code checks the split design before connecting a model. It produces the exact figure above; it does not train a model or calculate a performance score.

import numpy as np
from sklearn.model_selection import ShuffleSplit

ids = np.array(list("ABCDEFGHIJ"))
X = np.arange(len(ids)).reshape(-1, 1)
splitter = ShuffleSplit(n_splits=5, test_size=2, random_state=23)
splits = list(splitter.split(X))
validation_count = np.zeros(len(ids), dtype=int)

for draw, (train, valid) in enumerate(splits, start=1):
    assert len(train) == 8 and len(valid) == 2
    assert len(np.unique(train)) == 8
    assert len(np.unique(valid)) == 2
    assert not np.intersect1d(train, valid).size
    assert np.union1d(train, valid).size == len(ids)
    validation_count[valid] += 1
    print(draw, "validate:", ids[np.sort(valid)].tolist())
    # Fit a fresh candidate pipeline on train; score only valid.

print("Validation appearances:", dict(zip(ids, validation_count)))
print("Never validated:", ids[validation_count == 0].tolist())
assert validation_count.sum() == 5 * 2
assert validation_count.tolist() == [2, 0, 0, 0, 0, 2, 1, 2, 2, 1]

An integer test_size=2 requests exactly two validation examples. A fractional size is converted to an integer count, so small datasets can differ from the nominal percentage. With train_size=None, training uses the complement. If both sizes are supplied, some examples may be unused in a draw; audit that explicitly if it is intended.

Score draws without silently changing the weighting

For each draw, fit a fresh pipeline using its training indices and calculate the declared metric on its validation indices. Averaging those scores summarizes performance across the chosen random holdouts. Use the same draw list for every candidate you compare, as in the tuning chapter.

If every validation subset has equal size and the metric averages per-example losses, the average of draw scores equals the average over all validation appearances. In our run, A’s errors appear twice and G’s once. This is the actual weighting of this finite set of draws, even though the sampling design treats examples symmetrically.

Do not put predictions into one array indexed by row and repeatedly overwrite earlier values. Store the draw number, row ID, target, and prediction together. That preserves multiple predictions for A and makes missing predictions for B visible. Averaging predictions before taking the loss evaluates a different procedure; the repeated-prediction example shows why.

What additional draws can and cannot improve

More draws explore more allocations of the same dataset and tend to improve coverage. They leave the amount of data available to each fit unchanged. The original sample is still the only evidence about the population; the resulting fold scores share observations and training data.

For confidence intervals, do not treat every validation appearance as a fresh independent case. The distinction between split randomness and dataset uncertainty is explained in the preceding chapter. Additional shuffling cannot supply missing deployment conditions.

StratifiedShuffleSplit approximately preserves class proportions within the requested integer sizes. It still allows repeated validation appearances and does not ensure that every example is eventually validated. Apply the rare-class limitations when deciding whether the resulting folds support your metric.

Repeated customers need a group-aware split; future prediction needs time-aware splits. Finally, use these draws for development without consuming an independent final test set. If you select the best candidate from the averaged scores, evaluate that selection through the boundary described in nested CV or an untouched holdout.

References and further reading

• Scikit-learn: ShuffleSplit — random draws, train and validation sizes, and possible overlap across validation subsets.
• Scikit-learn: StratifiedShuffleSplit — class-wise stratification with the same repeated-holdout coverage limitations.
• Scikit-learn: cross-validation strategies — how random permutations differ from K-fold partitions and structured-data splitters.