Data Leakage in Splitting

September 2026 · 9 min read

A classifier reaches 73% test accuracy on independent random features and random labels. There is no predictive relationship to discover. The score comes from a mistake: the test labels helped choose the features before the classifier was trained.

Data leakage is information crossing an evaluation boundary where it was not allowed to influence the system. A different CSV file is not enough protection. We must trace what influenced the features, preprocessing, model choices, and final predictions.

A high test score with no real signal

Generate 400 rows, each with 5,000 independent noise features, and give each row an independent random binary label. Reserve 200 rows for training and 200 for test. Because the labels are independent of the features, performance on fresh examples should be around chance.

We compare two procedures. Both select 25 features using their association with the labels and fit logistic regression. In the first procedure, selection sees all 400 rows and labels. In the second, selection sees only the 200 training rows. The train/test IDs and classifier settings are otherwise the same. Defining split IDs first does not help if a later fit ignores that assignment.

Why does selection matter so much? Among 5,000 noise columns, some will align with random labels by accident. Selecting columns using the test labels keeps accidents that also look useful on those test rows. The classifier is then evaluated on a sample that helped construct its input representation.

Proper selection can also find accidental patterns, but those patterns were chosen using training labels only. Independent test labels can expose their failure to generalize. This experiment follows the feature-selection leakage demonstration in scikit-learn’s common-pitfalls guide, using our own dataset and settings.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(7)
X = rng.normal(size=(400, 5000))
y = rng.integers(0, 2, size=400)
train, test = train_test_split(
    np.arange(400), test_size=0.5, random_state=11, stratify=y
)

# Incorrect: held-out labels influence feature selection.
leaked_X = SelectKBest(f_classif, k=25).fit_transform(X, y)
leaked_model = LogisticRegression(max_iter=1000)
leaked_model.fit(leaked_X[train], y[train])
leaked_pred = leaked_model.predict(leaked_X[test])

# Correct: every learned step fits on training rows only.
proper_model = make_pipeline(
    SelectKBest(f_classif, k=25),
    LogisticRegression(max_iter=1000)
)
proper_model.fit(X[train], y[train])
proper_pred = proper_model.predict(X[test])

print("Leaked correct:", int((leaked_pred == y[test]).sum()))
print("Proper correct:", int((proper_pred == y[test]).sum()))
# Leaked correct: 146
# Proper correct: 101

Preprocessing can leak without using labels

Suppose training values are 2, 4, 6, and one missing entry. A mean imputer should learn 4 from the observed training values. If it also sees a held-out value of 100, its mean becomes 28. The test example has changed the value inserted into the training data.

This violates the usual evaluation setting where future examples do not help fit the system. It does not imply that every leaked transform raises every metric; some transformations have little effect for a particular model. The boundary is invalid regardless of the direction of the score change.

Use fit or fit_transform on training data, and the fitted object’s transform on held-out data. Scaling, imputation, learned vocabularies, principal components, and feature selection all need this distinction. A pipeline helps ensure these steps are refitted within each training fold during validation.

Look for the other routes across the boundary

The detailed repairs for shared entities and delayed outcomes are in group splitting and time-series splitting. They are separate checks: a pipeline can handle preprocessing correctly while the split still shares users or uses future labels.

Audit an actual prediction, working backward

Choose one held-out prediction and trace its dependencies. Which model version produced it? Which rows fitted that model and its transforms? Which records supplied each feature, and when did those records become available? Which validation results influenced the selected settings?

Then look for a held-out row, related source, or unavailable outcome in that dependency chain. Save enough provenance—row and group assignments, timestamps, fitted pipeline, and selection record—to check the answer. Disjoint row IDs are useful evidence, but they do not prove every dependency is separate.

A practical check is to rerun the full procedure with label permutations appropriate to the data structure. Persistently strong performance after destroying the intended feature–label relationship can reveal problems. It is a diagnostic, not a universal proof: preserving group or time dependence may require a more careful permutation scheme, and ordinary leakage can exist even when this check shows chance performance.

Repair the experiment before interpreting the metric

If leakage is found, rebuild the affected features or pipeline and rerun the evaluation with the correct boundary. A preprocessing mistake does not automatically mean every row is unusable. But if test outcomes have already guided development decisions, that test set no longer supplies an independent assessment of those decisions; use a fresh hold-out or a properly designed outer evaluation.

Report the corrected procedure and result. A lower score can be more useful because it describes the task the deployed model will actually face. The next chapter covers reproducible splits, so that a repaired experiment remains stable across reruns.

References and further reading

• Scikit-learn: data leakage and preprocessing — definitions, fit/transform separation, and the feature-selection noise experiment that motivated this example.
• Scikit-learn: evaluation and model selection — why test outcomes must remain outside the development loop.
• Scikit-learn: permutation_test_score — label-permutation testing and group-aware permutations.