Avoiding Leakage with Pipelines

September 2026 · 11 min read

A pipeline protects one particular boundary: the rows supplied to its fit call determine what its steps learn. Cross-validation can then fit a fresh pipeline on each training fold and transform that fold’s held-out rows using the saved state.

That protection depends on passing raw features into the evaluation procedure and choosing the correct split. A pipeline cannot remove future information already written into a feature, or turn a split of repeated observations into an evaluation on unseen people. This chapter focuses on checking those boundaries rather than repeating the random-label leakage experiment.

A held-out value should not change a fitted training mean

Fix one split: training inputs are [100, 102, 104, 106], while held-out inputs are [0, 2, 4, 6]. A scaler fitted on the training rows learns mean 103. A scaler fitted on all eight rows learns mean 53.

Now change only the held-out inputs. The training data has not changed, so its fitted mean should stay 103. A mean that moves in response to this edit reveals that held-out values entered the fitting calculation.

Place the complete recipe inside cross-validation

The leaking order is: fit preprocessing on the full dataset, transform it, and only then cross-validate an estimator on that transformed table. Each validation fold has already influenced the representation before the evaluation starts.

Pass Pipeline([("scale", StandardScaler()), ("model", Ridge())]) as the estimator to the CV function. Scikit-learn clones the estimator for each split. Passing a previously transformed feature table instead does not recover the original fitting boundary.

The same principle applies to imputers, category vocabularies, feature selection, dimensionality reduction, and learned feature engineering. Label-free statistics can leak too: the all-rows mean above never used a target.

Inspect the actual fitted objects

A score cannot tell us which rows a scaler saw. cross_validate can return both the fitted pipelines and the indices for each split. The following audit uses the eight inputs above in two folds and checks the learned means directly.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_validate

X = np.array([[0.], [2.], [4.], [6.], [100.], [102.], [104.], [106.]])
y = np.arange(8, dtype=float)
recipe = Pipeline([("scale", StandardScaler()), ("model", Ridge())])
result = cross_validate(
    recipe, X, y, cv=KFold(n_splits=2, shuffle=False),
    scoring="neg_mean_absolute_error", return_estimator=True, return_indices=True,
)
for fitted, train, valid in zip(
    result["estimator"], result["indices"]["train"], result["indices"]["test"]
):
    learned = fitted.named_steps["scale"].mean_[0]
    assert np.isclose(learned, X[train].mean())
    print("train", train.tolist(), "valid", valid.tolist(), "mean", learned)

train [4, 5, 6, 7] valid [0, 1, 2, 3] mean 103.0
train [0, 1, 2, 3] valid [4, 5, 6, 7] mean 3.0

Although this call also computes a prediction metric, the printed evidence concerns fitted state. The global mean, 53, appears in neither fitted scaler. Each learned mean matches that fold’s training inputs.

As a diagnostic, refit the first-fold recipe after changing only its validation features. With the same training rows and controlled randomness, the training state should remain unchanged. Checking one mean is a useful probe, not proof that every feature and component is safe; inspect learned vocabularies, selected columns, and feature construction as appropriate.

Three boundaries a pipeline does not choose for you

These boundaries are complementary. A correctly fitted pipeline can still be evaluated on the wrong population or receive information unavailable in deployment. Choose the prediction task first, then construct features and splits that reproduce it.

Supervised transforms need a suitable training representation

A target encoder may use labels to produce numerical features. Even within a training fold, directly encoding every row using its own label can make the training representation unusually easy to fit. A future row will not have that label available.

Scikit-learn’s TargetEncoder addresses this through internal cross-fitting in fit_transform. The enclosing pipeline preserves that interface, while later validation rows use statistics fitted on the training fold. The target-encoding chapter explains the two levels without conflating them.

Custom transformers need the same care. An object that reads a global table containing validation labels during its fit method can violate the boundary even though it sits inside a Pipeline. Composition controls the arguments it passes; it cannot inspect every external dependency of custom code.

After evaluation, refit deliberately

Once the recipe is selected, fit a fresh complete pipeline on the intended development data for final testing or deployment. It is normal for this final scaler to learn different statistics from the smaller CV fits.

Rows that served as validation data during model selection may enter that final development fit. Keep the separate final test outside the choice and fitting process. The CV estimate describes the evaluated learning procedure; it is not a measurement of a later model trained on more data.

In a from-scratch implementation, the boundary is the same: pass only the selected training inputs and targets to the pipeline fitting function, and use its returned state for the held-out rows.

References and further reading

• Scikit-learn: data leakage — how preprocessing before a split leaks information and how pipelines help.
• Scikit-learn: cross_validate — returning fitted estimators and split indices for inspection.
• Scikit-learn: cross-validation guide — evaluation splits, grouped and temporal observations, and fitting within folds.
• Scikit-learn: TargetEncoder — the internal cross-fitting behavior of the training transformation.