Saving and Serializing Transformers

September 2026 · 11 min read

Saving a preprocessing recipe is different from saving its fitted state. SimpleImputer(strategy="mean") describes what to learn. A fitted imputer also contains the mean it learned. A model needs the latter when a future row arrives.

Persist the fitted pipeline, including its transforms and estimator. Keep its input schema and a few prediction checks alongside it. This lets another process reconstruct the same predictor without learning new statistics from incoming data.

Why the estimator alone is incomplete

In the preceding pipeline example, the imputer fills missing inputs with 30 and the scaler uses center 30 and divisor √200. Ridge’s coefficient, about 0.8485, multiplies the standardized feature. Its intercept is 2.5.

If you save only ridge, it can still accept a finite raw number and return a result. But the number is being interpreted in the wrong units. Compare the same raw inputs through the complete pipeline and through its final estimator alone:

Package the predictor with the information needed to use it

For a mixed-column model, the schema includes required column names, types, and units. The fitted encoder contains its category order, and ColumnTransformer exposes the emitted feature names. Keep that learned order; rebuilding a vocabulary alphabetically from a different dataset can change which coordinate means what.

The manifest documents expectations; it does not enforce them by itself. The serving input adapter should check the schema before calling the predictor. A two-column array with swapped fields can have the expected shape while carrying the wrong meaning.

Save, load, and compare predictions

First run the complete Python example in Preprocessing Pipelines, which creates the fitted pipe. The following extension writes it to a versioned directory and reloads it. The three probes cover a missing input, the fitted center, and a value beyond the observed training range.

# First run the Python example in "Preprocessing Pipelines" to create pipe.
import json
import platform
from pathlib import Path
import joblib
import numpy as np
import scipy
import sklearn

artifact = Path("artifacts/toy-pipeline-v1")
artifact.mkdir(parents=True, exist_ok=True)
manifest = {
    "schema": {"input": "2D numerical array", "feature_order": ["x"], "missing": "NaN"},
    "versions": {
        "python": platform.python_version(), "numpy": np.__version__,
        "scipy": scipy.__version__, "scikit_learn": sklearn.__version__,
        "joblib": joblib.__version__,
    },
}
probes = np.array([[np.nan], [30.], [70.]])
expected = pipe.predict(probes)
joblib.dump(pipe, artifact / "pipeline.joblib", compress=3)
(artifact / "manifest.json").write_text(json.dumps(manifest, indent=2))
np.savez(artifact / "probes.npz", inputs=probes, predictions=expected)

# Load only the trusted artifact we just created, in this matching environment.
loaded = joblib.load(artifact / "pipeline.joblib")
np.testing.assert_allclose(loaded.predict(probes), expected, rtol=1e-12, atol=1e-12)
print(loaded.predict(probes).round(4).tolist())

[2.5, 2.5, 4.9]

The saved probe inputs and expected outputs can also be loaded in a fresh process. That catches problems hidden by the training notebook’s current variables. For classification, compare class order and probability columns as well as predicted labels; for a categorical pipeline, include a representative unseen category.

The numerical tolerance above is a strict check for this small example in a matching environment. Choose a justified tolerance for the deployed computation. Passing these probes establishes agreement on these cases; it does not measure predictive quality or cover every input.

Reloading should not call fit

joblib.load reconstructs the saved fitted object. Call predict on raw inputs, or transform on a saved preprocessing-only object. Calling fit again learns new state and creates a different predictor.

Refitting only the scaler while keeping the old estimator is particularly easy to miss. The estimator’s coefficients remain tied to the original representation. If new data motivates a changed transform, retrain and validate the complete recipe as a new version.

Likewise, rebuilding this pipeline from constructors with the same hyperparameters creates a new recipe; it does not restore learned statistics. The reproducibility chapter separates reloading a predictor from retraining one.

The environment is part of the reload contract

Use a matching Python and dependency environment for pickle-based scikit-learn artifacts. Loading models across scikit-learn versions is unsupported. Recording versions helps identify the environment; an environment lock or image is what lets another process recreate it.

Custom transformers also need their code. Keep custom classes in importable, versioned modules and verify loading in a fresh process that has those modules installed. Saving an object does not turn its Python dependencies into a standalone inference runtime.

Choose a persistence format for the intended use

Load pickle-based artifacts only from trusted sources. Loading can execute code; a model file is not just a table of coefficients. The example reloads the file it just wrote. A checksum can detect changed bytes relative to a trusted recorded checksum, but does not by itself establish who created a file.

For a small C++ predictor, the same principle applies without Python serialization: save the fitted fill, center, scale, coefficient, and intercept together with the schema and format version. If conversion changes the numeric representation, compare predictions on the same raw probes.

References and further reading

• Scikit-learn: model persistence — format tradeoffs, environment compatibility, custom code, and trusted artifacts.
• Joblib: dump — saving a fitted Python object and choosing compression.
• Joblib: load — reconstructing saved objects and the implications of pickle-based loading.