Reproducibility

September 2026 · 11 min read

A seed is one part of an experiment record. Reproducing a model also depends on the data content, row identities, feature construction, split assignments, training order, parameters, code, and numerical environment.

The practical question is what you want to reproduce. Loading a fitted pipeline and obtaining the same predictions is different from training a new pipeline and comparing its results. Testing whether a conclusion survives different reasonable seeds or splits is a third question.

Distinguish replay, retraining, and robustness

A perfectly repeatable leakage bug satisfies the second question while producing an invalid evaluation. Reproducibility makes an experiment inspectable; the data-boundary checks establish whether it evaluates the intended task.

The same seed can select different records

Start with row IDs A through H. With these eight rows in that order, scikit-learn’s train_test_split(test_size=2, random_state=7) selects positions 2 and 5 as the held-out rows, using zero-based positions: IDs C and F.

Rotate the input order once so it starts at B. The same random choice of positions now selects IDs D and G. The seed and test size stayed the same; the people or examples being evaluated changed.

Save ordered IDs and an immutable data reference

Record which rows belong to each split using stable identifiers. Save their order too: stochastic learners can consume rows or bootstrap positions in an order-dependent way. An ordered list lets a replay reconstruct the same training input even if a file is later loaded in a different order.

IDs alone do not establish that the feature values and targets are unchanged. Keep an immutable dataset snapshot or a content digest and a retrievable versioned reference. A row named C can contain different values after a data correction. Keep row IDs, inputs, and targets aligned when rebuilding the arrays.

What belongs in the experiment record?

The record should be readable without reconstructing a notebook’s execution history. For example, “random_state 23” needs to identify whether it belongs to the split, target encoder, or estimator. Those are separate sources of randomness.

Make random choices explicit

For many scikit-learn estimators and splitters, passing an integer seed restarts their random stream for repeated calls under the same inputs and configuration. Passing a mutable RandomState instance instead allows calls to consume and advance shared state. The order of calls can then matter.

Control the randomized components actually used by the recipe. A split seed does not set a forest’s bootstrap seed or an encoder’s internal fold assignment. The splitting and seeds chapter develops those distinctions; the example below records the split and forest seeds separately.

Random-number APIs and numerical libraries have compatibility policies, not a universal guarantee of identical results across all versions and machines. Pin the environment for an exact replay target, then verify it. Parallel reductions, precision, and nondeterministic execution can require additional controls or a stated numerical tolerance.

Replay a saved split after the rows are reordered

This small example writes ordered split IDs and model parameters to JSON. It rebuilds training arrays from those IDs before fitting a forest. The second replay receives the same records in a different order and recovers the same training and evaluation arrays.

import json
from pathlib import Path
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

rows = [{"id": chr(65 + i), "x": float(i), "y": float(i * i)} for i in range(8)]
ids = [row["id"] for row in rows]
train_ids, test_ids = train_test_split(ids, test_size=2, random_state=7)
record = {
    "split_seed": 7, "train_ids": train_ids, "test_ids": test_ids,
    "model_params": {"n_estimators": 20, "random_state": 23, "n_jobs": 1},
}
Path("toy-run.json").write_text(json.dumps(record, indent=2))
record = json.loads(Path("toy-run.json").read_text())

def replay(data, run):
    by_id = {row["id"]: row for row in data}
    if len(by_id) != len(data):
        raise ValueError("duplicate row IDs")
    # Reconstruct membership AND training order from the saved ID lists.
    X_train = [[by_id[i]["x"]] for i in run["train_ids"]]
    y_train = [by_id[i]["y"] for i in run["train_ids"]]
    X_test = [[by_id[i]["x"]] for i in run["test_ids"]]
    model = RandomForestRegressor(**run["model_params"]).fit(X_train, y_train)
    return model.predict(X_test)

first = replay(rows, record)
second = replay(rows[1:] + rows[:1], record)
np.testing.assert_allclose(first, second, rtol=0, atol=0)
print("Held-out IDs:", record["test_ids"])
print("Predictions match:", bool(np.array_equal(first, second)))

Held-out IDs: ['C', 'F']
Predictions match: True

The equality check concerns two fits in the same environment with unchanged data and saved training order. It does not claim cross-version or cross-hardware bit-for-bit identity. This teaching record focuses on splits and model parameters; a real run should add the data and runtime references shown above.

A missing saved ID should cause an explicit replay failure, not silently select a replacement row. The example rejects duplicate IDs and raises a lookup error for a missing required record.

Repeat to assess sensitivity, not to choose a lucky result

Once a run is reproducible, vary the source of randomness whose effect you want to understand. Holding splits fixed while varying a model seed probes learner randomness. Repeating a splitter probes sensitivity to held-out samples. These answer different questions.

Report the design and the distribution of results rather than selecting the best seed. For paired model comparisons, use matching saved split assignments where appropriate. The repeated cross-validation notes explain what repeated scores do and do not establish.

For a C++ pipeline, record the same data order, split identities, configuration, compiler/library environment, and any random-generator details. Matching a seed number across languages does not imply matching random streams or fitted models.

References and further reading

• Scikit-learn: controlling randomness — integer seeds, mutable random state, splitters, and repeated fitting.
• NumPy: random compatibility policy — the conditions and limits of reproducible random streams.
• Scikit-learn: model persistence — data, code, dependency versions, and environment information needed for reuse.
• Scikit-learn: train_test_split — the randomized partition operation used by the row-order example.