Reproducibility and Random Seeds

September 2026 · 8 min read

You set random_state=42, rerun the experiment next week, and get a different test score. The seed may be unchanged while the row order, dataset contents, library version, or training randomness has changed. A seed controls one part of an experiment; it does not describe the whole experiment.

This chapter separates reproducing an assignment from checking that a result is robust. Both matter. A perfectly repeatable split can still be unrepresentative, and a good method can produce slightly different scores across legitimate runs.

What the seed actually controls

A pseudo-random generator produces a sequence of numbers from an initial state. Setting a seed initializes that state. A splitter uses the resulting numbers to choose positions or shuffle rows. The same generator, seed, call sequence, and input can reproduce the same assignment.

Think of the seed as replaying a sequence of instructions. It does not identify the real-world examples those instructions will act on. Suppose a fixed shuffle selects positions 2 and 5 for test, using zero-based indexing:

The selected positions are an illustration, not the output of seed 42. They show why a database query with a different ordering can silently change the test examples. Adding or removing rows can change the permutation too.

Save the assignment as an experiment artifact

For a fixed benchmark, create the split once and save a mapping from stable row IDs to train, validation, or test. Reload that mapping for every candidate. Also save the dataset version and grouping rule: stable IDs alone do not prove that the contents attached to those IDs stayed unchanged.

Store the metric definition and selection procedure too. “Accuracy” is insufficient if one run excludes missing labels and another silently treats them as negatives. A reproducible number needs a reproducible denominator.

Separate splitting randomness from model randomness

A randomized forest, stochastic optimizer, or data augmentation procedure can introduce randomness after the split is fixed. Use distinct named controls for the split and training so you can vary one while holding the other constant:

SPLIT_SEED = 42
MODEL_SEED = 17

# Use SPLIT_SEED when constructing the split.
# Use MODEL_SEED for a model's random_state, if it has one.

The values 42 and 17 have no special statistical quality. Separate controls make the experiment easier to reason about: changing a training seed should not quietly draw a new test set. When comparing candidates, reuse the same held-out examples and the same planned evaluation schedule.

For growing data, an ID-based assignment can help

A saved split map is the simplest choice for a fixed dataset. For a growing collection, you may want each new example to receive an assignment without moving existing examples. One option is to apply a fixed hash function to an immutable row or group ID, then map the hash to a bucket.

Here is an example policy with 10,000 buckets. Buckets 0–5,999 mean train; 6,000–7,999 mean validation; 8,000–9,999 mean test. This aims for approximately 60/20/20 across many IDs, not exact subset counts.

import hashlib

def subset_for(stable_id, namespace="tickets-v1"):
    encoded = (namespace + "\0" + stable_id).encode("utf-8")
    digest = hashlib.sha256(encoded).digest()
    bucket = int.from_bytes(digest[:8], "big") % 10_000
    if bucket < 6_000:
        return "train"
    if bucket < 8_000:
        return "validation"
    return "test"

ids = [f"customer-{i:04d}" for i in range(100)]
original = {uid: subset_for(uid) for uid in ids}

# Reordering or appending IDs does not move an existing ID.
reordered = {uid: subset_for(uid) for uid in reversed(ids)}
expanded = {uid: subset_for(uid) for uid in ids + ["customer-0100"]}
assert original == reordered
assert all(expanded[uid] == role for uid, role in original.items())

The code uses a specified digest from Python’s hashlib, rather than relying on a process’s built-in object hash. Keep the ID encoding and normalization fixed: "0042" and "42" are different strings. Decide those conventions once and record them.

This scheme preserves assignment under growth, but it does not automatically stratify labels or enforce time boundaries. It can give poor proportions in small collections, and a group with many rows can dominate a subset. A future-period evaluation still needs a date boundary. A merge or correction to group identities may require a new split version and a leakage audit.

Repeatability is different from robustness

To replay a particular result, keep the split and model seed fixed. To study sensitivity to training randomness, keep the split fixed and run several training seeds chosen in advance. To study sensitivity to the development sample, use a planned repeated-validation or cross-validation design.

These repetitions answer different questions. Suppose changing the training seed changes a validation score substantially even on the same rows. That suggests the training procedure is unstable. If training is stable but changing development folds changes the ranking of candidates, the limited data may not distinguish them reliably.

Do not choose the seed with the highest test score and report only that result. It selects favorable noise just like repeatedly tuning other choices on test. Report the planned runs and their spread. Scores from overlapping folds are dependent, so a standard deviation across them is descriptive; it is not automatically a confidence interval.

Know the limits of an exact replay

A fixed seed does not guarantee identical outputs across arbitrary software versions, generator algorithms, call sequences, or hardware execution paths. NumPy’s random compatibility policy states the conditions under which streams are expected to match and the limits of that promise.

For an important result, retain the environment versions and fitted model as well as the configuration. Compare the actual row assignments first when a score changes; then compare preprocessing outputs, predictions, and the metric calculation. This turns “the seed stopped working” into a sequence of concrete checks.

The next chapter chooses between one held-out validation set and rotating folds. Both designs become more informative when candidates are evaluated on a shared, recorded schedule.

References and further reading

• Scikit-learn: controlling randomness — integer seeds, mutable RandomState objects, and robustness across fits and folds.
• NumPy: random-stream compatibility — what reproducible streams require and which changes can affect them.
• Python: hashlib — the SHA-256 digest API used in the order-independent assignment example.