Build a Reproducible Distributed ML Scoring Pipeline

September 2026 · 4 min read

Before you start: Spark fundamentals and basic model training/serving.

A model that works in a notebook still needs a reproducible answer to “Which data, code, and feature definitions produced this score?” At large scale, partitioning, retries, and incomplete writes become part of model correctness. This chapter builds a synthetic batch-scoring design for application risk; it is an instructional architecture, not a description of an employer’s internal system.

The run manifest is the unit of reproducibility

manifest = {
    "run_id": "risk-2026-08-01-v3",
    "input_snapshot": "apps-snapshot-42",
    "cutoff_utc": "2026-08-01T00:00:00Z",
    "feature_schema_version": "risk-features-v8",
    "code_revision": "record-the-git-commit",
    "environment_lock_hash": "record-the-lockfile-hash",
    "training_seed": 7,
    "model_artifact_hash": "record-the-artifact-hash",
    "expected_scoring_universe": "eligible-app-ids-v4",
}

Store schemas, feature order, categorical mappings, missing-value rules, and the score convention with the model. A classifier expecting feature columns [age,count] can return plausible nonsense if serving supplies [count,age]. Validate names, types, and ordering before inference rather than relying on coincidental DataFrame layout.

Fingerprint the dataset you actually mean

A hash of a small sample is a useful change detector, but it does not identify the whole dataset. A reproducibility fingerprint can combine immutable source-object versions, row counts, schema, transformation configuration, and partition checksums. If hashing row content, define canonical serialization, null handling, ordering, and duplicate semantics. XOR of row hashes alone can cancel duplicate pairs and miss changes.

A random seed does not make a distributed sample invariant to repartitioning. For a stable sample, hash a stable entity key with a fixed salt and select by hash range or deterministic top-K. Specify collision tie-breaking. Keep the sample’s identity with the run so a changed partition count cannot silently change the training population.

Control joins and inference placement

Before joining features, assert the intended cardinality. A duplicate lookup key can multiply rows and inflate counts. After joining, verify key uniqueness, row coverage, null rates, and denominator consistency. Track skewed keys and large metadata groups; one giant partition can dominate runtime even when total data volume looks manageable.

Load a small model once per worker or partition context where the execution engine allows, then score batches. Avoid loading it once per row or collecting the entire dataset into the driver. Size batches against feature width, model memory, and explanation overhead. Preserve stable entity keys alongside outputs so retries can be reconciled.

Validate the artifact before a wide scoring run

For deterministic requirements, record library versions, thread settings, hardware, and stochastic options. Floating-point reductions can differ across execution orders even with fixed seeds. Distinguish bitwise reproducibility from agreement within a declared tolerance; test the level the downstream decision actually requires.

Make retries idempotent

Write outputs under a run-specific staging prefix. Each partition has a completion record and checksum. Validate the full expected key universe, then publish a small pointer or manifest to the completed version using the storage system’s supported atomic or transactional primitive. Do not assume object-store directory renames are atomic.

Retries should overwrite or deduplicate the same logical partition, not append duplicate predictions. A consumer resolves one published manifest and reads only its listed objects. Keep the previous completed version available for rollback. Model version, feature version, and output version travel together.

Operational metrics explain model incidents

Monitor input freshness, missing features, join amplification, partition failures, output coverage, score distributions, and runtime. A score shift after a schema change may be a pipeline bug rather than concept drift. Replaying a frozen snapshot through old and new pipelines helps locate the difference. Review the learned model only after confirming the data and transformation contract.

Sources and further reading

• Spark SQL performance tuning — Partitioning, joins, and execution tuning.
• MLflow model signatures — Recording and validating model input/output contracts.