Splitting & Stratification
A data split is the first draft of your evaluation claim. Holding out random rows asks about new examples from the same collection. Holding out entire customers asks about unfamiliar customers. Holding out later dates asks about future predictions. These are different questions, even if every split is called “80/20.”
This series helps you choose the question, build the partition, and keep the rest of preprocessing inside its boundaries. Start here for the decisions; use the linked chapters for the worked examples and runnable code.
Decide what the model must generalize to
Before choosing a percentage, complete this sentence: “At evaluation time, the model should not already know…” The answer determines the unit you hold out.
| Intended prediction | What must be held out? | Start with |
|---|---|---|
| Another independent example from the same population | Individual rows, after checking duplicates and related sources | Random hold-out |
| An unfamiliar person, device, or source document | Whole entities or related-example groups | Group splitting |
| A later prediction from a changing process | Later dates, with feature and label availability enforced | Time-series splitting |
| Both a future period and an unfamiliar entity | A design satisfying both constraints | Combine the group and time checks |
For example, a support system may receive more tickets from existing customers or tickets from entirely new customers. A random ticket split and a customer hold-out can legitimately produce different scores because they measure different tasks. State which result matters before judging which score looks better.
Assign three roles before fitting anything
Once the unit is clear, separate fitting, selection, and final evaluation. A model includes every learned transform before the classifier, so a vocabulary or imputer belongs inside the same boundary as its weights.
The three-set chapter follows a 1,000-ticket example through candidate selection and final reporting. The hold-out chapter turns the roles into row assignments and explains the common two-stage fraction mistake: reserving 20% for validation after holding out 20% for test leaves 16% of the original data in validation.
Choose counts rather than blindly inheriting a ratio. The important quantity may be the number of positive cases, distinct customers, or forecast periods—not the total number of rows. A million measurements from ten devices still contain only ten devices for studying transfer between devices.
One split is evidence, not the exact answer
A held-out score changes when different examples are evaluated. The fitted model can also change when its training sample or initialization changes. Those are separate sources of variation, and one fixed split does not reveal both.
For a fixed classifier that gets 180 of 200 independent examples correct, 90% accuracy has an approximate standard error of 2.1 percentage points under a simple binomial model. The hold-out chapter introduces that calculation and its assumptions. Repeated observations from one entity or overlapping forecast targets require more care.
Cross-validation rotates the development roles and compares candidates on several held-out blocks. It can make a small dataset more useful, but it does not eliminate selection bias, dependence, or an invalid split. The comparison chapter works through a case where the candidate ranking changes across folds and counts the additional fits.
Preserve class counts when that matches the design
After choosing the independence boundary, inspect label counts. In classification, a stratified split allocates each class approximately in the requested proportions. That helps prevent random class-count drift, especially when a class is uncommon. It does not balance every class to the same size.
The chart below uses fixed illustrative counts. In the random allocation, training has 233 positives and test has 63. In the stratified allocation, those counts are 237 and 59. Both allocations account for the same 296 positives; their distributions between the subsets differ.
▶ Random vs Stratified Split
Constructed example: 1,000 rows contain 296 positives. Each split assigns 800 rows to training and 200 to test. Step through the class counts expressed as rates.
The stratified rates are close to the dataset’s 29.6%, rather than exactly equal, because examples come in whole numbers. Matching this class ratio does not prove the subsets match on geography, difficulty, repeated users, or future behavior. It also does not show that 29.6% is the true rate outside this collection.
The interactive stratification chapter lets you draw actual card assignments and see how a random test set can contain no positive at all. It also explains why forcing a stable ratio cannot compensate for having only one positive test example.
Trace information, not just filenames
A split is undermined if information crosses its boundary later. Feature selection on all labels can favor accidental patterns that look good on the held-out examples. Even fitting an unlabeled transform on all rows changes the system using information from the evaluation collection.
The leakage chapter demonstrates this with independent random features and labels: selecting features with access to test labels produces 73% accuracy, while fitting the selection only on training gives 50.5% in the same experiment. The lesson is to audit the whole data path, not to assume a high score proves leakage or a low score proves its absence.
Related documents and delayed labels create different routes across the boundary. Keep source families together when evaluating unfamiliar documents. For forecasts, check that every training label was available before the model fit and that every input was available before its prediction. A correct pipeline cannot compensate for an incorrect evaluation design.
Make the experiment repeatable and inspectable
Save the row or group assignments, data version, split rule, and relevant timestamps. Record the model and preprocessing configuration, metric definitions, and selection procedure. Use the same validation assignments across candidates so differences are not confounded by different evaluation examples.
A seed alone is insufficient: a shuffle can choose different real examples when the source table is reordered. The reproducibility chapter explains seeds, mutable random generators, saved split maps, and a stable ID-based assignment for growing data. It separates replaying a run from testing whether a method is robust across planned runs.
Read the chapter that resolves your next decision
Follow fitting, model selection, threshold choice, refitting, and final assessment through one project.
Partition aligned rows, choose sizes, check the split, and interpret sampling uncertainty.
Draw class-aware partitions, calculate the chance of a missing rare class, and understand the limits.
Expose identity memorization, preserve related examples, and choose row or group weighting.
Respect prediction time, delayed labels, gaps, and the forecast horizon.
Reproduce a misleading noise-only score and trace the information that caused it.
Save meaningful experiment artifacts and separate repeatability from robustness.
Compare evidence, data use, computation, and the role of independent final evaluation.
Carry the boundary into preprocessing
With the evaluation design fixed, continue to feature scaling. Fit its statistics only on the training portion of each experiment, then apply the fitted transform to held-out rows. The data-preparation overview places this step in the wider pipeline, and the C++ split implementation covers the low-level partition mechanics.
References and further reading
- Scikit-learn: cross-validation and evaluation — hold-outs, group and time-aware designs, and limitations of stratification.
- Scikit-learn: common pitfalls — learned preprocessing, leakage, and control of randomness.
- Forecasting: Principles and Practice — evaluating predictions at rolling forecast origins.