Train, Validation, and Test
A model scores 98% on the examples used to build it. Is it ready? That number cannot tell us: the system may have learned useful patterns, memorized the examples, or benefited from features we chose after inspecting their answers.
Train, validation, and test sets separate learning, choosing, and measuring. The distinction is about which decisions a row may influence. This chapter follows one small project from the first split to a final result; the hold-out chapter covers how to construct the split.
Three sets answer three different questions
Imagine predicting whether a support ticket needs escalation. Each row contains information available when the ticket arrives, and a label records whether escalation was needed. For now, suppose the 1,000 rows are independent and represent the tickets we expect to receive.
The training algorithm changes parameters using training rows. For a linear model, those parameters are coefficients; for a tree, they include split thresholds and leaf predictions. Validation answers a different question: which training recipe should we use? Tree depth, regularization strength, feature choices, and early-stopping time belong here.
The test set answers: how does the chosen system perform on held-out examples? It is not another place to pick the winner. The same boundaries apply whether the choices are made by a search program or by a person reading error reports. The scikit-learn evaluation guide explains why a second held-out set is needed when model selection is involved.
Follow one decision through the workflow
Suppose escalation is uncommon and missing an escalation is expensive. Before comparing models, we decide to maximize recall while requiring at least 80% precision on validation. Recall asks how many real escalations we catch; precision asks how many tickets we flag actually need escalation.
Train two candidate trees on the same 600 rows. The following counts are an illustrative outcome, not measurements from a supplied dataset. The 200 validation tickets contain 20 actual escalations:
| Candidate | Correctly flagged | All flagged | Precision | Recall |
|---|---|---|---|---|
| Shallow tree | 12 of 20 | 15 | 12 / 15 = 80% | 12 / 20 = 60% |
| Deeper tree | 16 of 20 | 25 | 16 / 25 = 64% | 16 / 20 = 80% |
The deeper tree catches more escalations, but it fails the precision requirement. Under the rule we chose, the shallow tree wins. We could also tune a probability threshold on validation, but that would be another development decision to record, not a reason to open the test set.
Freeze the selected pipeline and threshold. If it correctly flags 14 of the test set’s 20 escalations and flags 17 tickets overall, its test recall is 70% and its precision is about 82.4%. Report both counts and rates. The difference from validation is ordinary sampling variation unless further evidence suggests otherwise.
Passing 80% precision on a small validation sample does not prove future precision will stay above 80%. The test sample supplies independent evidence under the assumed sampling conditions; deployment can still bring different tickets.
Validation influences learning even without a gradient
Suppose you try 50 feature sets, compare all 50 on the same validation rows, and keep the highest score. Some candidates will benefit from quirks of those particular rows. Choosing the maximum favors those lucky candidates as well as useful ones. Repeating this process makes the validation set part of your design process.
That is its intended role. The mistake is presenting the winning validation score as an independent assessment. The test set provides that assessment only while its outcomes have not helped choose the system. Reading test errors and then changing the features also uses test information, even if no training function receives the test labels.
“Use the test set once” means one final evaluation stage after choices are fixed. Computing several planned metrics or subgroup summaries at that stage is reasonable. If the result prompts further development, describe it as development evidence and use fresh held-out data for the next independent assessment.
The split includes everything the pipeline learns
A model is more than its final classifier. The vocabulary, missing-value replacements, scaling statistics, feature selection, and classifier together determine a prediction. During comparison, fit all of them on the 600 training tickets. Apply their saved transformations to validation and test.
For example, fitting a text vocabulary on all 1,000 tickets lets held-out documents affect which features exist. A pipeline makes the intended data flow easier to enforce:
Scikit-learn’s leakage guide demonstrates this issue with learned preprocessing. Fixed operations such as parsing a date according to a predetermined format do not estimate population statistics, but feature availability still has to match prediction time.
Can we train on validation after choosing?
Yes. Once the recipe is fixed, one common procedure is to refit it from scratch on all 800 development tickets, then evaluate that refitted system on the 200 test tickets. This gives the final fit more data. The test score now describes the refitted system, not the earlier 600-ticket fit.
Refitting needs a rule. If early stopping selected a training duration, decide how that duration will be used in the final fit. A threshold or calibrator learned for one fitted model may not transfer perfectly to another. Either retain the selected model or plan the refitting and calibration procedure before looking at test results.
Does every project need a permanent validation set?
No. Cross-validation can perform the validation role inside the development data, allowing each row to help train some candidate fits. The final test set remains outside those comparisons. Nested cross-validation is another design for evaluating a selection procedure; it is covered in the cross-validation series.
Choose the boundary before the percentages
The independent-ticket assumption matters more than the illustrative 60/20/20 ratio. If one customer supplies many nearly identical tickets, a random row split may evaluate familiar customers. If the goal is future tickets, a split by time may be more appropriate. Use group splitting for unseen entities and time splitting for future periods.
Then choose sizes that leave enough training data and enough evaluation examples for the outcomes you care about. Our example’s 20 positive test cases make recall change by five percentage points for every single case. A large total row count can hide a very small amount of evidence about the rare class.
Save row identifiers, the split rule, the metric and threshold-selection rule, and the final pipeline. These are what let another run reproduce the evaluation. The next chapter turns these roles into a concrete partition.
References and further reading
- Scikit-learn: cross-validation and evaluation — training, validation, and test roles; model selection; and alternatives to a fixed validation set.
- Scikit-learn: data leakage — why learned transformations must respect the evaluation boundary.
- Scikit-learn: train_test_split — the splitting API used in the next chapter.