Feature Selection
Feature selection chooses which existing inputs reach a model. We might want better predictions, fewer measurements, faster inference or a smaller model to inspect. Those goals can agree, but they do not always lead to the same subset.
This series moves from inexpensive checks to methods that fit and inspect models. Start with the first chapter for intuition, then follow the question that matches your data. The overview connects the decisions; each chapter contains its own worked figures, equations and optional implementations.
Define the prediction before ranking columns
Suppose we predict delivery duration when a parcel leaves a warehouse. Its weight, carrier and planned route length may be available then. The actual delivery-completion timestamp is not. No feature score can make that future information a valid input.
Route length stored in both kilometers and meters is a different issue: both are available, but they represent the same measurement. A rare service category is different again: low frequency alone does not tell us whether it helps prediction. Keep availability, redundancy and predictive usefulness as separate questions.
Choose an evaluation metric that matches the task, such as mean absolute error in minutes, and identify the cost you want to reduce. Removing six encoded columns may save no data collection if all six come from one already-required source. Count the measurements or lookups you can actually eliminate.
Treat selection as part of training
A correlation cutoff, a retained-feature count and a lasso penalty are all modeling choices. If validation results determine them, the validation score has been used for selection. Use an untouched evaluation set or nested cross-validation to assess the resulting procedure. The first chapter demonstrates how selecting from random features can otherwise create an apparently strong model.
Choose how a subset will be proposed
Filters use feature statistics before fitting the final predictor. Variance and input–input correlation check variation and redundancy without using labels. Mutual information and chi-square scores use labels to measure individual associations. These checks are often inexpensive, but individual scores can miss features that help only together.
Wrappers search over subsets through repeated model fits. Recursive feature elimination removes weakly ranked inputs and refits, letting the remaining coefficients or importances change. The search costs more and depends on its learner and removal path; it does not guarantee the best possible subset.
Embedded and model-based methods use what a fitted model learns. Lasso produces sparse coefficients during optimization. Tree importances describe how splits used the inputs. Permutation importance is a separate inspection tool for measuring a fitted model's reliance on an input. Turning any ranking into a subset still requires a selection rule and evaluation.
Find the chapter for your question
See an extra column change a nearest-neighbor prediction, an interaction defeat individual screening, and selection leak evaluation information.
Compute variance from visible samples. Explore rare binary features, unit changes and the strict threshold used by the selector.
Read positive, negative and nonlinear patterns. Inspect how a greedy redundancy filter depends on the order of retained columns.
Build mutual information from counts, see a nonlinear association, and understand why an identifier can look informative in a finite sample.
Compare observed and expected counts. Distinguish a contingency-table test from scikit-learn's nonnegative feature scoring.
Follow actual refits, read elimination ranks, and evaluate whether the selected subset predicts better.
Watch lasso shrink three effects, compare prediction error with penalty cost, and inspect ambiguous choices between duplicate inputs.
Trace tree split credits and shuffle individual columns. Compare training importance with performance on separate validation data.
Decide what a useful reduction would look like
Return to the delivery example. Suppose an increase of at most 0.2 minutes in validation MAE is acceptable if it substantially reduces feature cost. Set that tolerance before inspecting the comparison. These hypothetical results illustrate the decision; they are not measurements from a dataset:
Use the same evaluation folds for each candidate so differences are paired on the same observations. Inspect variation across folds and important data slices before treating a small mean difference as reliable. A threshold on these illustrative means is a decision rule, not statistical proof that two models perform equally.
A model using all eligible inputs remains a valid outcome. Selection can remove useful signal or introduce unstable choices. Keep a reduction because its measured tradeoff serves the task, not because a smaller feature count is automatically better.
Carry the selected representation into prediction
Save the selected names and their order together with fitted encoders, imputers, scaling statistics and the final predictor. New rows must receive that same transformation. Refitting a selector independently on production data can change the meaning or order of the model's inputs.
Also record the development split, metric, selection settings and comparison against the full eligible feature set. That makes a future change in the chosen columns easier to investigate: the cause could be new data, a new representation, a different learner or a changed cost constraint.
References and further reading
- Guyon and Elisseeff: An Introduction to Variable and Feature Selection — selection objectives, individual relevance and evaluating subsets.
- Scikit-learn: Feature selection — filter, recursive and model-based selection tools.
- Scikit-learn: Data leakage — keeping learned preprocessing and feature selection inside training boundaries.