Why Select Features

September 2026 · 15 min read

Feature selection keeps a subset of the available columns. The goal may be better predictions, a cheaper input pipeline or a model that is easier to inspect. A smaller subset is useful when it serves those goals; fewer columns are not automatically better.

The feature-engineering series constructs candidate inputs. This series asks which to retain. Start here for the reasons and evaluation rules, then follow the method chapters for specific selection algorithms.

An extra column can change what “similar” means

Consider a deliberately small regression task whose true answer is $y=x$. We have three training rows A, B and C, each with the useful feature $x$ and an additional coordinate $z$. For the new query, $x=0.49$ and $z=0$.

Use a one-nearest-neighbor model: it copies the target of whichever training row is closest under the chosen input coordinates. Both coordinates in this example lie between 0 and 1, so the problem is not simply that one column uses huge units.

With $x$ alone, B is only 0.01 away. With both coordinates, its distance includes the difference of 1 in $z$. Euclidean distance squares both coordinate differences, adds them, and takes a square root:

A is then closer at distance 0.49, so the model predicts 0 instead of 0.5. Removing $z$ improves this prediction because it makes the comparison match the known target rule. In real data, we do not know that rule in advance; selection must earn its place through validation.

import numpy as np
from sklearn.neighbors import KNeighborsRegressor

X = np.array([[0.0, 0.0], [0.5, 1.0], [1.0, 0.0]])
y = X[:, 0]  # The constructed target depends only on x.
query = np.array([[0.49, 0.0]])
for columns in ([0], [0, 1]):
    model = KNeighborsRegressor(n_neighbors=1).fit(X[:, columns], y)
    distances = np.linalg.norm(X[:, columns] - query[:, columns], axis=1)
    print(columns, np.round(distances, 5), model.predict(query[:, columns]))
# x only: distances [0.49, 0.01, 0.51], prediction 0.5
# x and z: distances [0.49, 1.00005, 0.51], prediction 0.0

There are several reasons to reconsider a column

Redundancy is model-dependent. If two columns are exact copies, a linear model can distribute the same combined effect across different coefficient pairs. Repeating a coordinate also changes its weight in an ordinary Euclidean distance. But two noisy measurements of the same underlying quantity can contain complementary evidence; high correlation alone does not prove that one is disposable.

Some learners can reduce the influence of weak columns through regularization or fitted splits. That does not remove the cost of acquiring those inputs. Measure the complete pipeline's latency, memory, missing-input behavior and maintenance burden alongside predictive quality.

A weak feature alone can be useful with another

Suppose the target is 1 exactly when two binary inputs differ. In the four equally likely combinations below, either input alone sees an equal mix of labels. Together they determine the label perfectly.

A method that ranks each column in isolation can miss this joint relationship. Keeping the pair is only useful if the downstream model can represent the interaction, perhaps through a suitable tree or an explicit interaction feature. There is no model-independent “best list of columns” for every task.

Selection belongs inside the training boundary

A supervised selector uses labels to decide which inputs look promising. If it reads evaluation labels, the evaluation has already influenced the model—even when the final classifier is later fitted using only training rows.

To expose the problem, generate 200 rows with 2,000 independent random numeric features and balanced labels assigned independently of them. Split the rows into 100 for training and 100 for evaluation. Rank individual features with a classification F-score, keep 20 and fit logistic regression.

Among many noise columns, some happen to align with labels in this finite sample. Selecting with all labels retains columns partly because they matched the evaluation outcomes. That is why splitting after feature selection does not repair the problem. The scikit-learn leakage example illustrates the same failure mode with a separate experiment.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(17)
X = rng.normal(size=(200, 2000))
y = np.array([0, 1] * 100)
rng.shuffle(y)  # Labels are independent of all input features.
train, test = train_test_split(
    np.arange(len(y)), test_size=0.5, stratify=y, random_state=8,
)

# Correct boundary: the selector sees only training rows and labels.
pipeline = make_pipeline(
    SelectKBest(f_classif, k=20),
    LogisticRegression(max_iter=2000),
)
pipeline.fit(X[train], y[train])
print(pipeline.score(X[test], y[test]))  # 0.49 for this seeded example.

# Deliberately WRONG, only to reproduce the leakage demonstration.
leaked = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaked_model = LogisticRegression(max_iter=2000).fit(leaked[train], y[train])
print(leaked_model.score(leaked[test], y[test]))  # 0.79: invalid evaluation.

Evaluate the whole selection procedure

Start with a baseline that uses the eligible input set, then compare the selected subset using the same task-appropriate splits and metric. The selection method and the number of retained features are choices that need validation. Repeatedly trying subsets on the final test set turns that set into development data.

Use nested cross-validation when you need to estimate the performance of a procedure that tunes its subset internally. For grouped or temporal data, keep the relevant group or time boundaries in both the selection and model-fitting steps.

Inspect which features or feature families recur across training samples. Several correlated columns may substitute for one another while giving similar predictions. Unstable membership can complicate explanation and maintenance, but it does not by itself establish poor predictive performance. Compare both the selected inputs and the resulting model behavior.

Pick a method that matches the question

These approaches make different assumptions and have different computational costs. Feature selection preserves chosen original columns; a transformation such as PCA instead creates new combined coordinates. A predictive subset also does not, on its own, identify causal drivers.

Apply the saved subset consistently

Once selection is fitted, save the chosen feature names and order with the model. At prediction time, apply that same selection. Re-selecting from each new batch changes the input contract and can put the wrong measurements under the learned weights.

#include <iostream>
#include <stdexcept>
#include <vector>

std::vector<double> select_columns(const std::vector<double>& row,
                                   const std::vector<std::size_t>& columns) {
    std::vector<bool> seen(row.size(), false);
    std::vector<double> selected;
    selected.reserve(columns.size());
    for (auto column : columns) {
        if (column >= row.size()) throw std::out_of_range("unknown column");
        if (seen[column]) throw std::invalid_argument("repeated column");
        seen[column] = true;
        selected.push_back(row[column]);
    }
    return selected;
}

int main() {
    std::vector<double> query = {0.49, 0.0};
    for (double value : select_columns(query, {0})) std::cout << value << ' ';
    std::cout << '\n';  // 0.49: the saved selection retains x only.
}

References and further reading

• Guyon and Elisseeff: An Introduction to Variable and Feature Selection — the goals of selection, relevance, redundancy and feature validity.
• Scikit-learn: nearest-neighbor regression — prediction from neighboring observations and the role of distance.
• Scikit-learn: data leakage during preprocessing — why feature selection must not inspect evaluation labels.
• Scikit-learn: feature selection — filter, recursive and model-based selection interfaces.