Neel Mishra
← All Posts
Deep Learning · Data Preparation and Preprocessing · Feature Selection

Correlation Filter

September 2026 · 15 min read

A correlation filter looks for input columns that move together so strongly that you may prefer to keep only one. For example, the same measured length in metres and centimetres is an exact duplicate of information. Removing one can simplify a model and its input pipeline.

Less-than-perfect correlation requires more judgment. Two similar sensors may contain different useful details. This chapter explains the statistic, then a concrete selection rule whose decisions you can inspect. The series introduction covers the broader reasons to select features.

Look at the relationship before choosing a cutoff

Pearson correlation describes how closely two columns follow a straight-line relationship. It ranges from −1 to +1. The sign gives the direction; the magnitude gives the strength of the linear relationship.

Feature xFeature u−202−406u = 2x + 1. Correlation is +1: an exact increasing linear relationship.
Zero Pearson correlation does not rule out a relationship. In the curved example, u = x² determines the second feature exactly, but the symmetric pattern has correlation zero. Each point is one row containing two input features.

For redundancy checks, use absolute correlation: a value near −1 can reveal the same information just as clearly as a value near +1. A filter that checks only positive correlations misses reversed measurements. Changing units by a positive scale factor or adding an offset does not change Pearson correlation, unlike the raw variance in the previous chapter.

The equation measures whether deviations agree

For paired values $x_i$ and $u_i$ in training row $i$, let $\bar x$ and $\bar u$ be the two column means. Subtract those means. When both deviations have the same sign, their product is positive; when they point in opposite directions, it is negative.

Add the products, then divide by the sizes of the two deviation vectors. The denominator removes the units and prevents a larger measurement scale from inflating the result:

$$r(x,u)=\frac{\sum_i(x_i-\bar x)(u_i-\bar u)}{\sqrt{\sum_i(x_i-\bar x)^2}\sqrt{\sum_i(u_i-\bar u)^2}}.$$

A constant column makes a denominator factor zero, so its correlation is undefined. Handle it separately with the variance filter; do not replace an undefined correlation with zero and call that evidence of independence.

Read one pair at a time

Consider three synthetic training columns A, B and C. Their four rows are shown in the Python example below. A correlation matrix puts every pair's score in a table. Its diagonal compares each column with itself, and the entries mirror across that diagonal.

rABC
A1.0000.8940.707
B0.8941.0000.949
C0.7070.9491.000
At a cutoff of 0.85, A–B and B–C are flagged, but A–C is not. The table is rounded for reading; selection uses the unrounded values. These four rows illustrate an algorithm, not a reliable estimate of real-world correlations.

“A is similar to B” and “B is similar to C” do not imply that A and C pass the same cutoff. Consequently, finding flagged pairs is only the first step. We still need a rule for deciding which member of a pair to keep.

A simple rule: compare candidates with retained columns

Choose a priority order before filtering. You might prefer inputs that are cheaper to acquire, available more reliably, or easier to interpret. Then scan the columns in that order:

  1. Keep the first column.
  2. For each later column, compare it with the columns already kept.
  3. Drop it if any absolute correlation exceeds the cutoff; otherwise keep it.

This is a greedy rule: it makes each decision immediately without searching every possible subset. We use a strict comparison, $|r|>h$, where $h$ is the chosen cutoff. Equality is allowed. There is no universal cutoff that guarantees the best model.

Cutoff

Priority order

0.8940.9490.707ABC

Solid green line: pair exceeds the cutoff. Filled node: retained column.

  1. Keep A: first candidate.
  2. Drop B: its correlation with retained A exceeds 0.85.
  3. Keep C: its correlation with A does not exceed 0.85.
Retained columns, in order: A, C.
The priority order can change both the identity and the number of retained columns. At 0.85, starting with A keeps A and C; starting with B keeps only B. Compare the resulting models before adopting either subset.

Notice what happens to C when A comes first: B has already been removed, so B cannot block C. A shortcut that removes every column correlated with any earlier column would also remove C, producing a different algorithm. Comparing only with retained columns makes the intended rule explicit.

After the scan, no retained pair exceeds the cutoff on this training sample. That is the guarantee. It does not guarantee minimum feature count, preserved predictive quality or the absence of relationships involving several columns together.

Fit once, save the names, apply the same selection

Compute the matrix from training rows and retain the chosen names and their order with the model. At inference, select those columns from the new data; do not recompute correlations on each prediction batch.

Python: reproduce the matrix, both priority orders and a new prediction row
import numpy as np
import pandas as pd

def fit_filter(train, priority, threshold=0.85):
    if not np.isfinite(threshold) or not 0 <= threshold <= 1:
        raise ValueError("threshold must lie between 0 and 1")
    if (not train.columns.is_unique or len(priority) != len(train.columns)
            or len(set(priority)) != len(priority)
            or set(priority) != set(train.columns)):
        raise ValueError("priority must contain each column exactly once")
    if train.shape[0] < 2 or train.shape[1] == 0:
        raise ValueError("need at least two rows and one column")
    if not np.isfinite(train.to_numpy(dtype=float)).all():
        raise ValueError("resolve missing and nonfinite values first")
    corr = train.corr(method="pearson")
    if not np.isfinite(corr.to_numpy()).all():
        raise ValueError("remove constant or numerically invalid columns first")
    keep, dropped = [], {}
    for name in priority:
        conflict = next((k for k in keep if abs(corr.loc[name, k]) > threshold), None)
        if conflict is None:
            keep.append(name)
        else:
            dropped[name] = conflict
    return keep, dropped

train = pd.DataFrame({
    "A": [-1, -1, 1, 1],
    "B": [-1.5, -0.5, 0.5, 1.5],
    "C": [-2, 0, 0, 2],
})
print(train.corr().round(3))
keep, dropped = fit_filter(train, ["A", "B", "C"])
print(keep, dropped)  # ['A', 'C'], {'B': 'A'}
print(fit_filter(train, ["B", "A", "C"]))
# (['B'], {'A': 'B', 'C': 'B'})

new = pd.DataFrame({"A": [3], "B": [4], "C": [5]})
print(new.loc[:, keep])  # Save and reuse the fitted names and order.

The function deliberately requires complete finite numeric columns. The returned dictionary records which retained column caused each removal, making the selection auditable. In cross-validation, run this fit step separately inside each training fold before fitting the predictor; do not select once on the full dataset.

What the filter cannot decide for you

Feature–feature correlation is different from feature–target association. The filter above never reads the target. Ranking features by their correlation with the target is a separate supervised procedure, and it can miss curved or interacting effects. Neither calculation establishes causality.

A very similar pair may still contain predictive differences. Inspect what each input measures, compare the selected model against the full eligible feature set, and evaluate relevant subgroups. If you use target-based scores to choose a priority order or tune the cutoff, learn those choices within the training and validation boundary too.

Pairwise filtering also cannot remove every form of multicollinearity. For example, one column may be a linear combination of several others without having near-perfect correlation with any single one. Regularization or dimensionality reduction may fit that problem better; start with L1 selection or PCA.

Missing values, outliers and uncertain correlations

Pandas computes correlations using the available paired observations. With missing data, different pairs may therefore use different rows and sample counts. Set a minimum count, inspect the missingness, or use a training-fitted imputation policy before comparing scores.

Outliers and changes in the sampled population can alter the correlation matrix. A cutoff close to an estimated score can make membership unstable. Inspect scatterplots and compare selections across training folds. Spearman correlation compares ranks and can reveal monotonic curved relationships, but even rank correlation is not a general test of dependence.

Implement the same retained-column rule

The C++ version computes centered products directly and checks each candidate only against retained columns. It accepts a list of complete columns and a priority list of column indices. Use the saved-column projection from the introduction to apply the result.

C++17: Pearson correlation and greedy selection
#include <algorithm>
#include <cmath>
#include <iostream>
#include <stdexcept>
#include <vector>

double pearson(const std::vector<double>& x, const std::vector<double>& y) {
    if (x.size() != y.size() || x.size() < 2)
        throw std::invalid_argument("paired columns need at least two rows");
    long double mx = 0, my = 0;
    for (std::size_t i = 0; i < x.size(); ++i) {
        if (!std::isfinite(x[i]) || !std::isfinite(y[i]))
            throw std::invalid_argument("finite values required");
        mx += static_cast<long double>(x[i]);
        my += static_cast<long double>(y[i]);
    }
    mx /= x.size(); my /= x.size();
    long double xx = 0, yy = 0, xy = 0;
    for (std::size_t i = 0; i < x.size(); ++i) {
        long double dx = static_cast<long double>(x[i]) - mx;
        long double dy = static_cast<long double>(y[i]) - my;
        xx += dx * dx; yy += dy * dy; xy += dx * dy;
    }
    if (!std::isfinite(xx) || !std::isfinite(yy) || !std::isfinite(xy))
        throw std::overflow_error("rescale before computing correlation");
    if (xx <= 0 || yy <= 0) throw std::invalid_argument("constant column");
    double r = static_cast<double>((xy / std::sqrt(xx)) / std::sqrt(yy));
    return std::clamp(r, -1.0, 1.0);  // Contain floating-point rounding.
}

std::vector<std::size_t> fit_filter(
        const std::vector<std::vector<double>>& columns,
        const std::vector<std::size_t>& priority, double threshold) {
    if (!std::isfinite(threshold) || threshold < 0 || threshold > 1)
        throw std::invalid_argument("threshold outside [0,1]");
    if (columns.empty() || priority.size() != columns.size())
        throw std::invalid_argument("priority must cover every column");
    std::vector<bool> seen(columns.size(), false);
    for (auto j : priority) {
        if (j >= columns.size() || seen[j])
            throw std::invalid_argument("invalid or repeated priority index");
        seen[j] = true;
        if (columns[j].size() != columns[0].size())
            throw std::invalid_argument("unequal column lengths");
        pearson(columns[j], columns[j]); // Validate each input column.
    }
    std::vector<std::size_t> keep;
    for (auto j : priority) {
        bool conflict = false;
        for (auto k : keep) {
            if (std::abs(pearson(columns[j], columns[k])) > threshold) {
                conflict = true; break;
            }
        }
        if (!conflict) keep.push_back(j);
    }
    return keep;
}

int main() {
    std::vector<std::vector<double>> columns = {
        {-1,-1,1,1}, {-1.5,-.5,.5,1.5}, {-2,0,0,2}
    };
    for (auto j : fit_filter(columns, {0,1,2}, .85)) std::cout << j << ' ';
    std::cout << '\n';  // 0 2: keep A and C.
}

For $N$ rows and $P$ columns, the worst-case work is proportional to $NP^2$. This version computes correlations as needed instead of storing a full $P\times P$ matrix. It rejects constant or nonfinite inputs; extreme magnitudes may require rescaling. Near a cutoff, use adequate numerical precision and inspect the decision rather than rounding the matrix first.

References and further reading