Variance Threshold

September 2026 · 13 min read

Variance thresholding asks whether a column changes enough across the training rows to keep it. It is a simple cleanup step, especially for constant columns. It does not look at the target, so it cannot tell you whether a changing column is useful or a rare value matters.

The previous chapter explains why selection needs validation. Here we will make one small filter concrete: measure variation, compare it with a threshold, and save the surviving columns.

Start with eight visible training values

Imagine three columns in a training table. The first records the same setting for every row. The other two are flags: 1 means on and 0 means off. Each box below is one row, in the same order.

Measure spread around the mean

For one column, first calculate its mean. Then subtract that mean from every value, square each difference, and average those squares. Squaring prevents positive and negative deviations from cancelling. This average is the variance.

Let $x_i$ be the value in training row $i$, let $N$ be the number of training rows, and write $\bar x$ for their mean. We use the same divisor $N$ for both averages:

For the constant setting, the mean is 42 and every deviation is zero, so its variance is zero. The calculation describes the observed training values; it does not assert that the column can never change in future data.

This is the ddof=0 convention in NumPy. A sample-variance calculation with divisor $N-1$ produces a different number. When comparing your calculation with the selector, use the same convention.

For a binary flag, frequency determines variance

Let $p$ be the fraction of training values equal to 1. The mean is then $p$. Ones deviate from it by $1-p$; zeros deviate by $-p$. Averaging their squared deviations gives:

The rare flag has $p=1/8$ and variance $7/64=0.109375$. The balanced flag has $p=1/2$ and variance $0.25$, the largest possible variance for a 0/1 feature. A flag that is almost always 1 has low variance too; exchanging 0 and 1 leaves $p(1-p)$ unchanged.

Keep a column only if its variance exceeds the threshold

Choose a nonnegative threshold $h$. The rule is keep when $v>h$; equality is removed. With $h=0$, a column survives as long as it varies at all in the training data. A positive threshold removes some varying columns too.

Add two more columns to the example: eight lengths in metres and those same lengths multiplied by 100 to express them in centimetres. The values and code appear below. Change the threshold to see the decision for all five columns.

The same information can have a different variance

The lengths in metres are 1.00, 1.01, 0.99, 1.02, 0.98, 1.01, 0.99 and 1.00. Their mean is 1.00. Expressing them in centimetres multiplies every deviation from the mean by 100, so each squared deviation grows by $100^2=10{,}000$.

A threshold of 0.01 drops the metre column but retains the centimetre column. It has learned nothing new about usefulness. A raw variance threshold has squared units. A single positive threshold across unrelated measurements therefore needs a meaningful scale convention or separate treatment for different feature families.

Standardizing every nonconstant column to unit variance first does not solve this selection problem: a threshold below 1 will normally keep all of them. Standardization has removed the original spread differences you wanted to inspect. Handle constant columns or choose meaningful near-constant criteria before standardization; use scaling afterwards when the model needs it. The standardization chapter explains that separate purpose.

Low variance is not low predictive value

Suppose our target in this constructed eight-row table is exactly the rare flag. The flag then predicts that target perfectly, even though its variance is lower than the balanced flag's. A threshold at 7/64 would throw it away because the selector never sees the labels. This illustrates a limitation of the rule, not evidence that eight rows establish reliable generalization.

Conversely, a random measurement can vary widely and still be unrelated to the target. Use variance filtering as a simple input check, then evaluate the resulting model. For rare categories or events, inspect performance on those cases before removing their indicators. Low variation alone is not a reason to discard an operationally meaningful feature.

Fit the selection on training data and reuse it

The fitted selector stores which columns passed. Apply that same selection, in the same order, to validation data and new predictions. A column that was constant during training remains excluded even if its value changes later. Re-fitting on a prediction batch could change the model's input schema.

import numpy as np
import pandas as pd
from sklearn.feature_selection import VarianceThreshold

length = np.array([1, 1.01, .99, 1.02, .98, 1.01, .99, 1])
train = pd.DataFrame({
    "constant": [42] * 8,
    "rare_flag": [1, 0, 0, 0, 0, 0, 0, 0],
    "balanced_flag": [1, 1, 1, 1, 0, 0, 0, 0],
    "length_m": length,
    "length_cm": 100 * length,
})
selector = VarianceThreshold(threshold=7 / 64).fit(train)
print(selector.variances_)  # [0, .109375, .25, .00015, 1.5]
print(selector.get_feature_names_out())
# ['balanced_flag', 'length_cm']; equality with the threshold is removed.

new = pd.DataFrame([[43, 1, 0, 1.03, 103]], columns=train.columns)
print(selector.transform(new))  # [[0, 103]]; use the saved selection.
print(train.var(ddof=0).to_numpy())  # Same variance convention.

try:
    VarianceThreshold(threshold=2).fit(train)
except ValueError:
    print("No feature survives; revise the preprocessing decision.")

During cross-validation, put the selector inside the pipeline so each training fold learns its own selection. If you tune the threshold using prediction quality, treat it as a model-development choice and preserve a final evaluation boundary. The selection introduction covers that boundary in detail.

The calculation from scratch

The implementation below follows the two averages directly: one pass for the mean, a second for squared deviations. It accepts finite numeric columns, validates their lengths, and returns the retained indices. Save those indices and apply them using the previous chapter's column-selection function.

#include <cmath>
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <vector>

double variance(const std::vector<double>& column) {
    if (column.empty()) throw std::invalid_argument("empty column");
    long double total = 0;
    for (double x : column) {
        if (!std::isfinite(x)) throw std::invalid_argument("finite values required");
        total += static_cast<long double>(x);
    }
    const long double mean = total / column.size();
    long double squared = 0;
    for (double x : column) {
        const long double difference = static_cast<long double>(x) - mean;
        squared += difference * difference;
    }
    const double result = static_cast<double>(squared / column.size());
    if (!std::isfinite(result)) throw std::overflow_error("variance overflow");
    return result;
}

// Input is a list of columns, each containing the same training rows.
std::vector<std::size_t> fit_selection(
        const std::vector<std::vector<double>>& columns, double threshold) {
    if (!std::isfinite(threshold) || threshold < 0)
        throw std::invalid_argument("nonnegative finite threshold required");
    if (columns.empty()) throw std::invalid_argument("no columns");
    std::vector<std::size_t> keep;
    for (std::size_t j = 0; j < columns.size(); ++j) {
        if (columns[j].size() != columns[0].size())
            throw std::invalid_argument("unequal column lengths");
        if (variance(columns[j]) > threshold) keep.push_back(j);
    }
    if (keep.empty()) throw std::invalid_argument("no feature survives");
    return keep;
}

int main() {
    const std::vector<std::vector<double>> columns = {
        {42,42,42,42,42,42,42,42},
        {1,0,0,0,0,0,0,0},
        {1,1,1,1,0,0,0,0},
        {1,1.01,.99,1.02,.98,1.01,.99,1},
        {100,101,99,102,98,101,99,100}
    };
    std::cout << std::setprecision(12);
    for (const auto& column : columns) std::cout << variance(column) << ' ';
    std::cout << '\n';
    for (auto j : fit_selection(columns, 7.0 / 64)) std::cout << j << ' ';
    std::cout << '\n';  // Selected indices: 2 4.
}

References and further reading

• Scikit-learn: removing features with low variance — the baseline filter and the binary variance relationship.
• Scikit-learn: VarianceThreshold — fitting, saved feature names, missing-value support and the no-surviving-feature error.
• NumPy: variance — the squared-deviation calculation, divisor conventions and floating-point precision.
• Scikit-learn: StandardScaler — training-set mean and variance, and scaling to unit variance.