Mutual Information

September 2026 · 15 min read

Mutual information measures how much knowing a feature reduces uncertainty about the target, on average. It can recognize relationships that do not follow a straight line. That makes it useful for ranking candidate features, but a high score on training data is still only evidence to investigate.

The previous chapter compared input features with each other to find redundancy. Here we compare each feature with the target. This is a supervised selection step: the labels influence the ranking.

Does the feature change what we know about the label?

Start with eight training rows: four labels are 0 and four are 1. Without a feature, the two labels are equally likely in this table. Now group the rows by a feature that can take values A or B.

In the “some information” setting below, group A contains three zeros and one one; group B contains one zero and three ones. Knowing the group changes our label probabilities from 50/50 to 75/25 or 25/75. It helps, but it does not settle every prediction.

Put a number on that uncertainty

Entropy assigns a surprise of $-\log_2 p(y)$ to a label whose probability is $p(y)$, then averages over the labels. A rare outcome is more surprising than a common one. Let $Y$ denote the target and let the sum below run over its possible classes:

Log base 2 measures entropy in bits. For the balanced target, both classes have probability 1/2, giving $H(Y)=1$ bit. If a group contains only one label, its entropy is zero. Terms with zero probability contribute zero; we do not evaluate the logarithm of zero. The binary-entropy chapter develops this measure in more detail.

Inside group A, the entropy of proportions 3/4 and 1/4 is about 0.8113 bit. Group B has the reversed proportions and the same entropy. Since each group contains half the rows, the average uncertainty after observing the feature is also 0.8113 bit.

Write $H(Y\mid X)$ for that average remaining uncertainty. Each feature value $x$ contributes according to how often it occurs, $p(x)$. Mutual information, $I(X;Y)$, is what we remove from the original uncertainty:

For the example, $I(X;Y)=1-0.8113\approx0.1887$ bit. Unequal group sizes would need unequal weights; a tiny pure group should not dominate the score. This is the same entropy reduction used in decision-tree information gain when grouping by a split outcome.

For the true distribution, MI is nonnegative and equals zero exactly when the variables are independent. It is symmetric: $I(X;Y)=I(Y;X)$. For this discrete target, it cannot exceed $H(Y)$. It has no positive or negative direction and is not an accuracy percentage.

A curved relationship can contain useful information

Consider another balanced eight-row table. The feature takes values −2, −1, 1 and 2, each twice. The target is 1 at the outer values and 0 at the inner values. Increasing the feature does not consistently increase or decrease the target, but knowing the feature determines it.

Choose an estimator that matches the feature

For discrete values, we can estimate probabilities by counting feature–label pairs. For continuous measurements, almost every observed value may be distinct, so treating each raw number as a category is usually inappropriate. The score then reflects tiny training groups rather than a well-supported relationship.

Scikit-learn provides mutual_info_classif for a discrete target and mutual_info_regression for a continuous target. Mark discrete feature columns explicitly. With discrete_features='auto', dense inputs are treated as continuous and sparse inputs as discrete; an integer dtype alone does not specify the intended estimator.

Its estimators for continuous variables use nearest-neighbor distances. The neighbor count changes the bias–variance tradeoff, and a fixed random_state makes the small perturbations used for repeated values reproducible. Estimates can be noisy; negative estimates are clipped to zero. A reported zero is therefore not a proof of population independence.

import numpy as np
from sklearn.metrics import mutual_info_score
from sklearn.feature_selection import mutual_info_classif

# Rows are feature categories; columns are target classes 0 and 1.
tables = {
    "independent": np.array([[2, 2], [2, 2]]),
    "helpful": np.array([[3, 1], [1, 3]]),
    "perfect": np.array([[4, 0], [0, 4]]),
}
for name, counts in tables.items():
    nats = mutual_info_score(None, None, contingency=counts)
    print(name, nats / np.log(2))
# independent: 0; helpful: 0.1887218755; perfect: 1 bit.

y = np.array([0, 0, 0, 0, 1, 1, 1, 1])
helpful = np.array([0, 0, 0, 1, 0, 1, 1, 1])
X = np.column_stack([helpful, np.zeros(8, dtype=int), 1 - helpful])
scores = mutual_info_classif(X, y, discrete_features=True, random_state=0)
print(scores / np.log(2))  # [0.1887218755, 0, 0.1887218755]
# A feature and its reversed copy score equally; ranking does not remove redundancy.

# The nonlinear example: four discrete feature values, each observed twice.
curved_x = np.array([-2, -2, -1, -1, 1, 1, 2, 2])
curved_y = np.array([1, 1, 0, 0, 0, 0, 1, 1])
print(np.corrcoef(curved_x, curved_y)[0, 1])  # 0
print(mutual_info_score(curved_x, curved_y) / np.log(2))  # 1 bit

# One arbitrary ID per row: the empirical groups are always pure.
id_counts = np.eye(2, dtype=int)[y]  # Eight rows, one observed label in each.
print(mutual_info_score(None, None, contingency=id_counts) / np.log(2))  # 1 bit

A unique identifier exposes the estimation trap

Give each training row a different arbitrary ID and treat those IDs as discrete categories. Every observed category now contains exactly one label. The empirical conditional entropy is zero, regardless of how the labels were assigned.

This is one reason to inspect high-cardinality features rather than accepting their empirical MI scores at face value. Coarse binning is not an automatic remedy either: it changes the information available to the estimator and adds binning choices that must be learned within training.

A good individual score is not a complete subset strategy

Two duplicate features can each have a high MI score because each tells us the same thing about the target. A simple top-k ranking may keep both. Conversely, the equally likely XOR example has zero MI between either individual input and the target, while the pair determines it. Ranking columns independently cannot capture every useful combination.

Use MI to propose a candidate subset, then fit and evaluate the downstream model. Compute scores using training rows and labels only. Choose the feature count and estimator settings through validation; repeat selection inside every training fold. The selection leakage example explains why ranking on all labels invalidates later evaluation.

Inspect score stability and the meaning of selected inputs alongside predictive quality. MI does not establish causality, and an informative feature only helps if the model can use its relationship with the target.

Compute discrete MI directly from counts

The C++ implementation works with a count table such as the interactive example. It implements the discrete calculation, not the nearest-neighbor estimator used for continuous measurements.

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <vector>

using Counts = std::vector<std::vector<std::uint64_t>>;

double mutual_information_bits(const Counts& counts) {
    if (counts.empty() || counts[0].empty())
        throw std::invalid_argument("empty count table");
    std::vector<long double> rows(counts.size(), 0);
    std::vector<long double> cols(counts[0].size(), 0);
    long double total = 0;
    for (std::size_t i = 0; i < counts.size(); ++i) {
        if (counts[i].size() != cols.size())
            throw std::invalid_argument("ragged count table");
        for (std::size_t j = 0; j < cols.size(); ++j) {
            long double n = counts[i][j];
            rows[i] += n; cols[j] += n; total += n;
        }
    }
    if (total == 0) throw std::invalid_argument("no observations");
    long double information = 0;
    for (std::size_t i = 0; i < counts.size(); ++i) {
        for (std::size_t j = 0; j < cols.size(); ++j) {
            if (counts[i][j] == 0) continue; // Zero-mass terms contribute zero.
            long double n = counts[i][j];
            information += (n / total) * std::log2((n * total) / (rows[i] * cols[j]));
        }
    }
    return std::max(0.0, static_cast<double>(information)); // Roundoff guard.
}

int main() {
    std::cout << std::setprecision(12);
    for (const auto& table : std::vector<Counts>{
             {{2,2},{2,2}}, {{3,1},{1,3}}, {{4,0},{0,4}}})
        std::cout << mutual_information_bits(table) << '\n';
    // 0, 0.188721875541, 1
}

References and further reading

• MIT 6.441: entropy, conditional entropy and mutual information — the uncertainty-reduction interpretation and its information-theoretic definitions.
• Scikit-learn: mutual_info_classif — discrete-feature settings, continuous estimators, reproducibility and units.
• Scikit-learn: mutual_info_score — discrete count-table calculations and symmetry.
• Guyon and Elisseeff: An Introduction to Variable and Feature Selection — individual relevance, redundant features and subset evaluation.