Chi-Square Test

September 2026 · 15 min read

A chi-square test asks whether category counts depart from what we would expect if a feature and the target were independent. It can help screen categorical inputs for association with a categorical target. The score measures a discrepancy in counts; it does not promise useful predictions.

The mutual-information chapter measures information in a feature–label table. Here we use a different comparison: observed counts versus expected counts under independence. We will also distinguish the full-table test from scikit-learn's feature-scoring function.

Build the table, then state the independence assumption

Suppose a binary flag is present in 40 of 80 training rows. The target also has 40 examples of each class. Among the rows with the flag, 30 belong to class 0 and 10 to class 1; among those without it, the counts are reversed.

If the flag and class were independent, each flag group would have the overall class proportions: 50% class 0 and 50% class 1. A group of 40 would therefore contain 20 of each class on average.

For category row $i$ and target-class column $j$, call the observed count $O_{ij}$ and expected count $E_{ij}$. If $N$ is the total number of observations, the general calculation is:

For the top-left cell, $E=40\times40/80=20$. These expected counts are derived from the table's totals. They are not model predictions and need not be integers.

Turn the discrepancies into one score

Subtract expected from observed, square the difference so positive and negative deviations cannot cancel, and divide by the expected count. Then add the contribution from every cell:

In the example, every cell differs from its expectation by 10, so each contributes $10^2/20=5$. All four contributions add to 20. Dividing by the expectation makes a discrepancy of 10 count more when the expected count is small.

What does the p-value add?

The statistic needs a reference distribution to judge how unusual it is under independence. For an ordinary $R\times C$ table with positive margins, the large-sample reference is a chi-square distribution with $(R-1)(C-1)$ degrees of freedom.

Here there is one degree of freedom: once the row and column totals are fixed, choosing the top-left count determines the other three. A statistic of 20 has an approximate p-value of 0.000007744 under this reference.

The p-value is the probability, under the independence hypothesis and the test's assumptions, of a statistic at least this large. It is not the probability that independence is true, a measure of causal influence, or the fraction of future predictions the feature will get right.

When screening many columns, some small p-values can occur by chance. A threshold such as 0.05 applied repeatedly is not a feature-selection guarantee. If formal testing is the goal, account for the collection of tests; if prediction is the goal, validate the selected model and the chosen feature count.

Two APIs with “chi-square” in the name can score different counts

scipy.stats.chi2_contingency takes a category-by-class table, including both presence and absence for our binary flag. sklearn.feature_selection.chi2 instead takes a nonnegative feature matrix and labels. For each input column, it sums feature values within each class and compares those sums with the allocation expected from class frequencies.

This feature scorer is useful with counts or indicators, such as word-count features. It is not a reason to turn arbitrary category names into numeric magnitudes. For an ordinary categorical independence test, count the categories explicitly.

Nonnegative input is required by scikit-learn. Standardized values often contain negatives and are unsuitable. Arbitrarily shifting or rescaling values also changes the score: multiplying the present-flag column by 10 gives a score of 100, without adding information. Choose a representation that makes the summed counts meaningful.

import numpy as np
from scipy.stats import chi2_contingency
from sklearn.feature_selection import chi2

# Rows: flag present, flag absent. Columns: target class 0, target class 1.
observed = np.array([[30, 10], [10, 30]])
test = chi2_contingency(observed, correction=False)
print(test.expected_freq)  # [[20, 20], [20, 20]]
print(test.statistic, test.dof, test.pvalue)  # 20, 1, about 0.000007744
print((observed - test.expected_freq) ** 2 / test.expected_freq)
# Every cell contributes 5.

# The same 80 observations, now as a feature column and target vector.
y = np.array([0] * 40 + [1] * 40)
present = np.array([1] * 30 + [0] * 10 + [1] * 10 + [0] * 30)
score, p = chi2(present[:, None], y)
print(score, p)  # [10], [0.0015654]; sums of the PRESENT feature only.
print(chi2(np.column_stack([present, 1 - present]), y)[0])  # [10, 10]
print(chi2((10 * present)[:, None], y)[0])  # [100]: feature scaling changes it.

for counts in [np.array([[22, 18], [18, 22]]),
               np.array([[220, 180], [180, 220]])]:
    result = chi2_contingency(counts, correction=False)
    print(result.statistic, result.pvalue)
# 0.8, 0.37109337 and 8.0, 0.004677735: same rates, different sample sizes.

Check the observations behind the counts

The usual reference distribution assumes independent observations and a suitable sampling model. Repeated measurements of the same person or duplicated records do not automatically count as independent evidence.

Sparse expected counts can make the large-sample approximation inaccurate. Inspect the expected table, not only the observed one. For small tables, consider a suitable exact or resampling method; SciPy documents alternatives. An all-zero row or column represents an unobserved category and must be handled before this calculation, since it creates zero expected counts.

For continuous features, binning creates a categorical version, but the bin choices affect the test. Learn those choices on training data. Fit supervised selection inside each training fold and retain its column mapping for validation and prediction. The selection introduction explains why choosing features before the split leaks labels.

Compute the full-table statistic from scratch

The implementation follows the two equations above: build margins, derive expectations, and sum the cell contributions. It returns the statistic, degrees of freedom and expected table. It does not compute a p-value or implement scikit-learn's separate feature-sum scorer.

#include <cstdint>
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <vector>

struct ChiSquare {
    double statistic;
    std::size_t degrees_of_freedom;
    std::vector<std::vector<double>> expected;
};

ChiSquare contingency_test(const std::vector<std::vector<std::uint64_t>>& observed) {
    if (observed.size() < 2 || observed[0].size() < 2)
        throw std::invalid_argument("need at least two rows and two columns");
    const auto R = observed.size(), C = observed[0].size();
    std::vector<long double> rows(R, 0), cols(C, 0);
    long double total = 0;
    for (std::size_t i = 0; i < R; ++i) {
        if (observed[i].size() != C) throw std::invalid_argument("ragged table");
        for (std::size_t j = 0; j < C; ++j) {
            long double n = observed[i][j];
            rows[i] += n; cols[j] += n; total += n;
        }
    }
    for (auto n : rows) if (n == 0) throw std::invalid_argument("empty row category");
    for (auto n : cols) if (n == 0) throw std::invalid_argument("empty column category");
    ChiSquare result{0, (R - 1) * (C - 1),
                     std::vector<std::vector<double>>(R, std::vector<double>(C))};
    long double score = 0;
    for (std::size_t i = 0; i < R; ++i) {
        for (std::size_t j = 0; j < C; ++j) {
            long double expected = rows[i] * cols[j] / total;
            long double difference = static_cast<long double>(observed[i][j]) - expected;
            result.expected[i][j] = static_cast<double>(expected);
            score += difference * difference / expected;
        }
    }
    result.statistic = static_cast<double>(score);
    return result;
}

int main() {
    auto result = contingency_test({{30,10},{10,30}});
    std::cout << std::setprecision(12) << result.statistic << ' '
              << result.degrees_of_freedom << '\n'; // 20 1
}

References and further reading

• SciPy: chi2_contingency — expected counts, degrees of freedom, continuity correction and resampling alternatives.
• Scikit-learn: chi2 — the nonnegative feature-scoring interface.
• Scikit-learn implementation: univariate selection — the class-wise feature sums and expected allocation used by chi2.
• Guyon and Elisseeff: feature selection — the distinction between individual relevance scores and evaluating selected subsets.