LDA Classification: From Class Distributions to a Prediction

October 2026 · 15 min read

Classification follow-up to PCA and LDA.pdf · View a handwritten page

Suppose a new observation has measurements $(4,4)$. Should we call it class A or class B? Linear discriminant analysis (LDA) answers by comparing how well each class explains those measurements, while also accounting for how common each class is.

We will use the same small dataset throughout: find the two class centers, measure the spread around them, and calculate a prediction for $(4,4)$. The formulas use numbers introduced beside them. The Fisher LDA guide covers choosing a projection direction; you do not need that derivation to follow this classifier example.

Start with two measurements and a class label

Each row below is one training observation. The two numbers $x_1$ and $x_2$ are its measurements, and the label tells us whether it belongs to A or B. We use these eight rows to fit the model. The new point $(4,4)$ is held out of that calculation.

What does the LDA classifier assume?

Imagine the observations from each class as a cloud around an average point. LDA models each cloud as Gaussian, meaning a bell-shaped distribution. The centers may differ, but the model gives the clouds the same spread and orientation.

Here $c$ is a class label, A or B. The compact notation for the Gaussian assumption is:

Read this as "the measurements $x$, given class $c$, follow a Gaussian with that class's center and the shared covariance." A shared covariance can describe tilted ellipses as well as round clouds; it does not require independent measurements. The Gaussian assumption belongs to this probability model, not to the Fisher projection calculation.

Estimate the class means and priors

For class A, average each measurement over its four rows:

For B, the same calculation gives:

We estimate each prior from the fraction of training rows in that class. There are four A rows and four B rows out of eight, so both priors are $4/8=0.5$. Before looking at the new measurements, this fitted model gives the classes equal weight.

Estimate the shared covariance

Spread means variation around a class's own center. Use A's center $(2,2)$ for A points and B's center $(6,4)$ for B points. Subtracting the overall average would mix the gap between classes into their internal spread.

B has the same four offsets around its center $(6,4)$, so its squared differences also sum to $(2,2)$. Across both classes, each measurement therefore contributes $2+2=4$. Dividing by the eight training rows gives variance $4/8=0.5$ for each measurement.

Covariance records products between the two differences. Every offset here has a zero coordinate, so all those products are zero. The shared covariance matrix is a compact way to store the two variances and that zero covariance:

The diagonal 0.5 values are the variances. The off-diagonal zeros mean this fitted model has no within-class co-variation between the measurements. Other datasets can have nonzero off-diagonal entries.

Turn the fitted distributions into class probabilities

The values used in this section are A's center $(2,2)$, B's center $(6,4)$, variance 0.5 in each measurement, zero covariance between measurements, and priors 0.5 each. Both class clouds have the same spread.

The likelihood $p(x\mid c)$ is the Gaussian cloud's fit value, or density, at measurements $x$. It is not yet the probability that $x$ belongs to class $c$. Multiply it by that class's prior $\pi_c$ to get a relative class weight. Bayes' rule divides that weight by the total weight of all classes:

$c$ is the class we are asking about, such as B. The $k$ inside the sum is just the class label used while adding terms: first A, then B. It is not a measurement index or a count of observations.

Even when we ask about B, the denominator includes both A and B. With our equal priors, $\pi_A=\pi_B=0.5$:

The resulting probabilities add to 1. They are called posterior probabilities: the class probabilities after we have seen the measurements. With equal costs for mistakes, predict the class with the largest one. For $C$ classes labeled $1,\ldots,C$, the sum would contain one term for each of those labels.

How do we estimate and calculate the likelihood?

There are two steps. First, estimate the class centers and shared covariance from the training rows. We already obtained $\mu_A=(2,2)^\top$, $\mu_B=(6,4)^\top$, and $\Sigma=\begin{bmatrix}0.5&0\\0&0.5\end{bmatrix}$. Second, put a new point $x$ into the Gaussian probability density function (PDF) with those fitted values. We do not estimate $p(x\mid c)$ by counting how many training points exactly match $x$.

For two measurements, the Gaussian PDF is:

The unsubscripted $\pi$ in the PDF is the mathematical constant, about 3.14159. It is separate from the class prior $\pi_c$, which is 0.5 for A and B here.

Substitute our shared covariance. Its determinant and inverse are:

The same multiplier appears in front of both class densities:

Because the inverse covariance is diagonal with 2 in each position, the adjusted squared distance becomes:

$\mu_{c1}$ and $\mu_{c2}$ are the two entries of the class center. Multiplying that adjusted distance by $-1/2$ cancels the factors of 2. For these fitted parameters, the full PDF therefore simplifies to:

The general quadratic expression is squared Mahalanobis distance, which accounts for variances and correlations. Here it is twice ordinary squared Euclidean distance; with another covariance matrix, that simple relationship need not hold. The PDF returns a density value, not the probability of seeing that exact continuous-valued point or the probability of its class. Bayes' rule converts the two class densities and priors into class probabilities.

Classify the new point (4, 4)

Use $x=(4,4)$, A's center $(2,2)$, and B's center $(6,4)$. For each class, subtract its center and add the squares of the two differences:

Plug the point into class A's PDF

A's center is $(2,2)$, so the differences are $(2,2)$. The inverse covariance gives adjusted squared distance $2(2^2)+2(2^2)=16$. Use that value and the shared multiplier $1/\pi$ in the Gaussian PDF:

Plug the point into class B's PDF

B's center is $(6,4)$, so the differences are $(-2,0)$. Its adjusted squared distance is $2((-2)^2)+2(0^2)=8$. The covariance is shared, so the multiplier is still $1/\pi$:

B's density at this point is larger, meaning the measurements fit B's cloud better under this model. These density values are not class percentages. To obtain the class weights, multiply each density by its prior, 0.5:

Finally, divide B's weight by the sum of A's and B's weights. Using the unrounded PDF values gives:

For the exact expression, the common PDF multiplier $1/\pi$ and common prior 0.5 cancel from the ratio. This leaves $e^{-4}/(e^{-8}+e^{-4})=1/(1+e^{-4})$.

Predict B. The fitted model assigns about 98.2% probability to B and 1.8% to A. This is a probability for this point under the assumed model, not a measured 98.2% prediction accuracy on new data.

Why the class comparison becomes linear

The distance calculation already gave us the prediction. LDA can also compare classes using a weighted sum of the measurements, which explains the straight decision boundary.

For this dataset, the centers are still A = $(2,2)$ and B = $(6,4)$, with priors 0.5 each. The weights now include the full Gaussian density: $q_c=\pi_c p(x\mid c)$. Taking the log preserves their ordering. Since the shared PDF multiplier is $1/\pi$, the two log weights are $-\log\pi-d_A^2+\log0.5$ and $-\log\pi-d_B^2+\log0.5$. Expanding the squared distances gives:

The terms $-\log\pi-x_1^2-x_2^2$ are identical for both classes. Removing them cannot change which class wins. Call the remaining comparison score $\delta_c(x)$, where $c$ is A or B:

These are linear scores: multiply each measurement by its weight and add a constant. A larger score wins. A score is not a probability and can be greater than 1, because terms common to all classes have been removed.

B's score exceeds A's by 4. Exponentiating and normalizing gives $P(B\mid x)=1/[1+\exp(\delta_A-\delta_B)]=1/(1+e^{-4})$, the same 0.9820 probability as before.

Find the line where the classes tie

Use the linear scores just calculated: $\delta_A=4x_1+4x_2-8+\log0.5$ and $\delta_B=12x_1+8x_2-52+\log0.5$. Subtract A's score from B's:

If this difference is positive, B wins; if it is negative, A wins. The classes tie when it is zero:

That straight line is the decision boundary. For $(4,4)$, $2(4)+4=12>11$, so the point is on B's side, just as our probability calculation found.

The equal-prior boundary passes halfway between the class means along the classifier's score direction. In this example the shared covariance is the same in every direction, so the ordinary perpendicular bisector also gives that line. With a tilted or uneven shared covariance, the boundary need not be the Euclidean perpendicular bisector.

If we keep the centers and covariance fixed but give B a larger prior, the model needs less measurement evidence to choose B, expanding B's region. With more than two classes, calculate one score per class and choose the largest.

How this connects to the LDA projection

The Fisher projection turns measurements into a score along an axis. This classifier also supplies a rule for choosing a class and a probability. You can call its prediction method directly; a separate projection step is not required.

For the eight rows here, total within-class scatter is $S_W=\begin{bmatrix}4&0\\0&4\end{bmatrix}$ and the center difference is $g=(6-2,4-2)^\top=(4,2)^\top$. Fisher's unnormalized weights are $u=(1,0.5)^\top$. The classifier's weights $(8,4)^\top$ point along the same line, but have a different scale.

The projection guide works through a different ten-point dataset. Its numerical weights differ because the data differ, not because these two uses of LDA contradict each other.

Check the prediction in Python

The code fits the same eight training rows, estimates centers $(2,2)$ and $(6,4)$ and shared covariance $\begin{bmatrix}0.5&0\\0&0.5\end{bmatrix}$, then predicts for $(4,4)$. The lsqr option uses the same maximum-likelihood covariance convention as this guide.

import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

X = np.array([
    [1, 2], [3, 2], [2, 1], [2, 3],
    [5, 4], [7, 4], [6, 3], [6, 5],
], dtype=float)
y = np.array(["A"] * 4 + ["B"] * 4)
model = LinearDiscriminantAnalysis(solver="lsqr").fit(X, y)
print(model.covariance_)                 # [[0.5, 0.0], [0.0, 0.5]]
print(model.predict([[4, 4]]))           # ['B']
print(model.classes_)                   # ['A', 'B']
print(model.predict_proba([[4, 4]]))     # [[0.01798621, 0.98201379]]

The probability columns follow model.classes_: A first, B second. The returned row therefore means about 1.8% for A and 98.2% for B. Fit the model and any preprocessing on training data only, then evaluate on separate held-out data.

With many measurements and few training points, covariance shrinkage can make the estimated spread more stable. If the classes need different covariance shapes, quadratic discriminant analysis (QDA) allows separate matrices and can produce curved boundaries.

From the handwritten notes

This guide expands the relevant material in PCA and LDA.pdf. The complete original contains both topics; the blog treats PCA, LDA projection, and LDA classification separately.

View original page 5

Further reading:

• scikit-learn: LDA projections and classification
• scikit-learn: LinearDiscriminantAnalysis API