OLS, Normal Equations, and Projection Geometry
The previous page explained what linear regression estimates. This page explains how OLS finds the coefficients. The same answer appears in two ways: calculus gives the normal equations, while geometry shows that the fitted vector is the closest point in the column space of $X$.
Deriving the Normal Equations
OLS chooses the coefficient vector that minimizes the sum of squared residuals:
$$\text{RSS}(\beta) = (y-X\beta)^\top(y-X\beta) = y^\top y - 2\beta^\top X^\top y + \beta^\top X^\top X \beta.$$The minimum occurs where the gradient is zero:
$$\nabla_\beta\, \text{RSS}(\beta) = -2X^\top y + 2X^\top X\beta = 0 \quad\Longrightarrow\quad X^\top X\,\hat{\beta} = X^\top y.$$These are the normal equations. If the columns of $X$ are independent, $X^\top X$ is invertible and the unique solution is
$$\hat{\beta} = (X^\top X)^{-1}X^\top y.$$The Hessian is $2X^\top X$, which cannot have negative curvature. Therefore the objective is convex. Any solution of the normal equations is a global minimum. Full column rank makes that minimum unique.
Specializing to One Predictor
With one predictor, the fitted line is $a+bx$. At the OLS solution, the residuals sum to zero and have zero weighted sum against $x$. These conditions give
$$\hat{\beta}_1 = \frac{\sum_i (x_i-\bar{x})(y_i-\bar{y})}{\sum_i (x_i-\bar{x})^2} = \frac{S_{xy}}{S_{xx}}, \qquad \hat{\beta}_0 = \bar{y}-\hat{\beta}_1\bar{x}.$$For $x=(1,2,3,4,5,6)$ and $y=(2.2,3.8,5.1,5.6,7.4,8.3)$, $S_{xx}=17.5$ and $S_{xy}=20.9$. The fitted slope is $1.1943$, and the intercept is $1.22$.
Why Residuals Are Perpendicular to the Model Space
The normal equations can be written as $X^\top e=0$, where $e=y-X\hat\beta$. This means the residual vector has zero dot product with every column of $X$. With an intercept and one predictor, $\sum_i e_i=0$ and $\sum_i x_ie_i=0$. The fitted model has extracted every linear component available in those columns, so the remaining residual points in a perpendicular direction.
▶ Projecting a Vector onto the Model Space
The target is $y=(3,1,2)$. Each step finds the nearest vector in the span of $(1,1,0)$ and $(0,1,1)$, then shows the perpendicular residual.
Here $\hat\beta=(5/3,2/3)$, so $\hat y=(5/3,7/3,2/3)$. The residual is $(4/3,-4/3,4/3)$ and has zero dot product with both columns. Because the fitted vector and residual are perpendicular, $\|y\|^2=\|\hat y\|^2+\|e\|^2$.
The Hat Matrix Produces the Fitted Values
Substituting $\hat\beta=(X^\top X)^{-1}X^\top y$ into $\hat y = X\hat\beta$ gives
$$\hat y = X(X^\top X)^{-1}X^\top y = Hy, \qquad H := X(X^\top X)^{-1}X^\top,$$The matrix $H$ is called the hat matrix because it turns $y$ into $\hat y$. It projects $y$ onto the column space of $X$, meaning the set of all vectors the model can produce. The residual $e=(I-H)y$ is the part left outside that space. The fitted vector is the closest available model vector to $y$.
| Property of $H$ | Consequence |
|---|---|
| $H^\top = H$ | The projection is perpendicular rather than tilted. |
| $HH = H$ | Applying the same projection twice changes nothing. |
| $\operatorname{trace}(H)=p+1$ | Total leverage equals the number of fitted coefficients, including the intercept. |
| Eigenvalues are $0$ or $1$ | Model-space directions are kept; perpendicular directions are removed. |
Adding a predictor can only enlarge the model space, so training $R^2$ cannot decrease. This is also why training $R^2$ alone cannot justify a new predictor. Even a weak column may improve the in-sample fit slightly. Adjusted $R^2$ and held-out evaluation account for this problem more honestly.
Fitted and Residual Decomposition
Every response splits into $y=\hat y+e$: a fitted part inside the model space and a residual part perpendicular to it. This gives $\|y\|^2=\|\hat y\|^2+\|e\|^2$. With an intercept, the familiar centered decomposition of total, explained, and residual variation also holds.
Without an intercept. Residuals no longer have to sum to zero, and the usual centered $R^2$ decomposition can become misleading.
Near-Collinearity: Defined but Unstable
Highly correlated columns create a different problem. The inverse may exist, but small data changes can cause large coefficient changes. Direct inversion also loses numerical accuracy. The multicollinearity page explains the statistical problem, while the numerical-solvers page explains why QR and SVD are safer.
Final checks. Inputs must be finite. Unique coefficients need full column rank. A rank-deficient problem can use a pseudoinverse or an explicit constraint. Weighted least squares changes the condition to $X^\top We=0$. The C++ OLS implementation includes the corresponding dimension and rank checks.