Multicollinearity, VIF, and the FWL Theorem
Multicollinearity means that two or more predictor columns carry nearly the same information. It does not automatically bias OLS or ruin predictions. Its main effect is that the model cannot reliably decide how to divide credit among the correlated predictors, so individual coefficients become unstable and imprecise.
What Multicollinearity Changes, and What It Leaves Alone
The covariance formula explains the problem:
$$\operatorname{Var}(\hat\beta\mid X)=\sigma^2(X^\top X)^{-1}.$$Highly correlated columns make $X^\top X$ close to singular, which makes some coefficient variances large. Predictions can remain stable because opposite coefficient changes may cancel inside $X\hat\beta$. The model may predict well while the individual coefficients change sharply across samples.
Exact Rank Failure Versus Near Collinearity
Exact collinearity means one column is an exact combination of others. Coefficients are not unique and $X^\top X$ has no inverse. Near collinearity keeps a unique solution, but the condition number $\kappa_2(X)=\sigma_{\max}/\sigma_{\min}$ becomes large. Small data changes or roundoff can then cause large coefficient changes.
Predictions are most stable for feature combinations similar to the training data. A new row that separates predictors that were previously tied together can still have high prediction uncertainty.
The Variance Inflation Factor
Redundancy of predictor $X_j$ is measured by regressing it on all the other predictors. The resulting $R^2$ is denoted $R_j^2$. Then
$$\text{Var}(\hat\beta_j) = \frac{\sigma^2}{(n-1)\,\widehat{\text{Var}}(X_j)}\cdot\frac{1}{1-R_j^2},$$and the second factor is the variance inflation factor,
$$\text{VIF}_j = \frac{1}{1-R_j^2}.$$A VIF of $1$ means no inflation from the other predictors. As $R_j^2$ approaches $1$, almost no unique variation remains in $X_j$, so its coefficient variance grows rapidly. VIF measures coefficient instability, not prediction quality.
▶ Removing the Other Predictor Before Measuring a Slope
The animation removes the part of $x$ and $y$ explained by the control $z$. The slope between the two leftover vectors equals the coefficient on $x$ in the full regression.
The full regression is $y=5+2x+3z+e$. After removing the intercept and $z$, the leftover vectors are $\tilde x=(-1,1,-1,1)$ and $\tilde y=(-1,1,-3,3)$. Their slope is $8/4=2$, exactly matching the coefficient on $x$ in the full model.
The Frisch-Waugh-Lovell Theorem
The FWL theorem explains what “holding the other predictors fixed” means mathematically. For $y=X_1\beta_1+X_2\beta_2+\varepsilon$, the coefficient on $X_2$ can be found in three steps:
- Regress $y$ on the controls $X_1$ and keep the residual $\tilde y$.
- Regress $X_2$ on the same controls and keep the residual $\tilde X_2$.
- Regress $\tilde y$ on $\tilde X_2$. The slope equals the coefficient on $X_2$ in the full model.
The residual-maker matrix for $X_1$ is $M_1=I-X_1(X_1^\top X_1)^{-1}X_1^\top$. The second block of the full normal equations is $X_2^\top(y-X_1\hat\beta_1-X_2\hat\beta_2)=0$. Solving the first block for $\hat\beta_1$ and substituting yields
$$X_2^\top M_1X_2\hat\beta_2=X_2^\top M_1y,\qquad \hat\beta_2=(X_2^\top M_1X_2)^{-1}X_2^\top M_1y.$$The matrix $M_1$ removes the part explained by $X_1$. If $M_1X_2=0$, then $X_2$ has no unique variation beyond the controls, so its coefficient cannot be identified.
Detection and Remedies
| Situation | What to do |
|---|---|
| Moderately high VIF but stable predictions | The model may remain useful; individual coefficient intervals need cautious interpretation. |
| Two predictors measure nearly the same concept | Combining them or choosing one may avoid an arbitrary coefficient split. |
| Individual coefficients must remain stable | Ridge regression can trade some bias for lower variance. |
| Exact singularity | Remove a redundant column or impose a clear constraint. A common example is including every category indicator together with an intercept. |
Centering can reduce correlation created mechanically by polynomial or interaction terms. It does not remove genuine overlap between different predictors that measure the same underlying information.
Practical order. Rank and singular values detect exact problems. Condition numbers and VIF measure near-collinearity. Resampling shows whether coefficients are stable. Prediction checks show whether the instability matters for the intended use. Dropping a confounder only to reduce VIF can create omitted-variable bias, so a smaller VIF is not always a better model.
The C++ QR and SVD guide implements rank and condition-number checks without relying on $X^\top X$.