Linear Regression: Essential Interview Derivations

The squared-error objective for $X\in\mathbb R^{n\times p}$ is

$$L(\beta)=\|y-X\beta\|_2^2=y^\top y-2\beta^\top X^\top y+\beta^\top X^\top X\beta.$$

The gradient is $-2X^\top y+2X^\top X\beta$. Setting it to zero gives $X^\top X\hat\beta=X^\top y$. The Hessian is positive semidefinite, so this is a global minimum. Full column rank gives the unique solution $(X^\top X)^{-1}X^\top y$.

Substitution gives $\hat y=Hy$ with $H=X(X^\top X)^{-1}X^\top$. Since $H^\top=H$ and $H^2=H$, it is an orthogonal projection. The normal equations give $X^\top e=0$, so the residual is perpendicular to every model column. Therefore

$$\|y\|^2=\|\hat y\|^2+\|e\|^2.$$

With an intercept, residuals sum to zero and Total Sum of Squares equals Regression Sum of Squares plus Error Sum of Squares: $\text{SST}=\text{SSR}+\text{SSE}$. Here SSR means explained regression variation, while SSE means residual variation. Without an intercept, the usual centered $R^2$ decomposition may fail.

Under $y=X\beta+\varepsilon$, fixed $X$, $E[\varepsilon\mid X]=0$, and full rank,

$$\hat\beta=\beta+(X^\top X)^{-1}X^\top\varepsilon,\qquad E[\hat\beta\mid X]=\beta.$$

For general conditional error covariance $\Omega$,

$$\operatorname{Var}(\hat\beta\mid X)=(X^\top X)^{-1}X^\top\Omega X(X^\top X)^{-1}.$$

Under constant independent error variance, this reduces to $\sigma^2(X^\top X)^{-1}$. Normality is not needed for unbiasedness; it is needed for exact small-sample $t$ and $F$ distributions.

Any linear estimator has the form $\tilde\beta=Cy$. Unbiasedness requires $CX=I$. Decomposing $C$ into the OLS matrix plus another matrix $D$ makes the unbiasedness condition force $DX=0$. Under constant independent error variance,

$$\operatorname{Var}(\tilde\beta\mid X)=\sigma^2CC^\top=\sigma^2\left[(X^\top X)^{-1}X^\top X(X^\top X)^{-1}+(X^\top X)^{-1}X^\top D^\top+DX(X^\top X)^{-1}+DD^\top\right].$$

The cross terms vanish, leaving the OLS variance plus $\sigma^2DD^\top$. This extra term cannot reduce variance. Therefore OLS has the smallest variance among linear unbiased estimators.

For a new outcome $Y=f(x)+\varepsilon$, expected prediction error splits into

$$E_{D,\varepsilon}\!\left[(Y-\hat f(x))^2\right]=\sigma^2+\left(E_D[\hat f(x)]-f(x)\right)^2+\operatorname{Var}_D(\hat f(x)).$$

The three parts are irreducible noise, squared bias, and model variance across training samples. One fitted model cannot reveal its own bias and variance without resampling or a data-generating model.

For $L_\lambda(\beta)=\|y-X\beta\|^2+\lambda\|\beta\|^2$,

$$\nabla L_\lambda=-2X^\top y+2X^\top X\beta+2\lambda\beta=0,$$ $$\hat\beta_\lambda=(X^\top X+\lambda I)^{-1}X^\top y.$$

The conditional bias is $-\lambda(X^\top X+\lambda I)^{-1}\beta$. The added diagonal stabilizes the system. In standard practice, the intercept is excluded from the penalty.

The predictors are partitioned into controls $X_1$ and target predictors $X_2$. Eliminating the control coefficients from the normal equations gives

$$X_2^\top M_1X_2\,\hat\beta_2=X_2^\top M_1y,\qquad M_1=I-X_1(X_1^\top X_1)^{-1}X_1^\top,$$

This is OLS of the residualized outcome $M_1y$ on the residualized predictor $M_1X_2$. The slope equals the full-model coefficient. If $M_1X_2=0$, no unique variation remains and the coefficient is unidentified.

For one standardized coordinate, $z_j$ denotes its correlation with the residual left by the other predictors. The Lasso stationarity condition is

$$0\in-z_j+\beta_j+\lambda\,\partial|\beta_j|,\qquad \partial|\beta_j|=\begin{cases}\{\operatorname{sign}(\beta_j)\},&\beta_j\ne0,\\ [-1,1],&\beta_j=0.\end{cases}$$

If $|z_j|\le\lambda$, zero is optimal. Otherwise $z_j$ is moved toward zero by $\lambda$. The three cases combine into $\hat\beta_j=\operatorname{sign}(z_j)\max(|z_j|-\lambda,0)$.

Removing one row changes $X^\top X$ by a rank-one update. Sherman-Morrison gives the exact leave-one-out residual without refitting:

$$e_{(i)}=y_i-\hat y_{(i)}=\frac{e_i}{1-h_{ii}}.$$

PRESS sums the squared values $e_i/(1-h_{ii})$ and equals exact OLS leave-one-out error. Large leverage makes deletion residuals unstable. Coefficient influence and Cook's distance are also large only when a row combines meaningful leverage with a meaningful residual.