Linear Regression: Overview and Roadmap
Linear regression estimates a continuous number, such as price, demand, or temperature, from one or more predictors. It gives each predictor a coefficient, adds an intercept, and leaves a row-specific error for variation the model does not explain:
$$y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\varepsilon_i.$$Writing the same equation for all $n$ rows gives the compact matrix form
$$y = X\beta + \varepsilon,$$where $X\beta$ is the part explained by the model and $\varepsilon$ is the unexplained part. The first column of $X$ contains ones, so its coefficient becomes the intercept $\beta_0$. The word linear means linear in the coefficients. The columns of $X$ may still contain squared terms, interactions, or other prepared features.
| Symbol | Meaning |
|---|---|
| $n$ | number of data rows |
| $p$ | number of predictors, not counting the intercept |
| $X$ | all model inputs, arranged as $n$ rows and $p+1$ columns; the first column contains ones |
| $\beta$ | unknown true coefficients, including the intercept |
| $\hat{\beta}$ | coefficients estimated from the available sample |
| $\varepsilon = y-X\beta$ | unobservable differences from the true population regression function |
| $\hat{y} = X\hat{\beta}$ | predictions made by the fitted model |
| $e = y-X\hat{\beta}$ | observable differences between the data and the fitted predictions |
From the Question to a Checked Model
A useful regression workflow has four stages: decide what to estimate, fit the model, test its predictions, and inspect why it may fail.
- Target and inputs. The target may be a future average, a descriptive relationship, or a causal effect supported by a suitable study design. Each row of $X$ contains the intercept, predictors, and any justified transformations. These choices determine what the model means and what shapes it can represent.
- Coefficient fitting. Ordinary least squares chooses the coefficient vector that makes the squared sample errors as small as possible: $$\hat\beta=\arg\min_b\sum_{i=1}^{n}(y_i-x_i^\top b)^2 =\arg\min_b (y-Xb)^\top(y-Xb).$$ Differentiation gives $X^\top X\hat\beta=X^\top y$. Independent columns give a unique solution. QR or SVD is usually safer than explicitly forming an inverse.
- Prediction checks. Predictions are $\hat y=X\hat\beta$, and residuals are $e=y-\hat y$. Training error measures fit to known rows. Held-out RMSE, MAE, and $R^2$ measure performance on unseen rows. A large train-test gap suggests overfitting, leakage, instability, or distribution shift.
- Diagnosis and revision. Residual plots reveal curvature or changing error spread. Leverage and Cook's distance flag influential rows. Condition numbers and VIF expose unstable coefficients. Standard errors and confidence intervals are reported only when their assumptions are reasonable. Failed checks may require different features, uncertainty estimates, or estimators.
A failed check sends the workflow back to the target, feature design, or fitting method.
What Must Be Checked
- Independent columns: duplicate or perfectly dependent columns prevent unique coefficients.
- No leftover average error pattern: $E[\varepsilon\mid X]=0$ is needed for the usual coefficient interpretation. Robust standard errors cannot repair an omitted confounder.
- Error spread and dependence: the textbook standard-error formula assumes constant variance and independent errors. Other covariance estimators are needed when those assumptions fail.
- Observed range: interpolation stays inside the data range. Extrapolation trusts the fitted shape where no observations support it, and a high training $R^2$ does not make that safe.
Loss Functions Choose the Optimal Estimator
For a fixed input $x$, a loss function decides which summary of $Y\mid X=x$ is optimal. The table shows the population minimizer
$$a^*(x)=\arg\min_a E[L(Y,a)\mid X=x].$$| Loss | Formula | Optimal estimator | Estimator formula |
|---|---|---|---|
| Squared error | $L(y,a)=(y-a)^2$ | Conditional mean | $a^*(x)=E[Y\mid X=x]$ |
| Absolute error | $L(y,a)=|y-a|$ | Conditional median | $a^*(x)=F^{-1}_{Y\mid X}(0.5\mid x)$ |
| Pinball loss | $\rho_\tau(y-a)=(y-a)\big(\tau-\mathbf1\{y<a\}\big)$ | Conditional $\tau$ quantile | $a^*(x)=F^{-1}_{Y\mid X}(\tau\mid x)$ |
| Asymmetric squared loss | $|\tau-\mathbf1\{y<a\}|(y-a)^2$ | Conditional $\tau$ expectile | $\tau E[(Y-a)_+\mid x]=(1-\tau)E[(a-Y)_+\mid x]$ |
| Huber loss | $\frac12r^2$ for $|r|\le\delta$; $\delta(|r|-\frac12\delta)$ otherwise | Huber robust location | $E[\operatorname{clip}(Y-a,-\delta,\delta)\mid x]=0$ |
| 0-1 classification loss | $L(y,a)=\mathbf1\{y\ne a\}$ | Most probable class | $a^*(x)=\arg\max_k P(Y=k\mid X=x)$ |
| Binary log-loss | $-y\log p-(1-y)\log(1-p)$ | Conditional probability | $p^*(x)=P(Y=1\mid X=x)$ |
| Brier loss | $L(y,p)=(y-p)^2$ | Conditional probability | $p^*(x)=P(Y=1\mid X=x)$ |
With no predictors, the word “conditional” disappears: squared error gives the mean, absolute error gives the median, and pinball loss gives a quantile.
▶ One Sample, Different Optimal Summaries
For values $(1,2,3,8,9)$, each step changes the loss and marks its optimal constant prediction.
A Small Worked Fit
The simplest baseline predicts the sample mean $\bar y$ for every row. The Total Sum of Squares (SST) measures how much the observations vary around that baseline:
$$\text{SST} = \sum_i (y_i - \bar{y})^2.$$After fitting a line, SST splits into the Regression Sum of Squares (SSR), also called explained variation, and the Error Sum of Squares (SSE), also called residual variation:
$$\text{SST} = \text{SSR} + \text{SSE}, \qquad \text{SSR}=\sum_i(\hat{y}_i-\bar{y})^2, \qquad \text{SSE}=\sum_i(y_i-\hat{y}_i)^2,$$Therefore $R^2=\text{SSR}/\text{SST}$ is the fraction explained by the line. With one predictor, $R^2$ also equals the squared sample correlation $r^2$ between $x$ and $y$.
Notation note. This series uses SSR for Regression Sum of Squares. Some books use SSR to mean “sum of squared residuals”; in this series the residual quantity is called SSE or RSS.
▶ Raw Data to Estimand, OLS, Decomposition, and Inference
All displayed values are calculated from $x=(1,\ldots,6)$ and $y=(2.2,3.8,5.1,5.6,7.4,8.3)$. Each step advances one analysis stage and keeps the fitted and residual geometry visible.
The data behind the animation is the six points $x=(1,\ldots,6)$ and $y=(2.2,3.8,5.1,5.6,7.4,8.3)$, which give $\text{SST}=25.34$, $\text{SSE}=0.379$, $\text{SSR}=24.96$, and $R^2=0.985$. The sample correlation on the same data is $r=0.9925$, and $r^2=0.985$ to three decimal places, matching $R^2$ exactly. This equivalence is specific to simple regression with one predictor; with multiple predictors, $R^2$ still measures explained variance but no longer equals the square of a single pairwise correlation.
Worked check. Here $\bar x=3.5$, $\bar y=5.4$, $S_{xx}=17.5$, and $S_{xy}=20.9$, so $\hat\beta_1=1.1943$ and $\hat\beta_0=1.2200$. The residual degrees of freedom are $6-2=4$, $\hat\sigma=\sqrt{0.3794/4}=0.3080$, and $\operatorname{se}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}=0.0736$. A zero-variance predictor, a duplicated column, or $n\le p+1$ blocks the usual unique fit or residual variance estimate.
Roadmap for This Track
The twelve published chapters move from the basic meaning of the model to fitting, inference, numerical methods, diagnostics, and production use.
- What Linear Regression EstimatesReady · the conditional mean function, population parameters versus sample estimates, and why the model targets the best linear approximation to $E[Y\mid X]$.
- OLS, Normal Equations, and Projection GeometryReady · deriving $\hat{\beta}$, the normal equations, and the projection-onto-column-space geometry behind least squares.
- Gauss-Markov, Standard Errors, Tests, and IntervalsReady · the Gauss-Markov theorem, standard errors, hypothesis tests, and confidence intervals.
- Multicollinearity, VIF, and the FWL TheoremReady · diagnosing multicollinearity with the variance inflation factor and the Frisch-Waugh-Lovell theorem.
- Model Specification, Transformations, and InteractionsReady · choosing functional form: transformations, interaction terms, categorical encodings, and specification errors.
- Solving Least Squares Reliably: Cholesky, QR, and SVDReady · comparing solvers for least squares and why numerical stability matters as much as speed.
- Gradient Descent for Linear RegressionReady · batch, stochastic, and mini-batch gradient descent, and when to prefer iterative solvers over the closed form.
- Ridge, Lasso, and Elastic NetReady · penalized least squares, bias-variance tradeoffs, and a Bayesian reading of each penalty.
- Robust and Quantile RegressionReady · least absolute deviations, Huber regression, and quantile regression for non-mean targets and heavy-tailed errors.
- Residual Diagnostics, Influence, and Evaluation MetricsReady · residual plots, leverage, Cook's distance, and evaluation metrics.
- Linear Regression Production Failure ModesReady · data drift, extrapolation, leakage, and silent assumption violations in deployed models.
- Linear Regression: Essential Interview DerivationsReady · the derivations most likely to appear in interviews, worked from first principles.
The matching implementation series, built from scratch in standard-library C++ with an explicit fit/predict/evaluate lifecycle, starts at From Scratch · C++ · Linear Regression, including a compilable ordinary-least-squares solver and comparisons against QR and SVD based alternatives.