Model Specification, Transformations, and Interactions
In linear regression, linear means linear in the coefficients, not necessarily in the original predictors. Columns such as $x^2$, $\log x$, or $x_1x_2$ can still be placed in $X$ and fitted with ordinary least squares. Choosing those columns is called model specification.
Linear in the Parameters, Not in the Predictors
The quadratic model $y=\beta_0+\beta_1x+\beta_2x^2+\varepsilon$ is still linear in $\beta_0,\beta_1,\beta_2$. Its design matrix is $[1,x,x^2]$, so the normal equations still apply. A model such as $y=\beta_0e^{\beta_1x}+\varepsilon$ is nonlinear in $\beta_1$ and needs a nonlinear optimizer.
Transformations, Logs, Polynomials, and Basis Terms
| Specification | Interpretation and caveat |
|---|---|
| $Y=\beta_0+\beta_1\log X+\varepsilon$ | Useful when equal percentage changes in positive $X$ matter more than equal unit changes. |
| $\log Y=\beta_0+\beta_1X+\varepsilon$ | Turns coefficient changes into multiplicative changes in positive $Y$; converting back to a mean needs care. |
| $1,x,x^2,\ldots,x^k$ | Captures curvature inside the data range but may behave wildly outside it. |
| Basis functions $B_1(x),\ldots,B_m(x)$ | Splines and related bases allow local curvature while keeping the model linear in coefficients. |
Centering, $x_c=x-\bar x$, makes lower-order terms easier to interpret and can reduce mechanical correlation with $x_c^2$ or interactions. It does not remove genuine overlap between different predictors.
Omitted-Variable Bias
If the true model contains $x_1$ and $x_2$ but the fitted model drops $x_2$, the slope on $x_1$ becomes
$$E[\hat\beta_1^{\text{short}}] = \beta_1 + \beta_2\,\delta_1.$$The bias disappears only when $x_2$ has no effect ($\beta_2=0$) or is unrelated to $x_1$ in the auxiliary regression ($\delta_1=0$). Otherwise the fitted slope on $x_1$ absorbs part of the omitted predictor's effect.
Missing Curvature: an Omitted Quadratic Term
The true relationship $y=2+4.5x-0.5x^2+\varepsilon$ bends downward. Fitting only a straight line to seven fixed rows gives RSS $17.72$ and $R^2=0.34$. Adding $x^2$ reduces RSS to $0.42$ and raises $R^2$ to $0.98$. The straight-line slope is distorted because the missing $x^2$ term is related to $x$.
Interaction Terms and Marginal Effects
An interaction allows the slope of one predictor to change with another predictor. For $y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_1x_2+\varepsilon$, the slope with respect to $x_1$ is
$$\frac{\partial y}{\partial x_1} = \beta_1 + \beta_3x_2,$$Therefore $\beta_1$ is the slope only when $x_2=0$. At other values of $x_2$, the interaction changes the slope.
▶ How an Interaction Changes a Slope
For $\hat y=2+1.5x+0.8z+0.6xz$, each step changes $z$ and shows the resulting line. The slope on $x$ is $1.5+0.6z$.
In a ten-point example, a model without the interaction forces both groups to share one slope and has RSS $12.46$. Adding $x\cdot d$ allows separate slopes and reduces RSS to $0.46$.
Hierarchy principle. A model containing $x_1x_2$ normally keeps both $x_1$ and $x_2$. Removing a main effect imposes an extra constraint that is often hard to justify.
Categorical Encoding and the Dummy-Variable Trap
A categorical predictor with $k$ levels usually becomes $k-1$ indicator columns plus an intercept. The omitted level is the reference category. Including all $k$ indicators together with an intercept creates an exact dependency because the indicators sum to the ones column. This is the dummy-variable trap.
Changing the reference category changes coefficient labels but not fitted values. Unseen categories at prediction time need an explicit policy defined during training.
Missingness and Extrapolation
Dropping rows with missing values is valid only under assumptions about why values are missing. Simple mean imputation reduces apparent variation and can distort relationships. Any imputation rule must be learned from training data only. More difficult missingness mechanisms may need a separate model or sensitivity analysis.
Transformations can make extrapolation more dangerous. Polynomial powers grow quickly, interactions may be evaluated at unseen combinations, and logarithms behave sharply near zero. Predictions outside the observed joint feature region should be flagged rather than treated as equally supported.
| Symptom | Likely specification issue |
|---|---|
| Residuals form a smooth curve | The mean function may be missing a transformation or basis term. |
| A coefficient changes sign when controls are added | An omitted variable or interaction may be changing the conditional relationship. |
| $X^\top X$ is singular | Some columns are exact combinations of others, often because of dummy encoding. |
| Residual spread changes with fitted value | The mean may be correct while the constant-variance assumption is wrong. |
Testing the Functional Form
A residual-versus-fitted plot is the first check. Random scatter around zero supports the chosen mean shape. Curvature, grouping, or a fan shape points to a problem. Ramsey's RESET test adds powers of the fitted values and tests them together; rejection suggests that nonlinear structure remains. The C++ diagnostics guide implements the supporting residual and metric calculations.
Final checks. A transformation does not remove confounding. A log transform does not guarantee normal residuals. RESET can detect missing shape but does not identify the correct replacement. Residual plots, cross-validation, influence checks, and domain knowledge must be considered together.
The next page compares the numerical methods used to solve least-squares problems reliably.