Residual Diagnostics, Influence, and Evaluation Metrics
One score and one coefficient table are not enough to trust a regression model. Evaluation measures prediction on suitable held-out data. Diagnostics look for wrong model shape, changing error variance, dependence, and observations that strongly control the fit.
Residual Structure
Residuals are $e_i=y_i-\hat y_i$. In a residual-versus-fitted plot, curvature suggests a missing nonlinear term, a fan suggests changing variance, bands suggest groups or rounding, and long time-ordered runs suggest dependence. A Q-Q plot checks tail shape for inference; it does not test whether the mean function is linear.
Raw residual spread also depends on leverage. Studentized residuals divide by an estimated standard deviation so rows become more comparable.
Leverage Is Not Influence
Leverage $h_{ii}=x_i^\top(X^\top X)^{-1}x_i$ depends only on the predictor values. A row far from the center of the design has high leverage. High leverage alone does not mean high influence; a row also needs a meaningful residual. Cook's distance combines both:
$$D_i=\frac{e_i^2}{p\,\hat\sigma^2}\frac{h_{ii}}{(1-h_{ii})^2}.$$Thresholds such as $h_{ii}>2p/n$ or $D_i>4/n$ only flag rows for investigation. A valid rare case may be the most important row in the dataset.
Metrics Encode Different Costs
In the formulas below, SST means Total Sum of Squares, the total variation around the mean. SSE means Error Sum of Squares, the variation left in the residuals. The explained part is the Regression Sum of Squares (SSR), with $\text{SST}=\text{SSR}+\text{SSE}$ for OLS with an intercept.
| Metric | Definition | Strength | Failure mode |
|---|---|---|---|
| RMSE | $\sqrt{n^{-1}\sum e_i^2}$ | Same units, emphasizes large errors | Dominated by heavy tails |
| MAE | $n^{-1}\sum|e_i|$ | Robust summary, same units | Hides tail severity |
| $R^2$ | $1-\text{SSE}/\text{SST}$ | Relative in-sample variance reduction | Can rise mechanically with features |
| Adjusted $R^2$ | $1-\frac{\text{SSE}/(n-p)}{\text{SST}/(n-1)}$ | Penalizes parameter count | Not a substitute for validation |
| PRESS | $\sum(e_i/(1-h_{ii}))^2$ | Exact OLS leave-one-out residuals | Unstable near leverage one |
Test $R^2$ can be negative, meaning the model is worse than its baseline. MAPE fails at zero targets and overweights small targets. RMSLE requires nonnegative values. The metric should match the real cost of errors, and average metrics should be paired with subgroup and tail results.
Train versus Test $R^2$, Worked
Adding predictors cannot lower training $R^2$, because OLS can always set a new coefficient to zero. Test $R^2$ has no such guarantee. The table adds pure-noise predictors to a model with only twelve training rows:
| Parameters (incl. intercept) | Train $R^2$ | Test $R^2$ |
|---|---|---|
| 2 | 0.948 | 0.866 |
| 4 | 0.952 | 0.863 |
| 7 | 0.977 | 0.519 |
| 10 | 0.999 | 0.550 |
| 12 | 1.000 | −2.928 |
Training $R^2$ reaches $1$ as the model becomes large enough to interpolate the twelve rows. Test $R^2$ falls below zero because the noise columns do not generalize. The growing train-test gap is the important sign of overfitting.
Leverage and Cook's Distance, Row by Row
▶ Which Observation Moves the Fit?
The animation fits one line to five fixed points, then shows each row's residual, leverage, and Cook's distance.
Validation Design and Statistical Tests
The split must match deployment. Time data needs chronological validation, repeated entities need group splits, and spatial data may need spatial blocks. Every learned preprocessing step belongs inside the resampling loop.
Breusch-Pagan and White tests look for changing variance. Durbin-Watson and autocorrelation checks look for time dependence. Tests should be read with plots and effect sizes. Robust standard errors fix uncertainty under heteroscedasticity, not bias from a wrong mean function.
Diagnostics under Robust and Regularized Fits
OLS leverage, Cook's distance, and PRESS do not transfer unchanged to Huber, LAD, Ridge, or Lasso. Robust fits use row weights, while penalized fits are biased by design. OLS-style standard errors and $p$-values beside Lasso coefficients are not valid without a specialized correction or a bootstrap of the complete selection process.
Edge Cases and Production Implications
Metric code needs explicit behavior for constant targets, missing values, zero denominators, weights, and non-finite predictions. Weighted fits may have an effective sample size smaller than the row count.
Production monitoring should track residual distributions, error by subgroup and range, missingness, feature drift, distance from training support, and performance against a simple baseline.
The C++ diagnostics guide implements metrics, leverage, Cook's distance, PRESS, inference, and a Breusch-Pagan style check.