Robust and Quantile Regression
Squared loss gives large residuals very large weight, so one extreme response can pull the fitted line strongly. Absolute, Huber, and pinball losses reduce this sensitivity or target a different part of the outcome distribution. No loss is always best; each answers a different question.
Squared, Absolute, and Huber Loss
$$\rho_\delta(r)=\begin{cases}\frac12r^2,&|r|\le\delta,\\ \delta(|r|-\frac12\delta),&|r|>\delta.\end{cases}\qquad \psi_\delta(r)=\rho_\delta'(r)=\begin{cases}r,&|r|\le\delta,\\ \delta\operatorname{sign}(r),&|r|>\delta.\end{cases}$$Huber loss behaves like squared loss for small residuals and like absolute loss for large ones. LAD minimizes absolute residuals and targets the conditional median. Huber gives a compromise between mean fitting and tail resistance. Neither protects against an extreme predictor row, because leverage comes from $X$, not only from the residual.
| Loss | Population target | Residual score | Important limitation |
|---|---|---|---|
| $r^2$ | Conditional mean | Unbounded linear | Tail errors dominate |
| $|r|$ | Conditional median | Bounded subgradient | Nondifferentiable; less efficient for Gaussian errors |
| Huber | Huber location | Clipped | Threshold and scale must be calibrated |
| Pinball | Conditional $\tau$ quantile | Asymmetric bounded subgradient | Separate fits can cross |
How One Extreme Point Changes the Fit
The score $\psi(r)=\rho'(r)$ describes how strongly one residual pulls on the coefficients. Adding a tiny amount of contamination at $(x_0,y_0)$ changes the fitted parameter at the rate
$$\psi(y_0-x_0^\top\beta)\,x_0=M\,\frac{d\beta(F_\varepsilon)}{d\varepsilon}\bigg|_{\varepsilon=0},\qquad M=E_F\!\left[\psi'(y-x^\top\beta)\,xx^\top\right],$$Thus $\operatorname{IF}(x_0,y_0)=M^{-1}\psi(r_0)x_0$. Squared loss uses $\psi(r)=r$, so residual influence grows without limit. LAD uses only the sign of $r$, Huber clips the score, and pinball loss uses bounded asymmetric scores. The factor $x_0$ remains, so a very unusual predictor row can still have large influence.
Pinball Loss and Conditional Quantiles
Pinball loss at level $\tau$ penalizes under-prediction and over-prediction differently. Its minimum is the conditional $\tau$ quantile. At $\tau=0.5$, it becomes half of absolute loss, so quantile regression becomes median regression.
Different quantile slopes show how different parts of the conditional distribution change with $X$. A 0.1 and 0.9 pair can form a prediction band, but its coverage must be checked on held-out data.
Prediction Intervals versus Confidence Intervals
A confidence interval for the mean at $x_0$ includes coefficient uncertainty. A prediction interval for one new outcome also includes fresh outcome noise:
$$\operatorname{Var}(y_0-x_0^\top\hat\beta)=\sigma^2\left(1+x_0^\top(X^\top X)^{-1}x_0\right).$$With $x_0^\top(X^\top X)^{-1}x_0=0.05$, $\hat\sigma^2=4$, and $t\approx2$, the mean-interval half-width is about $0.89$, while the prediction-interval half-width is about $4.10$. More data can shrink uncertainty about the mean, but it cannot remove new-observation noise.
Optimization, Scale, and Tuning
LAD and pinball losses are convex but have a sharp point, so common solvers include linear programming and subgradient methods. Huber regression often uses IRLS, giving full weight to small residuals and weight $\delta/|r_i|$ to large ones.
A common Huber choice is $\delta=1.345\hat\sigma$, where $\hat\sigma$ comes from a robust scale such as MAD. Validation can also choose $\delta$ using Huber loss. Extreme quantiles near $0$ or $1$ have fewer relevant tail observations and much higher variance.
How Pinball Loss Selects a Quantile
▶ Comparing Candidate 80th Percentiles
For data $(1,2,3,8,9)$ and $\tau=0.8$, each step computes the total pinball loss for a candidate $q$.
Counterexamples, Edge Cases, and Production
Changing the loss does not fix a missing curve or omitted variable. Robust response loss also does not control an extreme predictor row.
Separate quantile lines may cross, so a predicted 90th percentile can fall below the median. Extreme quantiles are noisy, ties create nonunique solutions, and censored data needs a censoring-aware model.
Useful monitoring includes matched loss, held-out quantile coverage, interval width, subgroup coverage, crossing rate, residual-scale drift, and the fraction of Huber residuals being clipped.
The C++ robust and quantile guide implements Huber IRLS, LAD, pinball loss, and interval checks.