Ridge, Lasso, and Elastic Net
Regularization deliberately shrinks coefficients. A small loss in training fit can produce more stable coefficients, better test performance, or a simpler model. With centered $y$ and standardized predictors, Elastic Net minimizes
$$\min_\beta\left\{\frac{1}{2n}\|y-X\beta\|_2^2+\lambda\left[\alpha\|\beta\|_1+\frac{1-\alpha}{2}\|\beta\|_2^2\right]\right\}.$$Ridge uses $\alpha=0$, Lasso uses $\alpha=1$, and Elastic Net uses a value between them. The intercept is normally not penalized. Standardization matters because coefficient size depends on feature units.
Why Ridge Shrinks and Lasso Creates Zeros
The penalized problem can also be viewed as minimizing squared error while keeping coefficients inside a constrained region. Ridge uses a smooth $L_2$ sphere. Lasso uses an $L_1$ diamond with corners on the coordinate axes. The best loss contour often touches the Lasso diamond at a corner, where at least one coefficient is exactly zero. A smooth Ridge boundary usually produces small but nonzero coefficients.
| Constraint region | Boundary shape | Generic tangency | Consequence |
|---|---|---|---|
| $\|\beta\|_2^2\le t$ | Smooth sphere | Usually away from an axis | Shrinkage without exact zeros |
| $\|\beta\|_1\le t$ | Diamond with sharp corners | Often at a corner | Exact zeros |
| Elastic Net constraint | Rounded diamond | Corners remain, with added curvature | Sparsity plus more stable groups |
Ridge as Spectral Shrinkage
For the convention $\|y-X\beta\|^2+\lambda\|\beta\|^2$, differentiating gives
$$\hat\beta_{\text{ridge}}=(X^\top X+\lambda I)^{-1}X^\top y.$$In the SVD basis, Ridge multiplies each OLS direction by $\sigma_j^2/(\sigma_j^2+\lambda)$. Directions with small singular values shrink the most. These are exactly the directions that are weakly identified under multicollinearity. Ridge also gives a unique solution when $\lambda>0$ even if $X$ is rank deficient.
With Gaussian errors, Ridge is also the MAP estimate under a zero-mean Gaussian prior on standardized coefficients. This interpretation does not remove Ridge bias.
Bias-variance example. In one orthogonal direction, $\hat\beta_\lambda=\hat\beta_{\text{OLS}}/(1+\lambda)$. Increasing $\lambda$ adds bias but lowers variance. For true $\beta=0.4$ and noise variance $1$, the MSE is minimized near $\lambda=6.25$.
| $\lambda$ | Bias$^2$ | Variance | MSE |
|---|---|---|---|
| 0 | 0 | 1.000 | 1.000 |
| 0.5 | 0.018 | 0.444 | 0.462 |
| 1 | 0.040 | 0.250 | 0.290 |
| 2 | 0.071 | 0.111 | 0.182 |
| 5 | 0.111 | 0.028 | 0.139 |
| 6.25 ($\lambda^\star$) | 0.119 | 0.019 | 0.138 |
| 10 | 0.132 | 0.008 | 0.140 |
MSE first falls as variance drops, then rises once bias becomes too large. The ideal $\lambda$ depends on unknown population values, so real models choose it with validation rather than this formula.
Lasso, Soft Thresholding, and Sparsity
The absolute-value penalty has a sharp point at zero. This allows a coefficient to become exactly zero. Coordinate descent updates one coefficient at a time with the soft-threshold rule
$$\beta_j\leftarrow\frac{S(z_j,\lambda\alpha)}{1+\lambda(1-\alpha)},\qquad S(z,\gamma)=\operatorname{sign}(z)\max(|z|-\gamma,0).$$Here $z_j$ is the correlation between feature $j$ and the residual left after the other features are accounted for. The KKT condition for this one-coordinate problem is
$$0\in-z_j+\beta_j+\lambda\alpha\,\partial|\beta_j|+\lambda(1-\alpha)\beta_j,$$When $|z_j|\le\lambda\alpha$, zero satisfies this condition. Larger positive or negative correlations survive after being moved toward zero. This is the source of Lasso sparsity.
Lasso can select one predictor arbitrarily from a group of strongly correlated predictors. Elastic Net adds an $L_2$ term, making grouped retention more stable.
Duplicate-feature example. If $X_1=X_2$, many coefficient pairs give the same predictions. Ridge prefers an even split because it minimizes squared coefficient size. Lasso may place the whole effect on either copy. Elastic Net encourages a more even allocation while retaining sparsity.
The Regularization Path
A regularization path shows coefficients across many values of $\lambda$. Ridge paths are smooth. Lasso and Elastic Net paths change in pieces as variables enter or leave the active set. At $\lambda_{\max}$, every Lasso coefficient is zero. Lower values allow more variables to enter. Warm starts reuse the previous solution to make the path faster to compute.
Properties and Counterexamples
| Question | Ridge | Lasso | Elastic net |
|---|---|---|---|
| Exact zeros | Generally none | Common | Common when $\alpha>0$ |
| Correlated group | Shrinks together | May pick one arbitrarily | Often keeps a group |
| Closed form | Yes for squared loss | No general matrix form | No general matrix form |
| Primary benefit | Conditioning and variance | Sparse approximation | Compromise between both |
Counterexample. Sparse truth does not guarantee that Lasso selects the correct variables. Correlated columns may produce good prediction with the wrong selected support.
A Ridge Coefficient Path
▶ Coefficients Shrink as Lambda Grows
With $X^\top X=I$ and OLS coefficients $(3,-1.5,0.4)$, each Ridge coefficient becomes $\beta_j^{OLS}/(1+\lambda)$.
Cross-Validation and Production Implications
Lambda must be chosen using training folds, not the final test set. The one-standard-error rule chooses the most regularized model whose validation error is still close to the minimum, often gaining stability with little predictive cost.
Stored model artifacts should include feature order, scaling constants, intercept policy, $\lambda$, $\alpha$, objective convention, tolerance, and convergence status.
The C++ regularized-regression guide implements Ridge and coordinate descent with warm starts and deterministic validation.