Regularization, Identifiability, and Perfect Separation
Logistic regression can classify every training row correctly and still fail to produce finite coefficients. This happens under separation: some linear boundary places every positive row on one side and every negative row on the other. Increasing the coefficient sizes then makes the model more confident without changing a single predicted class, so the unpenalized log-loss keeps falling forever.
Regularization stops this growth by charging the model for large coefficients. It also reduces unstable estimates when predictors are noisy or strongly correlated. This page explains separation, L1 and L2 penalties, penalty tuning, and the difference between regularization and class weighting.
Complete and Quasi-Complete Separation
The vector $\tilde{x}_i$ includes the intercept and all features for row $i$. Complete separation means that some $\beta$ gives
$$\beta^\top\tilde{x}_i>0\text{ for every }y_i=1,\qquad \beta^\top\tilde{x}_i<0\text{ for every }y_i=0.$$Every row lies strictly on the correct side of the boundary. Quasi-complete separation allows one or more rows to sit exactly on the boundary while no row lies on the wrong side.
Separation appears often in small datasets, datasets with many features, categories that occur in only one class, and features accidentally derived from the label. Many solvers do not report a direct error. Instead, the coefficients and standard errors become very large, predicted probabilities become almost exactly $0$ or $1$, and the coefficient norm keeps growing after each iteration. A strong but finite effect eventually settles; separation does not.
Why the Maximum-Likelihood Estimate Diverges
For any $\beta$ that separates the classes, multiplying it by $c>1$ keeps every row on the same side of the boundary. The only change is confidence: positive-row probabilities move closer to $1$, and negative-row probabilities move closer to $0$. Every row's log-loss becomes smaller, so
$$J(c\beta)\ \text{is strictly decreasing in }c,\qquad \lim_{c\to\infty}J(c\beta)=0,$$under complete separation. Zero is the greatest lower bound of the loss, but no finite coefficient vector reaches it. Therefore the maximum-likelihood estimate does not exist as a finite value. Under quasi-complete separation, the coefficients in the separating direction can still diverge while other coefficient directions remain finite.
For example, $x=(-3,-2,-1,1,2,3)$ and $y=(0,0,0,1,1,1)$ are perfectly separated by $x=0$. Unpenalized Newton steps produce slopes $0.857,\,1.475,\,2.196,\,3.050,\,3.992,\,4.972,\dots$. Each slope is larger than the last, and there is no finite destination. The first animation compares this runaway fit with a penalized fit that settles.
L2 and L1 Penalties, Excluding the Intercept
A penalty makes large coefficients expensive. The two common choices are L2, which sums squared slopes, and L1, which sums absolute slopes:
$$J_{\text{L2}}(\beta)=\frac{1}{n}\sum_{i=1}^{n}\big[\operatorname{softplus}(z_i)-y_i z_i\big]+\frac{\lambda}{2}\sum_{j=1}^{d}\beta_j^2,\qquad J_{\text{L1}}(\beta)=\frac{1}{n}\sum_{i=1}^{n}\big[\operatorname{softplus}(z_i)-y_i z_i\big]+\lambda\sum_{j=1}^{d}|\beta_j|.$$The strength $\lambda$ controls the tradeoff. At $\lambda=0$, the fit is unpenalized. A larger $\lambda$ accepts more training loss in exchange for smaller slopes.
- L2 regularization shrinks slopes smoothly toward zero. It usually keeps every feature, adds $\lambda I$ to the slope block of the Hessian, and gives a unique slope vector even when slope columns are collinear.
- L1 regularization has a sharp corner at zero. That corner can set some slopes exactly to zero, so L1 can perform feature selection. It is convex but does not add the smooth curvature that L2 adds.
Both penalties grow as separated slopes grow, so both can create a finite optimum. One exception is a dataset containing only one outcome class. If the intercept is unpenalized, it can still diverge because slope regularization cannot create information about the missing class.
Penalties also depend on feature scale. The same relationship written in smaller units needs a larger coefficient and would receive a larger penalty. Standardizing the non-intercept features before fitting puts their penalties on comparable scales. The scaler must be learned inside each training fold, and fitted coefficients can later be converted back to raw units.
The Bayesian MAP View
A penalty can also be read as a prior preference for smaller slopes. Maximizing likelihood plus that prior gives a maximum a posteriori (MAP) estimate.
- L2 corresponds to independent Gaussian priors, $\beta_j\sim\mathcal{N}(0,\tau^2)$. Since $-\ln\Pr(\beta)$ contains $\sum_j\beta_j^2/(2\tau^2)$, a larger $\lambda$ corresponds to a tighter prior around zero.
- L1 corresponds to independent Laplace priors, $\Pr(\beta_j)\propto e^{-|\beta_j|/b}$. The prior's sharp peak at zero matches L1's tendency to produce exact zero slopes.
Leaving the intercept unpenalized corresponds to placing no shrinkage preference on the baseline log-odds. The prior view does not add evidence to the dataset; it makes the extra assumption behind the finite estimate explicit.
A Unique Estimate Is Not New Information
Regularization makes an ill-posed fit well defined, but it does not make the data more informative. Under separation, the data rewards ever-larger vectors in any separating direction. The penalty chooses a finite scale by balancing fit against coefficient size. Under collinearity, many coefficient vectors can produce the same scores. L2 breaks that tie by choosing the smallest squared-norm slope vector.
In both cases, the penalty supplies an extra selection rule. The resulting estimate is unique under the L2 conditions above, but that uniqueness comes partly from the chosen penalty, not from the likelihood alone.
Coefficient Divergence Versus the Penalized Path
The animation fits the separable dataset two ways at once and tracks the slope across iterations. The unpenalized fit climbs without bound, the visible symptom of a maximum-likelihood estimate that does not exist. The L2-penalized fit with $\lambda=0.5$ converges in a few Newton steps to a finite slope of $0.7015$.
▶ Unpenalized Divergence Against a Converging L2 Fit
On the separable data $x=(-3,-2,-1,1,2,3)$, $y=(0,0,0,1,1,1)$: the unpenalized slope grows every iteration while the L2 fit settles. Both trajectories are computed at runtime by Newton's method.
The Regularization Path and Tuning
A regularization path records the fitted coefficients across many $\lambda$ values. A large $\lambda$ gives strong shrinkage and slopes near zero. A small $\lambda$ gives weak shrinkage and larger slopes. On separated data, the slopes diverge as $\lambda\to0$ because the path approaches the nonexistent unpenalized estimate.
The second animation shows this path. The slope is $0.1626$ at $\lambda=5$, $0.7015$ at $\lambda=0.5$, and $2.1815$ at $\lambda=0.02$.
▶ The L2 Regularization Path
Each step lowers the penalty $\lambda$ and refits; the slope climbs and would diverge as $\lambda\to0$ because the data are separable. Every point is computed at runtime.
The unpenalized training loss always prefers the weakest penalty, so $\lambda$ must be chosen on held-out data. Cross-validation follows a simple cycle:
- The training data is split into folds.
- For each fold, feature scaling is learned from its training portion and applied to its validation portion.
- A logarithmic grid of $\lambda$ values is fitted, and the value with the best average validation score, such as log-loss, is selected.
Nearby path values often use similar coefficients, so each fit can start from the previous fit to save time. This is called a warm start.
Regularization Is Not Class Weighting
The two settings answer different questions. Class weighting changes how much each row contributes to the loss. It can give a minority class more influence or represent unequal mistake costs. Regularization limits how much coefficient size the model may use while reducing that loss.
Class weighting does not bound a separating coefficient vector. Scaling that vector still pushes every weighted row loss toward zero. Regularization does bound it. Conversely, regularization does not tell the model that one class or one kind of error matters more. Both settings can be used together because they control different parts of the objective.
| Concern | Class weighting | Regularization |
|---|---|---|
| Acts on | the loss (per-row weights) | the coefficients (magnitude penalty) |
| Primary effect | intercept and threshold, class balance | slope magnitude and variance |
| Fixes separation? | no | yes |
| Addresses imbalance or cost? | yes | not directly |
Edge Cases and the Implementation
- Silent separation. A category absent from one class, or a feature derived from the label, can separate the data. A coefficient that grows every iteration is the main warning.
- Penalized intercept. Including the intercept in the penalty pulls all probabilities toward $0.5$ and removes exact mean calibration.
- Unstandardized features. Raw feature units can cause unequal shrinkage of otherwise equivalent predictors. Scaling learned inside each training fold avoids that distortion.
- Tuning leakage. Choosing $\lambda$ or scaling statistics before the validation split leaks validation information into training and makes error estimates too optimistic.
The C++ regularization section implements the penalized gradient and Hessian and shows the finite fit on separated data. A dedicated regularized logistic regression chapter is also planned.