Diagnostics and Production Failure Modes
A logistic model needs two kinds of checks. Before deployment, diagnostics ask whether the fitted shape is adequate, whether a few rows control the result, and whether estimation is stable. After deployment, monitoring asks whether features, probabilities, and outcomes still behave like the data used for development.
These checks catch different failures. A model can fit the training data poorly from the start, or it can begin well and decay later. A particularly quiet failure keeps a strong ranking while its probabilities lose calibration.
Deviance, Pseudo-R-squared, and Likelihood-Ratio Tests
Deviance measures how far the fitted likelihood is from a saturated model that reproduces every observed label:
$$D=-2\big(\ell_{\text{model}}-\ell_{\text{saturated}}\big),$$Smaller deviance means better in-sample fit, and zero is the saturated fit. The null model contains only an intercept. Comparing the fitted log-likelihood with that null gives McFadden's pseudo-$R^2$:
$$R^2_{\text{McF}}=1-\frac{\ell_{\text{model}}}{\ell_{\text{null}}},$$A value near zero means little improvement over the intercept-only model; a larger value means more improvement. It is not explained variance and should not be compared numerically with linear regression's $R^2$. It can also be negative when the fitted model is worse than the null model.
For nested models, the likelihood-ratio statistic compares a reduced model with a full model:
$$\text{LR}=2\big(\ell_{\text{full}}-\ell_{\text{reduced}}\big)\ \sim\ \chi^2_{q},$$Here, $q$ is the number of added coefficients. The chi-square reference is asymptotic and needs regular estimation conditions, so separation or boundary parameters can invalidate it. For non-nested candidates, AIC and BIC compare fit while penalizing additional parameters.
Residuals and Goodness-of-Fit
A raw binary residual is $y_i-p_i$. Since $y_i$ is only zero or one, individual raw residuals have limited shape. Two scaled versions are more useful:
$$r_i=\frac{y_i-p_i}{\sqrt{p_i(1-p_i)}},\qquad d_i=\operatorname{sign}(y_i-p_i)\sqrt{-2\big[y_i\ln p_i+(1-y_i)\ln(1-p_i)\big]},$$The Pearson residual divides by the Bernoulli standard deviation. The deviance residual assigns each row its signed share of deviance, with $\sum_i d_i^2=D$. Large values mark rows that the model considered unlikely.
Patterns matter more than one residual. Residuals are examined against continuous features, time, groups, and the fitted logit. The Hosmer-Lemeshow test instead groups rows by predicted probability and compares observed with expected counts. Its result changes with the chosen groups, has low power in small samples, and can reject tiny deviations in huge samples. It is a smoke test, not a replacement for a reliability diagram.
Logit Linearity, Influence, and the Calibration Slope
Logistic regression assumes linearity in the log-odds, not in probability. A spline fit or partial-effect plot can reveal a curve that a straight logit misses. The Box-Tidwell test adds $x\ln x$, but requires a positive predictor and tests only that particular departure. A useful curve can be represented with a transform, spline, or interaction.
The IRLS view supplies a working hat matrix:
$$H=W^{1/2}X\big(X^\top W X\big)^{-1}X^\top W^{1/2},\qquad W=\operatorname{diag}\big(p_i(1-p_i)\big),$$A large diagonal value $H_{ii}$ marks a row with unusual feature leverage. Cook's distance and dfbeta-style measures then estimate how much the fit or one coefficient changes when that row is removed. Large coefficients, growing standard errors, and a nearly singular Hessian instead point toward separation.
The calibration slope must be measured on held-out predictions or corrected for training optimism. An intercept of $0$ and slope of $1$ are ideal. A slope below $1$ means the old logits are too extreme, a common sign of overfitting. Measuring this slope on the same rows used for fitting can hide exactly that overfitting.
Misspecification Beyond Logit Linearity
Several failures can remain after each individual feature looks reasonable:
- Missing interaction. A feature has different effects across groups, but the model forces one shared effect.
- Missing curve. Residuals follow a pattern across a continuous predictor.
- Omitted confounder. A coefficient absorbs association that belongs to an unmeasured cause.
- Collinearity. Similar columns produce unstable coefficients even when predictions remain stable.
A spline benchmark or flexible reference model can expose structure missed by the linear logit. Sparse category or interaction cells need special attention. If one cell contains only one outcome, an unpenalized coefficient can diverge. Penalization, Firth's bias-reduced likelihood, more data, or removal of an unsupported interaction can make the estimate finite. Categories should be merged only when their real meanings support the merge.
Drift: Covariate, Prior, and Concept
Drift is not one problem. The changed probability determines the response:
| Type of drift | What changes | Consequence and response |
|---|---|---|
| Covariate shift | $\Pr(x)$ changes, $\Pr(y\mid x)$ stays fixed | Different kinds of rows arrive, but the conditional model remains valid. Aggregate performance may change because the case mix changed. |
| Prior or label shift | $\Pr(y)$ changes, $\Pr(x\mid y)$ stays fixed | The base rate changes. Under this assumption, a log-odds intercept correction restores probabilities. |
| Concept drift | $\Pr(y\mid x)$ changes | The learned relationship is stale. Fresh labels and model refitting are usually required. |
When prior shift assumptions truly hold and the old probabilities were calibrated, the correction is
$$\Delta\beta_0=\operatorname{logit}(\pi_{\text{new}})-\operatorname{logit}(\pi_{\text{old}}),$$which is the same correction used after case-control resampling. The three drift types can look similar in an unlabeled score dashboard. Delayed labels are needed to verify whether only the case mix moved or the conditional relationship changed.
Monitoring the Score Distribution with PSI
The population stability index (PSI) compares a current score histogram with a reference histogram:
$$\mathrm{PSI}=\sum_{b}\big(c_b-r_b\big)\ln\frac{c_b}{r_b},$$Here, $r_b$ and $c_b$ are the reference and current fractions in bin $b$. PSI needs no labels, so it can produce an early warning. Zero-count bins require a documented smoothing rule.
Values below $0.1$, from $0.1$ to $0.25$, and above $0.25$ are often called small, moderate, and large shifts. These are conventions, not universal statistical thresholds. Historical variation and operational impact should set the actual alert levels.
▶ Accumulating the Population Stability Index Across Score Bins
Reference and current score histograms as fractions per bin. Each step reveals one bin, contrasts the two bars, and adds its divergence term to the running PSI against the moderate and major shift bands. Terms and total are computed at runtime.
In the animation, the lowest score bin falls from $30\%$ to $18\%$, while the top bins gain mass. PSI reaches $0.148$, which crosses the conventional moderate-shift line. This proves only that the score distribution moved. It does not identify covariate, prior, or concept drift.
Train-Serve Skew and Leakage
Train-serve skew means a feature has different definitions offline and online. Common causes include different missing-value defaults, timezones, units, rounding, category maps, or freshness windows. Both pipelines can run successfully while sending the model different meanings.
Target leakage occurs when training uses information unavailable at prediction time. Offline metrics then look excellent because the feature contains part of the answer, and live performance collapses when that information disappears.
Serving validation covers column names, types, ranges, units, category vocabularies, freshness, and missing rates. Training and serving ideally share one versioned transformation. Golden records provide a second check by fixing expected feature vectors and predictions for known inputs.
Feedback Loops and Selection Bias
A model can change which labels become observable. A blocked transaction never reveals its natural outcome. An item that was never shown never reveals whether it would have been clicked. Retraining only on observed outcomes then reinforces the model's existing choices.
Random exploration, holdout traffic, or propensity corrections can recover information about suppressed cases. Cost changes also require a new threshold. Prevalence changes require updated probabilities before the same cost-based probability threshold remains meaningful.
Silent Failures and What to Monitor
Monitoring has two timing layers:
| Available immediately | Available after labels arrive |
|---|---|
| Request volume, schema validity, feature freshness, missing rates, feature and score distributions, latency, fallback rate | Prevalence, log-loss, Brier score, calibration intercept and slope, ROC-AUC, AP, confusion counts, and realized cost |
Aggregate values can hide a failure limited to one region, customer group, device, or time period, so important metrics are also monitored by segment. The monitored quantity should match the downstream use: probability-consuming systems need calibration checks, while ranking systems need ranking checks.
Monitoring, Retraining, and Incident Workflow
Alert levels are based on historical variation and meaningful effect sizes. With very large traffic, statistical significance alone can flag changes too small to matter.
- The alert is checked against data freshness, label delay, and monitoring errors.
- The change is localized by model version, feature, segment, and time.
- Pipeline faults are contained with a fallback, rollback, or reduced traffic.
- The repair follows the diagnosis: prior correction for verified prior shift, recalibration for stable ranking with bad probabilities, or refitting for changed conditional relationships.
- The candidate is evaluated on a later time window and canaried with rollback criteria.
The data snapshot, feature code, model, calibrator, threshold, cost matrix, and monitors form one versioned decision system. Retraining is a response to changed relationships or new information, not an automatic answer to every distribution alert.
Edge Cases and the Implementation
- Grouped goodness-of-fit. Hosmer-Lemeshow depends on group choices and sample size, so reliability plots carry more detail.
- Drift diagnosis. An intercept correction is valid only for verified prior shift, not general concept drift.
- Unlabeled alerts. PSI detects distribution movement but cannot show whether predictions became wrong.
- Offline limits. Shadow or canary deployment can expose train-serve skew that offline evaluation never exercised.
The C++ diagnostics section computes deviance, residuals, and evaluation summaries. A dedicated evaluation and calibration chapter is also planned.