Logistic Regression: Essential Interview Derivations
This chapter collects the short derivations that explain logistic regression from first principles. Each result follows from the one before it: sigmoid, likelihood, gradient, Hessian, Newton and IRLS, softmax, regularization, coefficient interpretation, and cost-sensitive decisions.
Derivation 1: The Sigmoid and Its Derivative
The linear score $z=\beta^\top\tilde x$ can be any real number. The sigmoid maps it into $(0,1)$:
$$\sigma(z)=\frac{1}{1+e^{-z}},\qquad 1-\sigma(z)=\sigma(-z),\qquad \sigma'(z)=\sigma(z)\big(1-\sigma(z)\big).$$The derivative follows directly:
$$\frac{d}{dz}(1+e^{-z})^{-1}=\frac{e^{-z}}{(1+e^{-z})^2} =\frac{1}{1+e^{-z}}\frac{e^{-z}}{1+e^{-z}} =\sigma(z)\big(1-\sigma(z)\big).$$The inverse function is the logit, $\sigma^{-1}(p)=\ln\frac{p}{1-p}$. Therefore $z$ is the log-odds. The derivative is largest at $z=0$, where it equals $0.25$, and approaches zero for very large positive or negative scores.
Derivation 2: Log-Loss from the Bernoulli Likelihood
For one binary label, the Bernoulli probability can represent both cases in one expression:
$$\Pr(y_i\mid x_i)=p_i^{\,y_i}(1-p_i)^{1-y_i}.$$It equals $p_i$ when $y_i=1$ and $1-p_i$ when $y_i=0$. Assuming independent rows, the sample likelihood multiplies these terms. Taking the logarithm turns the product into a sum. Negating and averaging gives log-loss:
$$J(\beta)=-\frac{1}{n}\sum_{i=1}^{n}\big[y_i\ln p_i+(1-y_i)\ln(1-p_i)\big]=\frac{1}{n}\sum_{i=1}^{n}\big[\operatorname{softplus}(z_i)-y_i z_i\big],$$The second form uses $\operatorname{softplus}(z)=\ln(1+e^z)$ and is stable because it avoids taking the logarithm of a rounded probability. Minimizing binary cross-entropy is therefore exactly the same problem as maximizing the Bernoulli likelihood.
Derivation 3: The Gradient Is a Residual
For one row, $\ell_i=\operatorname{softplus}(z_i)-y_i z_i$. Since $\operatorname{softplus}'(z)=\sigma(z)$,
$$\frac{\partial \ell_i}{\partial z_i}=\sigma(z_i)-y_i=p_i-y_i.$$The chain rule contributes $\partial z_i/\partial\beta=\tilde x_i$. Averaging all rows gives
$$\nabla J(\beta)=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)\,\tilde x_i=\frac{1}{n}X^\top(p-y).$$The pattern is simple: probability residual times feature. At an unpenalized optimum, $X^\top(p-y)=0$. The intercept column contains only ones, so its equation is $\sum_i(p_i-y_i)=0$. Thus the average training probability equals the training positive rate.
The animation checks the analytic gradient against a central finite difference. Agreement between the two is a practical test for a derivation or implementation error.
▶ Verifying the Gradient Against Finite Differences
A six-row dataset with an intercept and two features at $\beta=(0.2,-0.5,0.8)$. Each step reveals one coordinate, computing the analytic residual gradient $\tfrac1n X^\top(p-y)$ beside the central difference $\tfrac{J(\beta+\varepsilon e_j)-J(\beta-\varepsilon e_j)}{2\varepsilon}$. All values are computed at runtime.
The two columns agree to about ten decimal places. The remaining difference comes from finite-difference truncation and floating-point rounding.
Derivation 4: The Hessian and Convexity
Differentiating again uses $\partial p_i/\partial z_i=p_i(1-p_i)$. Each row contributes a feature outer product multiplied by this nonnegative weight:
$$\nabla^2 J(\beta)=\frac{1}{n}\sum_{i=1}^{n}p_i(1-p_i)\,\tilde x_i\tilde x_i^\top=\frac{1}{n}X^\top W X,\qquad W=\operatorname{diag}\big(p_i(1-p_i)\big).$$For any direction $v$,
$$v^\top\nabla^2 J(\beta)\,v=\frac{1}{n}\sum_{i=1}^{n}p_i(1-p_i)\,(\tilde x_i^\top v)^2\ge0,$$Every term is nonnegative, so the Hessian is positive semidefinite and the loss is convex. At any finite coefficient vector, every probability lies strictly between $0$ and $1$. If $X$ has full column rank, the Hessian is then positive definite and the loss is strictly convex.
Derivation 5: Newton's Step Is Reweighted Least Squares
Newton's update is $\beta^{+}=\beta-\big(\nabla^2 J\big)^{-1}\nabla J$. Substituting the gradient and Hessian cancels the shared $1/n$:
$$\beta^{+}=\beta-\big(X^\top W X\big)^{-1}X^\top(p-y).$$The working response is $\xi=X\beta+W^{-1}(y-p)$. Its weighted least-squares solution is
$$\big(X^\top W X\big)^{-1}X^\top W \xi =\beta+\big(X^\top W X\big)^{-1}X^\top(y-p) =\beta^{+}.$$One Newton step and one weighted least-squares fit therefore produce the same coefficients. The probabilities change after each step, so $W$ and $\xi$ also change. Repeating this process gives iteratively reweighted least squares (IRLS). Numerical code solves the linear system rather than forming the inverse.
Derivation 6: The Softmax Gradient and Its Jacobian
For $K$ classes, softmax gives $p_k=e^{z_k}/\sum_m e^{z_m}$. Subtracting $m=\max_k z_k$ from every score leaves the probabilities unchanged and gives the stable log-normalizer
$$\operatorname{LSE}(z)=m+\ln\sum_k e^{z_k-m},\qquad \log p_k=z_k-\operatorname{LSE}(z).$$The common factor cancels between numerator and denominator. The softmax Jacobian is
$$\frac{\partial p_k}{\partial z_j}=p_k\big(\delta_{kj}-p_j\big),$$The diagonal case $j=k$ gives $p_k(1-p_k)$; an off-diagonal case gives $-p_kp_j$. Combining this Jacobian with one-hot cross-entropy $\ell=-\sum_k y_k\ln p_k$ gives
$$\frac{\partial \ell}{\partial z_j}=\sum_k y_k\big(p_j-\delta_{kj}\big)=p_j-y_j,$$the same predicted-minus-observed pattern as the binary model. Across all rows and classes, the gradient is $X^\top(P-Y)$.
Derivation 7: L2 Regularization Restores a Finite, Unique Solution
Under complete separation, increasing coefficient magnitude keeps lowering unpenalized loss. An L2 penalty on the slopes grows in the opposite direction:
$$R=\operatorname{diag}(0,1,\ldots,1),\qquad J_\lambda(\beta)=J(\beta)+\frac{\lambda}{2}\beta^\top R\beta,$$ $$\nabla J_\lambda=\frac{1}{n}X^\top(p-y)+\lambda R\beta,\qquad \nabla^2 J_\lambda=\frac{1}{n}X^\top W X+\lambda R.$$The zero in $R$ excludes the intercept. The Hessian gains $\lambda$ curvature in every slope direction, so collinear or separating slopes cannot grow for free. If both outcome classes are present and $\lambda>0$, the usual intercept-plus-slopes problem has a unique finite solution. If every label is the same, the unpenalized intercept can still diverge because slope regularization cannot invent the missing class.
Derivation 8: Coefficients as Log-Odds Ratios
Holding the other included features fixed, and assuming no interaction changes the effect of $x_j$, a one-unit increase adds $\beta_j$ to the log-odds:
$$\log\operatorname{odds}(x_j+1)-\log\operatorname{odds}(x_j)=\beta_j.$$Exponentiating gives
$$\frac{\operatorname{odds}(x_j+1)}{\operatorname{odds}(x_j)}=e^{\beta_j}.$$Thus $e^{\beta_j}$ is the conditional odds ratio for a one-unit change. The probability change is not constant because the sigmoid is steepest near probability $0.5$ and flatter near $0$ or $1$. Interactions also make the odds ratio depend on the interacting feature.
The decision boundary $\beta^\top\tilde x=0$ is a hyperplane. This is why logistic regression is a linear classifier even though the sigmoid is nonlinear.
Derivation 9: The Cost-Sensitive Threshold
False-positive and false-negative costs are denoted $C_{\mathrm{FP}}$ and $C_{\mathrm{FN}}$, with zero cost for correct actions. At calibrated probability $p$, the positive decision has expected cost $(1-p)C_{\mathrm{FP}}$ and the negative decision has expected cost $pC_{\mathrm{FN}}$. The positive action is cheaper when
$$(1-p)C_{\mathrm{FP}}\le pC_{\mathrm{FN}}\quad\Longleftrightarrow\quad p\ge\frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.$$The threshold is $0.5$ only when the two mistake costs are equal. Prevalence is already inside a calibrated posterior probability and is not multiplied into the threshold again. Scores fitted under a changed sampling prior need an intercept correction first.
ROC, PR, and Calibration Distinctions
| Question | Reasoned answer |
|---|---|
| Can AUC remain unchanged after every probability is squared? | Yes. Squaring is strictly increasing on $[0,1]$, so every positive-negative ordering is preserved. Calibration and log-loss can change sharply. |
| Why can ROC-AUC look strong when precision is poor? | FPR divides false positives by all negatives. With a huge negative class, many false alarms can be a small FPR, while precision exposes them in $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$. |
| Does PR-AUC have a prevalence-free baseline? | No. A random ranking has expected precision equal to prevalence, so PR curves from populations with different base rates are not directly comparable without context. |
| Can a constant predictor be perfectly calibrated? | Yes. Reporting the prevalence for every case is calibrated in the large and conditionally at its only forecast value, but has no discrimination or resolution. |
| Does choosing a new threshold improve calibration? | No. A threshold changes hard decisions, not probability values. Recalibration changes the score-to-probability map, while thresholding selects an operating point. |
| Does Platt scaling improve ROC-AUC? | A strictly increasing fitted map preserves all ranks and therefore preserves ROC-AUC. Isotonic pooling can introduce ties and slightly alter it. |
Interactive Derivation Dependency Map
The animation shows the dependency chain. The sigmoid and likelihood produce the gradient. The gradient and sigmoid derivative produce the Hessian. The gradient and Hessian produce Newton and IRLS. A calibrated posterior and mistake costs produce the decision threshold.
▶ Building the Logistic Derivation Graph
Quick Reference and the Implementation
The central results compress into a single reference table.
| Quantity | Result |
|---|---|
| Sigmoid derivative | $\sigma'(z)=\sigma(z)(1-\sigma(z))$ |
| Objective | $J=\tfrac1n\sum[\operatorname{softplus}(z_i)-y_iz_i]$ |
| Gradient | $\nabla J=\tfrac1n X^\top(p-y)$ |
| Hessian | $\nabla^2 J=\tfrac1n X^\top W X,\ W=\operatorname{diag}(p_i(1-p_i))\succeq0$ |
| Newton step | $\beta^{+}=(X^\top W X)^{-1}X^\top W \xi,\ \xi=X\beta+W^{-1}(y-p)$ |
| Softmax Jacobian | $\partial p_k/\partial z_j=p_k(\delta_{kj}-p_j)$ |
| Stable log-sum-exp | $m+\ln\sum_k e^{z_k-m}$ with $m=\max_k z_k$ |
| L2 Hessian | $\tfrac1n X^\top W X+\lambda R,\ R=\operatorname{diag}(0,1,\ldots,1)$ |
| Cost threshold | $t^\star=C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$ |
| Coefficient | $e^{\beta_j}$ is the odds ratio for a unit change |
The C++ gradient and Hessian section exercises these formulas with stable sigmoid, softplus, log-loss, and a finite-difference gradient check. A dedicated sigmoid, log-loss, and gradient chapter is also planned.