Maximum Likelihood and the Log-Loss Derivation
Log-loss comes directly from the Bernoulli likelihood. The derivation explains why the loss rewards correct probabilities, why its gradient is $p-y$, why the objective is convex, and why stable code works from logits rather than rounded probabilities.
The Bernoulli Likelihood
For one row, the probability of the observed binary label is
$$\Pr(y_i\mid x_i;\beta)=p_i^{\,y_i}(1-p_i)^{1-y_i},$$This equals $p_i$ for label $1$ and $1-p_i$ for label $0$. Assuming conditionally independent rows, the sample likelihood is
$$L(\beta)=\prod_{i=1}^{n}p_i^{\,y_i}(1-p_i)^{1-y_i}.$$Maximum likelihood chooses coefficients that make the observed labels most probable. Taking logs turns the product into a stable sum:
$$\ell(\beta)=\sum_{i=1}^{n}\Big[y_i\ln p_i+(1-y_i)\ln(1-p_i)\Big].$$Minimizing the negative average log-likelihood gives log-loss, also called binary cross-entropy:
$$J(\beta)=-\frac{1}{n}\sum_{i=1}^{n}\Big[y_i\ln p_i+(1-y_i)\ln(1-p_i)\Big].$$A confident correct prediction gets a small loss. A confident wrong prediction gets a large loss.
The Logit Form and the Stable Softplus
Computing log-loss from a rounded probability can require $\log(0)$. Working directly from the logit avoids this problem. The softplus function is
$$\operatorname{softplus}(z)=\ln(1+e^{z}),$$A single stable expression then handles both labels:
$$\ell_i(z_i,y_i)=\operatorname{softplus}(z_i)-y_i z_i=\begin{cases}\operatorname{softplus}(-z_i)=-\ln\sigma(z_i)&y_i=1,\\[2pt]\operatorname{softplus}(z_i)=-\ln\big(1-\sigma(z_i)\big)&y_i=0.\end{cases}$$The mean loss is
$$J(\beta)=\frac{1}{n}\sum_{i=1}^{n}\big[\operatorname{softplus}(z_i)-y_i z_i\big],\qquad z_i=\beta^\top\tilde{x}_i.$$A sign-aware softplus implementation keeps the exponent non-positive and stays finite for every finite logit.
The Gradient and the Hessian
The derivative of softplus is the sigmoid, so the per-row derivative is simply predicted probability minus label:
$$\frac{\partial \ell_i}{\partial z_i}=\sigma(z_i)-y_i=p_i-y_i.$$Applying the chain rule gives the batch gradient
$$\nabla J(\beta)=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)\,\tilde{x}_i=\frac{1}{n}X^\top(p-y).$$This has the same residual-times-feature pattern as linear regression. With an intercept, the fitted average probability equals the training positive rate.
Differentiating again gives the Hessian
$$\nabla^2 J(\beta)=\frac{1}{n}\sum_{i=1}^{n}p_i(1-p_i)\,\tilde{x}_i\tilde{x}_i^\top=\frac{1}{n}X^\top W X,\qquad W=\operatorname{diag}\big(p_i(1-p_i)\big).$$Every weight lies between $0$ and $0.25$, so the Hessian is positive semidefinite.
Convexity
For any direction $v$,
$$v^\top\nabla^2 J(\beta)\,v=\frac{1}{n}\sum_{i=1}^{n}p_i(1-p_i)\,(\tilde{x}_i^\top v)^2\ge0,$$The sum cannot be negative, so log-loss is convex. Any finite stationary point is a global minimum. Full rank and positive weights make the minimum unique. Collinearity creates flat directions, while perfect separation removes the finite optimum by driving probabilities toward $0$ and $1$.
▶ Newton Steps on Convex Log-Loss
Each step builds a local quadratic approximation to the one-parameter loss and jumps to its minimum.
The sequence $0\to0.6667\to0.7530\to0.7563$ shows rapid Newton convergence. Positive curvature keeps the loss bowl convex along the path.
Log-Loss Is a Proper Scoring Rule
Log-loss is a strictly proper scoring rule: expected loss is minimized by reporting the true probability. If the true probability is $q$ and the model reports $p$,
$$\mathbb{E}_{y\sim\text{Bernoulli}(q)}\big[\ell(p,y)\big]=-\big[q\ln p+(1-q)\ln(1-p)\big],$$The derivative is zero only at $p=q$. Log-loss therefore rewards honest probability estimates rather than only correct class labels.
Saturation and Numerical Failure
For large logits, the sigmoid may round to exactly $0$ or $1$. A probability-first formula then evaluates $\log(0)$ and can produce infinity or NaN. The logit-based softplus form stays finite.
The animation compares both formulas at increasingly extreme logits.
▶ Naive Probability-First Loss Versus the Stable Softplus Form
Each step reveals one logit and label, computing the naive loss from an already-rounded probability and the stable loss from the raw logit. Values are computed at runtime.
Assumptions, Edge Cases, and the Implementation
- Independent rows: clustered data needs dependence-aware methods.
- No closed form: the sigmoid makes $X^\top(p-y)=0$ nonlinear, so coefficients are found iteratively.
- No perfect separation: separated classes have no finite unregularized MLE.
- Intercept check: average fitted probability equals the training positive rate.
The C++ sigmoid, log-loss, and gradient guide implements these stable formulas and verifies them at extreme logits.