Logistic Regression: Overview and Roadmap
Logistic regression estimates the probability of a binary outcome, such as pass or fail. It first computes a linear score and then converts that score into a probability:
$$\Pr(Y=1\mid x)=\sigma(z),\qquad z=\beta_0+\sum_{j=1}^{d}\beta_j x_j=\beta^\top\tilde{x},\qquad \sigma(z)=\frac{1}{1+e^{-z}},$$The augmented row $\tilde x=(1,x_1,\dots,x_d)^\top$ includes the intercept. The score $z$ can be any real number. The sigmoid $\sigma(z)$ maps it into the range $0$ to $1$.
| Symbol | Meaning |
|---|---|
| $n$ | number of data rows |
| $d$ | number of predictors, not counting the intercept |
| $x_i,\ \tilde{x}_i$ | row $i$ before and after adding the leading $1$ |
| $y_i\in\{0,1\}$ | binary label for row $i$ |
| $z_i=\beta^\top\tilde{x}_i$ | unbounded linear score |
| $p_i=\sigma(z_i)$ | predicted probability of class $1$ |
| $\beta$ | coefficient vector including intercept $\beta_0$ |
The Bernoulli Model and the Logit Link
A binary label follows a Bernoulli distribution with probability $p$. Its mean is $p$ and its variance is $p(1-p)$. Logistic regression connects this probability to the linear score through the logit:
$$\operatorname{logit}(p)=\ln\frac{p}{1-p}=z,$$The inverse link is the sigmoid, $p=\sigma(z)$. The Bernoulli variance changes with $p$, so unequal outcome variance is built into the model rather than treated as a violation. Probit and complementary log-log are alternative links, but logit is common because coefficients have a simple odds-ratio interpretation and the training objective is convex.
Probability Versus Decision
The model outputs a probability, not a class label. A threshold $t$ creates the label: predict class $1$ when $p_i\ge t$. Changing the threshold changes decisions but does not change the fitted probabilities.
This separation matters when error costs or class balance change. The same probability model can use a high-recall threshold or a high-precision threshold without retraining. Accuracy alone can hide this tradeoff.
Three Separate Questions
Target: which probability or coefficient is the analysis trying to estimate, and for which population?
Training: which coefficients minimize Bernoulli log-loss? Gradient descent, Newton's method, or IRLS can solve this convex problem.
Evaluation: do the probabilities rank cases well, match observed frequencies, and support useful decisions at the chosen threshold? Ranking, calibration, and decision cost are different properties.
Relationship to Linear Regression
Both models use the score $z=\beta^\top\tilde x$. Linear regression returns the score as a continuous prediction and minimizes squared error. Logistic regression converts the score to a probability and minimizes log-loss. Linear coefficients use outcome units; logistic coefficients use log-odds units.
Newton's method for logistic regression can be written as a weighted least-squares update:
$$\beta^{+}=\big(X^\top W X\big)^{-1}X^\top W\,\xi,\qquad \xi_i=z_i+\frac{y_i-p_i}{p_i(1-p_i)},\qquad W=\operatorname{diag}\big(p_i(1-p_i)\big),$$The working response $\xi$ and weights $W$ change after every iteration. This is IRLS. The same design-matrix and conditioning issues from linear regression still matter.
| Aspect | Linear regression | Logistic regression |
|---|---|---|
| Outcome | continuous $y\in\mathbb{R}$ | binary $y\in\{0,1\}$ |
| Link | identity, $\mathbb{E}[y\mid x]=z$ | logit, $\ln\frac{p}{1-p}=z$ |
| Loss | squared error | negative Bernoulli log-likelihood (log-loss) |
| Variance | constant $\sigma^2$ (assumed) | mean-linked $p(1-p)$ |
| Solution | closed-form normal equations | iterative, no closed form |
| Gradient residual | $\hat y_i-y_i$ | $p_i-y_i$ |
| Coefficient units | units of $y$ per unit of $x_j$ | log-odds per unit of $x_j$ |
Both gradients use a residual times a feature. Linear regression uses $\hat y_i-y_i$; logistic regression uses $p_i-y_i$. The sigmoid removes the closed form, so logistic training needs iteration.
From Labels to Decisions
The animation uses eight study-hour rows. It fits the coefficients, converts scores into probabilities, applies a threshold, and reports the resulting metrics.
▶ Score, Probability, Threshold, and Metrics
Every value is computed from the eight rows. Each step reveals one stage of the fitted classifier.
The fitted 0.5 boundary is $2.25$ study hours. Two rows fall on the wrong side, so training accuracy is six of eight and average log-loss is $0.3129$.
Assumptions, Counterexamples, and Edge Cases
- Linear log-odds: curved relationships need transformed features, splines, or interactions.
- Independent rows: clustered or repeated rows need dependence-aware methods.
- No perfect separation: exact class separation makes unregularized coefficients diverge.
- Stable coding and population: swapping the positive class flips signs, while deployment shift can break calibration.
Roadmap for This Track
The ten published chapters move from probability and interpretation through fitting, optimization, regularization, multiclass models, thresholds, metrics, calibration, and production use.
- Probability, Log-Odds, Decision Boundaries, and InterpretationReady · the sigmoid and logit, odds ratios and their units, marginal effects, interactions, threshold-specific boundaries, and the non-collapsibility caution.
- Maximum Likelihood and the Log-Loss DerivationReady · the Bernoulli likelihood, cross-entropy, the numerically stable softplus form, convexity, and the gradient and Hessian.
- Gradient Descent, Newton's Method, and IRLSReady · first-order and second-order optimization, the iteratively reweighted least-squares working response, conditioning, and convergence.
- Regularization, Identifiability, and Perfect SeparationReady · why separable data breaks the maximum-likelihood estimate, and how L2 and L1 penalties and a Bayesian prior restore a finite solution.
- Multiclass Logistic and Softmax RegressionReady · one-vs-rest against the multinomial model, the numerically stable softmax, the categorical likelihood, and reference-class identifiability.
- Decision Thresholds, Class Imbalance, and Error CostsReady · choosing a threshold, handling imbalance, and weighing false positives against false negatives.
- Confusion Matrix, ROC-AUC, PR-AUC, and Log-LossReady · ranking, thresholded, and probabilistic metrics as complementary views of a classifier.
- Probability Calibration and ReliabilityReady · reliability diagrams, the Brier score, Platt scaling, and isotonic regression.
- Diagnostics and Production Failure ModesReady · fit diagnostics and the ways deployed classifiers silently fail.
- Logistic Regression: Essential Interview DerivationsReady · the log-loss gradient, the Hessian, and the softmax generalization worked from first principles.
The matching implementation series, built from scratch in standard-library C++ with an explicit fit, predict, and evaluate lifecycle, is the published overview at From Scratch · C++ · Logistic Regression, which covers the stable sigmoid and log-loss primitives, the gradient and Hessian, solver choices, separation, thresholds, softmax, and a deterministic numerical smoke test. Its implementation sequence is represented by the planned stubs for sigmoid, log-loss, and gradients, Newton and IRLS, regularized logistic regression, and multiclass softmax. The theory pages link to the relevant overview sections and matching stubs rather than repeating full programs.