Logistic Regression: From Scores to Learning

October 2026 · 9 min read

Expanded from Logistic Regression.pdf · View a handwritten page

Naïve Bayes estimated how inputs occur within each class. Logistic regression directly estimates the chance of a positive label. We will start with a few learners, turn a score into a probability, and calculate one training update.

What are we trying to predict?

Begin with the illustrative score $z=-1.5+x$. These are starting coefficients, not fitted estimates. A score can be any real number, so we pass it through the sigmoid to obtain a probability between 0 and 1:

Zero score gives probability 0.5. Large positive scores approach probability 1; large negative scores approach 0. For finite scores, the mathematical sigmoid never reaches either endpoint exactly.

Understand what the linear score means

The right-hand side comes from our choice of model. We choose a linear score $z=b+wx$, which can be any real number. Passing it through the sigmoid converts that score into a probability between 0 and 1. This choice is equivalent to assuming that the log-odds change linearly with the input.

To see the connection, start with the sigmoid and rearrange it:

The left-hand side is the odds: the probability of class 1 divided by the probability of class 0. Taking the natural logarithm cancels the exponential. Substitute our chosen score $z=b+wx$:

Thus $b$ is the log-odds when $x=0$, and $w$ is the change in log-odds for a one-unit increase in $x$. Training learns these two coefficients from the labeled data.

With $w=1$, one additional hour adds 1 to log odds and multiplies odds by $e\approx2.718$. It does not add a fixed amount to probability. From 0 to 1 hour, our probability rises by about 0.1951; from 1 to 2 hours, it rises by about 0.2449.

Why is logistic regression called regression?

The regression relationship is between the inputs and the log-odds. Ordinary linear regression models the mean output directly as $b+wx$. Logistic regression models the log-odds as $b+wx$, then uses the sigmoid to recover a probability. This makes it a generalized linear model: the mean response is connected to a linear score through a transformation.

The observed labels are still 0 or 1. The model estimates $p$, the probability of label 1; for a binary outcome, this is also its mean. We learn $b$ and $w$ from the observed labels, without needing observed log-odds for each row.

Classification happens when we apply a threshold to that probability. Changing the threshold changes the predicted class without changing the fitted probability model. Logistic regression is therefore a regression model used for classification.

Use the observed label to choose the right probability

For a passed learner, the model assigns the observed label probability $p$. For a failed learner, it assigns $1-p$. A single expression handles both cases when labels are 0 or 1:

For Asha, the observed label is 0, so the relevant probability is $1-0.1824=0.8176$. For Cara, the observed label is 1, so it is 0.6225. Multiply those observed-label probabilities over independent training rows to obtain the conditional likelihood.

Taking the negative logarithm produces binary cross-entropy, also called binary log loss:

The row losses are about 0.2014, 0.4741, 0.4741, and 0.2014. Their average is 0.3377, using natural logarithms. A positive example predicted at 0.9 has loss $-\log0.9\approx0.1054$; predicted at 0.1, it has loss about 2.3026. Being confidently wrong is expensive.

The same BCE loss, with different labels

The notes also write the loss as:

This is binary cross-entropy expressed through the score $z$, using labels $-1$ and $+1$. Our earlier BCE formula uses the probability $p=\sigma(z)$ and labels $y\in\{0,1\}$. The conversion is $t=2y-1$: a positive label becomes $+1$, and a negative label becomes $-1$.

To see why they match, substitute $p=1/(1+e^{-z})$ and $1-p=1/(1+e^z)$. Taking the negative logarithm of either fraction gives the logarithm of its denominator—the expression in the last column.

The sign of $t$ selects the appropriate case inside $\log(1+e^{-tz})$. Use the signed label $t$ in that formula; use the 0/1 label $y$ in the earlier BCE formula.

Using 0/1 labels, a stable implementation can calculate $\log(1+e^z)-yz$ with a log-sum-exp operation. This avoids first rounding the sigmoid to 0 or 1 and then taking its logarithm.

Why the gradient becomes prediction minus label

The sigmoid derivative is $dp/dz=p(1-p)$. The loss derivative with respect to $p$ is $-y/p+(1-y)/(1-p)$. Multiplying these two derivatives cancels the denominators:

Since $z=b+wx$, the intercept gradient is $p-y$ and the slope gradient is $(p-y)x$. Average across our four rows:

The mean intercept gradient is 0; the mean slope gradient is about −0.2312. At learning rate 0.5, the new intercept remains −1.5 and the slope becomes $1-0.5(-0.2312)\approx1.1156$. The mean loss drops to about 0.3150.

import numpy as np

x = np.array([0., 1., 2., 3.])
y = np.array([0., 0., 1., 1.])
b, w = -1.5, 1.0
z = b + w*x
p = 1 / (1 + np.exp(-z))
db = np.mean(p - y)
dw = np.mean((p - y) * x)
b, w = b - 0.5*db, w - 0.5*dw
new_z = b + w*x
loss = np.mean(np.logaddexp(0, new_z) - y*new_z)
print(b, w, loss)  # about -1.5, 1.1156, 0.3150

For a design matrix $X$ with $n$ rows and $d$ columns, the average-loss gradient is $X^\top(p-y)/n$. Its shape is $d$, matching the coefficient vector. An all-ones column incorporates the intercept into the same expression.

A convex objective still needs a finite solution

The Hessian is the matrix of second derivatives: it describes how the gradient changes as we change the coefficients. Let $\beta$ contain all coefficients, including the intercept; in our one-input model, $\beta=(b,w)^\top$. Use an $n\times d$ design matrix $X$ with an all-ones column for that intercept, so $z=X\beta$. Write the average loss as $J=\frac1n\sum_i\ell_i$.

Differentiate one gradient component

For coefficient $\beta_j$, the gradient derived above is:

Now differentiate with respect to coefficient $\beta_k$. The training inputs $X_{ij}$ and labels $y_i$ are fixed; only $p_i$ changes. Since $p_i=\sigma(z_i)$ and $z_i=\sum_jX_{ij}\beta_j$, the chain rule gives:

The sigmoid supplies $p_i(1-p_i)$, and differentiating the linear score supplies $X_{ik}$. Substituting into the gradient gives entry $(j,k)$ of the Hessian:

Collect those entries into a matrix

Put the sigmoid derivatives on the diagonal of an $n\times n$ matrix $W$:

Multiplication by $W$ scales row $i$ of $X$ by $p_i(1-p_i)$. Multiplying on the left by $X^\top$ then produces exactly the sum in $H_{jk}$. Therefore:

The result has shape $d\times d$, one row and column per coefficient. $W$ contains values computed from the current predictions; its entries change when $\beta$ changes.

Why this establishes convexity

For any coefficient direction $v$, substitute $H=X^\top WX/n$ into the curvature:

Here $(Xv)_i$ is entry $i$ of the vector $Xv$. Because $W$ is diagonal, each entry is multiplied by its weight $W_{ii}$ and by itself, giving $W_{ii}(Xv)_i^2$. The probabilities enter through that weight: $W_{ii}=p_i(1-p_i)$.

Since $0\le p_i\le1$, every term is a nonnegative weight times a square. This makes the Hessian positive semidefinite, so the loss is convex in the coefficients. Convexity still leaves one question: is the best value reached at finite coefficients?

Our toy data are perfectly separable: a threshold at 1.5 hours classifies every row correctly. Scaling both the slope and intercept of $x-1.5$ upward makes the probabilities more extreme and drives the unregularized loss toward zero. No finite coefficient vector reaches that limiting optimum. Convexity alone does not guarantee a finite MLE.

Use a prior to moderate the coefficients

A zero-mean Gaussian prior on the slopes produces an L2 penalty under MAP. If the prior variance is $\tau^2$, the summed negative log posterior is:

Dividing the whole expression by $n$ gives average loss plus $(\lambda/2)\lVert w\rVert^2$, with $\lambda=1/(n\tau^2)$. The slope gradient gains $\lambda w$. We leave the intercept unpenalized. With both classes present, this penalty prevents the separating slopes in our example from growing without bound.

Library regularization parameters may use different loss scalings. Read the objective definition before treating a parameter value as the same $\lambda$ used here.

Turn a probability into a decision

At threshold 0.5, predict the positive class when $b+w^\top x\ge0$. This is a hyperplane in the model's feature space. In our starting one-feature example, the boundary is $x=1.5$.

At a general probability threshold $q$, the score threshold becomes $\log(q/(1-q))$. Using $q=0.8$ with the starting coefficients requires $x\ge1.5+\log4\approx2.886$. The underlying probabilities stay the same; the decision requirement changes.

We have now connected a score, its probability, an observed-label likelihood, and a training update. The clustering guide changes the task: we will remove the labels and group points by distance.

From the handwritten notes

This guide expands Logistic Regression.pdf with worked tables and intermediate steps. The PDF contains all the original pages.

View original page 3

Further reading:

• Stanford CS229: logistic regression and regularization