Probability, Log-Odds, Decision Boundaries, and Interpretation
Logistic regression connects three scales: a linear score, positive odds, and a probability between $0$ and $1$. The score and probability rank rows in the same order, but the probability curve bends. The same feature change can therefore produce a large probability change near $0.5$ and a small change near $0$ or $1$.
The Sigmoid, the Logit, and the Odds
The sigmoid maps score $z$ to probability $p$, and the logit maps probability back to score:
$$p=\sigma(z)=\frac{1}{1+e^{-z}},\qquad z=\operatorname{logit}(p)=\ln\frac{p}{1-p}.$$Odds are $p/(1-p)$. Odds of $3$ correspond to probability $0.75$. Log-odds are $\log(p/(1-p))$, which can take any real value. The sigmoid derivative is $p(1-p)$, largest at $p=0.5$ and close to zero in the tails.
| log-odds $z$ | odds $e^{z}$ | probability $\sigma(z)$ |
|---|---|---|
| $-2$ | $0.135$ | $0.119$ |
| $-1$ | $0.368$ | $0.269$ |
| $0$ | $1$ | $0.500$ |
| $1$ | $2.718$ | $0.731$ |
| $2$ | $7.389$ | $0.881$ |
Coefficients as Odds Ratios, and Why Units Matter
A one-unit increase in $x_j$ adds $\beta_j$ to the log-odds and multiplies the odds by $e^{\beta_j}$. This multiplier is the odds ratio:
$$\frac{\operatorname{odds}(x_j+1)}{\operatorname{odds}(x_j)}=\frac{e^{\beta_0+\cdots+\beta_j(x_j+1)+\cdots}}{e^{\beta_0+\cdots+\beta_j x_j+\cdots}}=e^{\beta_j}.$$The odds ratio is constant, but the probability increase is not. In the study-hours example, one extra hour multiplies odds by $e^{2.5646}=12.9953$. Probability rises from $0.3450$ at two hours to $0.8725$ at three hours.
An odds ratio is not a risk ratio except approximately for rare outcomes. “Twice the odds” should not be reported as “twice the probability.”
Marginal Effects on the Probability Scale
The probability-scale effect is the derivative
$$\frac{\partial p}{\partial x_j}=\beta_j\,\sigma(z)\big(1-\sigma(z)\big)=\beta_j\,p(1-p).$$This effect depends on $p$. It is largest at $p=0.5$ and small near $0$ or $1$. The study-hours model has effect $0.6411$ per hour at the boundary but only $0.0282$ near a confident failure prediction.
The effect at the mean evaluates one synthetic average row. The average marginal effect averages the derivative over real rows and is often more representative.
Decision Boundaries Are Threshold-Specific
A threshold $t$ predicts class $1$ when $p\ge t$. Because the sigmoid is increasing, this is equivalent to $z\ge\operatorname{logit}(t)$. The boundary is
$$\beta^\top\tilde{x}=\operatorname{logit}(t)=\ln\frac{t}{1-t}.$$Changing the threshold shifts the boundary but does not change probabilities. In the study model, thresholds $0.25$, $0.50$, and $0.75$ move the boundary to $1.8216$, $2.2500$, and $2.6784$ hours.
| threshold $t$ | $\operatorname{logit}(t)$ | boundary $x$ (hours) |
|---|---|---|
| $0.25$ | $-1.0986$ | $1.8216$ |
| $0.50$ | $0.0000$ | $2.2500$ |
| $0.75$ | $1.0986$ | $2.6784$ |
With two features, the boundary is a straight line. Curved boundaries require transformed features or another model.
Log-Odds, Odds, and Probability in Motion
The animation shows the same fitted model on the log-odds, odds, and probability scales. Equal feature steps produce equal log-odds changes and constant odds multipliers, but unequal probability changes.
▶ Score, Odds, and Probability
The score is a line, the odds change multiplicatively, and the probability follows an S-curve.
Interactions Multiply Odds Ratios
An interaction lets the odds ratio for one feature depend on another. In $z=\beta_0+\beta_1x_1+\beta_2x_2+\beta_{12}x_1x_2$, the odds ratio for $x_1$ is $e^{\beta_1}$ when $x_2=0$ and $e^{\beta_1+\beta_{12}}$ when $x_2=1$.
With $\beta_1=0.8$ and $\beta_{12}=0.6$, the odds ratio changes from $2.2255$ to $4.0552$. Interactions are simple on the log-odds scale but not additive on the probability scale.
Non-Collapsibility and Causal Cautions
Odds ratios are non-collapsible: a conditional odds ratio can differ from the marginal odds ratio even when the added variable is not a confounder. This is a mathematical property of odds, not proof of bias.
For example, both strata below have odds ratio $4$:
$$\text{OR}_{Z=0}=\frac{0.5/0.5}{0.2/0.8}=4,\qquad \text{OR}_{Z=1}=\frac{0.8/0.2}{0.5/0.5}=4.$$After averaging over $Z$, the marginal odds ratio is $3.4490$, not $4$. Adding a genuine outcome predictor can therefore change a logistic coefficient even when it is unrelated to the exposure.
Interpretation Checklist and Edge Cases
- Scale. Log-odds are additive and unbounded, odds are multiplicative, probabilities are bounded. A coefficient is linear only in log-odds.
- Unit. An odds ratio is per one unit of its predictor; rescaling the predictor changes the number without changing the model.
- Odds ratio versus risk ratio. These quantities are not equal except as an approximation when the outcome is rare in every relevant subgroup.
- Marginal-effect summary. The effect at the mean overstates the typical sensitivity when the sample spreads into the tails; the average marginal effect is usually preferable.
- Separation is invisible here. If a predictor perfectly orders the outcome, its coefficient and odds ratio diverge and any finite reported value is an artifact of the stopping rule, a failure treated on the regularization page.
The C++ logistic overview implements the stable sigmoid and classifier contract. The C++ sigmoid and gradient guide develops the first complete fit.