Softmax: Turn Competing Scores into Probabilities
Most activations in this series process each scalar independently. Softmax processes a vector together. Increasing one score can reduce another class's probability because the classes share a denominator.
Normalize positive weights across classes
Let $z_1,\ldots,z_K$ be the logits, or unrestricted scores, of $K$ mutually exclusive classes. Softmax exponentiates each score and divides by the sum:
The exponential makes each weight positive. The denominator sums those weights, so the outputs add to one. A logit is not itself a probability, and a probability vector alone does not establish calibration.
| Class | Shifted weight | Probability |
|---|---|---|
| 1 | $e^0=1$ | 0.6652 |
| 2 | $e^{-1}\approx0.3679$ | 0.2447 |
| 3 | $e^{-2}\approx0.1353$ | 0.0900 |
Why subtracting the largest score is safe
Adding the same number to all logits multiplies both the numerator and denominator by the same factor, which cancels. Use that freedom to subtract $M=\max_j z_j$:
Every exponent is now nonpositive and at least one exponential is 1. This avoids overflow for large finite logits. Some extremely small weights can still underflow to zero; compute log probabilities directly when a loss needs them.
The stable log probability is $z_i-\operatorname{logsumexp}(z)$, where logsumexp means the logarithm of the sum of exponentials. A stable logsumexp implementation performs the same shift internally.
The derivative connects different classes
Because every output $p_i$ shares the denominator $S=\sum_{k=1}^{K}e^{z_k}$, changing a single logit $z_j$ affects every class probability $p_1,\ldots,p_K$. Softmax therefore has a $K\times K$ Jacobian matrix of partial derivatives:
Here $\delta_{ij}$ is the Kronecker delta ($\delta_{ij}=1$ when $i=j$, and $\delta_{ij}=0$ when $i\neq j$). Let's derive this formula step by step from the definition of $p_i$.
Step 1: Set up the numerator and denominator derivatives
Write $p_i$ as a quotient with denominator $S$:
When we differentiate the sum $S$ with respect to a specific logit $z_j$, every term $e^{z_k}$ with $k\neq j$ is constant, leaving only the $k=j$ term:
Now apply the quotient rule $\frac{\partial}{\partial z_j}\!\left(\frac{u}{S}\right)=\frac{u'S-uS'}{S^2}$ to $u=e^{z_i}$. There are two cases depending on whether $j=i$ or $j\neq i$.
Step 2: Case 1 — same class ($j=i$)
When we differentiate $p_i$ with respect to its own logit $z_i$, the numerator derivative is $\frac{\partial}{\partial z_i}(e^{z_i})=e^{z_i}$ and the denominator derivative is $\frac{\partial S}{\partial z_i}=e^{z_i}$. The quotient rule gives
Factor out $\frac{e^{z_i}}{S}$ and substitute $p_i=\frac{e^{z_i}}{S}$:
Because $0<p_i<1$, this diagonal derivative is always positive: raising class $i$'s own logit always increases $p_i$.
Step 3: Case 2 — different class ($j\neq i$)
When $j\neq i$, the numerator $e^{z_i}$ does not contain $z_j$, so its derivative with respect to $z_j$ is $0$. Only the denominator $S$ changes ($\frac{\partial S}{\partial z_j}=e^{z_j}$):
Split the fraction into $\frac{e^{z_i}}{S}=p_i$ and $\frac{e^{z_j}}{S}=p_j$:
This off-diagonal derivative is always negative: raising another class $j$'s logit enlarges the shared denominator $S$ and pushes $p_i$ down. Using $\delta_{ij}=1$ for $i=j$ and $\delta_{ij}=0$ for $i\neq j$, both cases combine into $\frac{\partial p_i}{\partial z_j}=p_i(\delta_{ij}-p_j)$.
Temperature changes sharpness
For $T>0$, apply softmax to $z/T$. A smaller $T$ sharpens the probabilities; a larger $T$ makes them more even. Finite positive temperature preserves the ranking of logits. Temperature is a deliberate modeling or calibration choice, not automatically an accuracy improvement.
Advantages and disadvantages
| Aspect | Advantages | Disadvantages |
|---|---|---|
| Output distribution | Produces nonnegative weights that sum to one, fitting mutually exclusive classes or competition between allowed attention positions. | Couples the outputs, so independent labels need separate sigmoids. A normalized distribution is not necessarily a calibrated one. |
| Learning | With a one-hot target, cross-entropy gives the simple logit gradient $p_i-\mathbf1[i=y]$. | The softmax-only Jacobian becomes small near a one-hot output. Do not confuse that local slope with the combined loss gradient or apply softmax again before a logits-based loss. |
| Numerics | A common logit shift leaves the result unchanged, allowing the maximum-subtraction stabilization above. | Naive exponentials can overflow, and tiny weights may still underflow after shifting. Use log-softmax when you need log probabilities. |
Choose according to the prediction task
Use softmax across the class dimension when exactly one class is selected. Use separate sigmoids for labels that can coexist. A scalar regression output may need neither. In attention, softmax instead distributes weight over allowed positions; the normalized axis and mask define the competition.
The design chapter now brings the series together: choose requirements first, then construct and test a function.