Machine Learning · Loss Functions · Entropy · Part 3

Cross-Entropy and KL Divergence

July 2026 · 18 min read

Entropy measures the average surprise under the true distribution $P$. Cross-entropy measures the average code length when data still comes from $P$, but the code is designed for another distribution $Q$:

$$H(P,Q) = -\sum_i p_i\log_2 q_i.$$

If $Q$ matches $P$, cross-entropy equals entropy. If $Q$ is wrong, the code wastes bits. That waste is KL divergence:

$$H(P,Q) = H(P) + D_{KL}(P\|Q).$$

Coding Interpretation

A model that predicts $Q$ is implicitly assigning shorter codes to outcomes it thinks are likely. If the true distribution is $P$, then cross-entropy is the average code length actually paid. The best possible average length is $H(P)$, and the extra cost is nonnegative.

▶ Cross-Entropy as Extra Code Length

The true distribution stays fixed. Step through three model distributions. When $Q$ approaches $P$, cross-entropy falls and KL waste shrinks.

Classification Loss

For multiclass classification, the target distribution is often one-hot. If the correct class is $y$, then $p_y=1$ and all other $p_i=0$. Cross-entropy reduces to

$$L = -\log q_y.$$

This is why a classifier is punished sharply for assigning low probability to the correct class. It is not enough for the correct class to be the argmax. The probability should be high if the evidence is strong.

Soft Labels

When labels are soft, cross-entropy uses the whole target distribution. This appears in label smoothing, distillation, weak supervision, and annotator-disagreement modeling. A target like $(0.7,0.2,0.1)$ says the first class should receive most probability, but the other classes are not impossible.

cross_entropy(P, Q):
    total = 0
    for each class i:
        if P[i] > 0:
            total += -P[i] * log(Q[i])
    return total

KL is directional. $D_{KL}(P\|Q)$ is not the same as $D_{KL}(Q\|P)$. The first measures the cost of using a code designed for $Q$ when the data follows $P$.

Why Minimizing Cross-Entropy Works

For fixed data distribution $P$, the entropy term $H(P)$ does not depend on the model. Minimizing cross-entropy therefore minimizes $D_{KL}(P\|Q)$. In words, the model is trained to make its predicted distribution close to the empirical target distribution.