Machine Learning · Loss Functions · Entropy · Part 5

Maximum Entropy and Calibration

July 2026 · 16 min read

Maximum entropy is a principle for choosing a probability distribution when only partial information is known. Among all distributions satisfying the known constraints, choose the one with the highest entropy. It is the least committed distribution consistent with the evidence.

The Principle

With no constraints except "there are $K$ outcomes," the maximum-entropy distribution is uniform. With a known mean, the maximum-entropy distribution tilts just enough to match that mean and no more. The principle avoids inventing certainty that the evidence does not justify.

▶ Constraints Lower Maximum Entropy

Start from a uniform distribution. Each step adds information, so the allowed maximum-entropy distribution becomes less spread out.

Softmax Intuition

Softmax turns logits into probabilities:

$$p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}.$$

One way to read softmax is as an entropy-regularized choice. Larger logits push probability toward preferred classes, while entropy keeps the distribution spread unless the logit evidence is strong. Temperature scaling makes this explicit:

$$p_i(T)=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}.$$

High temperature produces higher-entropy probabilities. Low temperature produces sharper, lower-entropy probabilities.

Label Smoothing

Label smoothing replaces a one-hot target with a mixture of the one-hot label and a small uniform distribution. For $K$ classes and smoothing $\epsilon$, the correct class receives $1-\epsilon+\epsilon/K$, while every class receives at least $\epsilon/K$.

This discourages infinite confidence. It can improve calibration because the training target no longer says that every non-target class is literally impossible.

Calibration

A classifier is calibrated if examples predicted with probability $0.8$ are correct about $80\%$ of the time. Cross-entropy encourages probability quality, but modern high-capacity models can still become overconfident. Entropy-related tools help diagnose and correct this:

Prediction entropy flags uncertain examples.
Temperature scaling adjusts confidence without changing class rankings.
Label smoothing reduces overconfident training targets.
KL divergence compares predicted distributions with reference distributions.

Entropy is not indecision for its own sake. Maximum entropy says to remain uncertain only where the evidence does not justify certainty. Strong evidence should still produce sharp predictions.

Series Wrap-Up

The entropy sub-series started with surprise, specialized to binary entropy, extended to cross-entropy and KL divergence, applied entropy reduction to decision trees, and ended with maximum-entropy thinking in probabilistic models. These ideas reappear across popular loss functions because losses are often ways to compare distributions, not just numbers to minimize.