Jensen-Shannon Divergence Foundations
Jensen–Shannon (JS) divergence compares two distributions through a shared middle distribution. Instead of comparing $P$ directly with $Q$, first average their probabilities. For each outcome $i$:
$$m_i=\frac{p_i+q_i}{2},\qquad M=\tfrac12(P+Q).$$This is a mixture: imagine flipping a fair coin to choose whether to sample from $P$ or $Q$. The overall outcome distribution is $M$. Now measure each distribution's KL divergence to that mixture and average the two costs:
$$JS(P,Q)=\tfrac12D_{KL}(P\|M)+\tfrac12D_{KL}(Q\|M).$$We use this equal-weight definition and base-2 logs throughout, so JS is measured in bits. All distributions below describe the same outcomes, A, B, and C.
See How the Shared Mixture Works
Each bar shows one outcome's probability on a 0-to-1 scale. The blue $M$ bar is halfway between the green $P$ and orange $Q$ probabilities for that outcome.
P
First distribution
M
Average of P and Q
Q
Second distribution
For example, the initial probability of A in $M$ is $(0.70+0.10)/2=0.40$. Each panel still sums to 1.
The distributions overlap, but differ. Both KL costs are positive, and their average is between 0 and 1 bit.
Try “Identical” to see JS become 0, then “No overlap” to reach 1 bit. Swapping P and Q exchanges the two KL costs while preserving their average.
JS Properties for Interviews
| Question | Answer |
|---|---|
| Positive? | Nonnegative: $JS\geq0$, because it averages two nonnegative KL divergences. |
| When zero? | Exactly when $P=Q$. Then the mixture equals both distributions, and both KL terms are zero. |
| Symmetric? | Yes: $JS(P,Q)=JS(Q,P)$. Swapping the inputs leaves $M$ unchanged and exchanges the two terms in the average. |
| Upper limit? | At most 1 bit with base-2 logs. With natural logs, the bound is $\ln2$ nats. These bounds use the equal weights $\tfrac12,\tfrac12$ above. |
| When maximal? | When the supports are disjoint: no outcome has positive probability in both distributions. |
| Ever infinite? | No. The mixture has positive probability wherever either input does. If both assign zero to an outcome, its terms contribute zero. |
| A metric? | JS divergence itself is not a metric. Its square root, $\sqrt{JS(P,Q)}$, is the Jensen–Shannon distance and satisfies the triangle inequality. |
The divergence and its bounds come from Lin's original paper. The square-root metric result is established by Endres and Schindelin.
Why the Mixture Prevents Infinity and Caps JS
For an outcome that can occur under $P$, the mixture includes at least half of its $P$ probability. Thus the denominator can never be zero, and the ratio can never exceed 2:
$$m_i=\frac{p_i+q_i}{2}\geq\frac{p_i}{2} \quad\Longrightarrow\quad\frac{p_i}{m_i}\leq2\qquad(p_i>0).$$Taking base-2 logs gives a value at most 1. Averaging these values with weights $p_i$, which sum to 1, gives $D_{KL}(P\|M)\leq1$. The same reasoning applies to $Q$. Each KL is also nonnegative, so their average lies between 0 and 1 bit.
The “No overlap” figure uses $P=(1,0,0)$ and $Q=(0,0,1)$. Their mixture is $M=(0.5,0,0.5)$: A and C remain possible. Each side pays $\log_2(1/0.5)=1$ bit to the mixture, so JS is 1 bit. Direct KL between $P$ and $Q$ would instead be infinite.
The Same Quantity Written with Entropy
Substitute $D_{KL}(P\|M)=H(P,M)-H(P)$, and similarly for $Q$. The two cross-entropy terms average to $H(M)$ because their probability weights average to $m_i$. Therefore:
$$JS(P,Q)=H(M)-\tfrac12H(P)-\tfrac12H(Q).$$In words: uncertainty in the mixed distribution minus the average uncertainty within the two original distributions.
Why Say “Distance” Only After Taking the Square Root?
A metric must satisfy the triangle inequality: the direct distance cannot exceed the sum of distances through a third distribution. JS itself can violate this. Take $P=(1,0)$, $Q=(0.5,0.5)$, and $R=(0,1)$:
$$JS(P,R)=1>JS(P,Q)+JS(Q,R)\approx0.623.$$The square-root version satisfies that inequality for all distributions; symmetry alone was not enough to make JS a metric.