Machine Learning · Loss Functions · Entropy · Part 4

Entropy and Information Gain

July 2026 · 17 min read

Decision trees use entropy as an impurity measure. A node is impure when its label distribution is uncertain. A split is useful when it sends rows into children whose label distributions are less uncertain than the parent. The reduction is information gain.

Gain Formula

For a parent node with $N$ rows and children $c$, information gain is

$$\text{Gain} = H(\text{parent}) - \sum_c \frac{N_c}{N}H(\text{child}_c).$$

The child entropies are weighted by child size. A tiny pure child should not dominate the score if most rows remain mixed in the other child.

▶ Parent Entropy minus Weighted Child Entropy

Move through candidate thresholds. The best split is the one that produces the largest entropy reduction after weighting the two children by size.

Pure Children

If a split creates pure children, the weighted child entropy is zero and the gain equals the parent entropy. If the children have the same class proportions as the parent, the weighted child entropy equals the parent entropy and the gain is zero.

Many-Valued Split Bias

Plain information gain can favor features with many possible split outcomes. Such features can fragment data into tiny children that look pure by accident. C4.5 reduces this bias with gain ratio, which divides information gain by the intrinsic information of the split.

Entropy gives a local split score. The split with the largest gain is locally best at the current node. A greedy tree still does not guarantee the globally smallest or most accurate tree.

Connection to the Decision Tree Series

This post isolates the entropy part of decision-tree splitting. The full tree-building story, including Gini impurity and sorted threshold sweeps, appears in Decision Trees: Impurity and Information Gain.