Class Weights
Class weighting changes how much each training example contributes to the loss. A rare-class example can receive more influence without being copied or replaced. The labels and original observations stay the same.
Use the earlier population of 990 negatives and 10 positives. We will calculate balanced weights, inspect their total influence, then show why a weighted classifier’s output is not automatically a probability for the original population.
Multiply each loss before averaging
Suppose example $i$ has label $y_i$ and incurs loss $\ell_i$. The loss measures how poorly the current prediction fits that example. Assign a positive class weight $w_{y_i}$, so every example of the same class receives the same multiplier.
For this chapter, define the weighted objective as the weighted sum of losses divided by the total weight. With $n$ training examples:
A positive example with weight 10 contributes ten times as much as one with weight 1 if their unweighted losses are equal. The actual influence also depends on the prediction error and model. Total class weight alone is not the class’s total loss.
We use this normalization to compare weighted averages. Libraries may normalize their data loss differently or combine it with regularization differently. Multiplying every weight by a common constant leaves the average above unchanged, but it need not leave every library’s regularized fit unchanged at fixed hyperparameters.
What “balanced” weights actually balance
A common heuristic gives every represented class the same total weight. Let $K$ be the number of classes and $n_c$ the number of training examples in class $c$. Without additional sample weights, scikit-learn’s balanced rule is
Multiplying by the class count gives $n_cw_c=n/K$, so every class receives the same total weight. For two classes with counts 990 and 10:
| Class | Count | Weight per example | Total weight |
|---|---|---|---|
| Negative | 990 | 1,000 / (2 × 990) ≈ 0.5051 | 500 |
| Positive | 10 | 1,000 / (2 × 10) = 50 | 500 |
The positive-to-negative weight ratio is 99. It balances total weight, not the number of observations, the number of errors or the information available about each class. A rare positive subtype still has only the examples originally collected.
Change the relative weight and inspect the consequence
The figure below fixes each negative weight at 1 and varies the positive weight. At positive weight 99, both classes have total weight 990. This has the same relative weighting as the balanced rule above for our normalized objective.