The Imbalance Problem

September 2026 · 12 min read

A classifier can be 99% accurate while missing every case you care about. Suppose only 10 of 1,000 items are defective. Predicting “not defective” for every item gets the other 990 right, but catches no defects.

Class imbalance means the class counts differ. It becomes a modeling problem when the data, training objective or evaluation hides mistakes that matter. The task is to learn useful distinctions and make useful decisions; equal class counts are not themselves the goal.

Look at the counts before the percentage

Call a defective item positive and an ordinary item negative. These names identify the class we are trying to detect; they do not mean good and bad. In this dataset the positive class is the minority, and its prevalence, or share of observations, is 10 / 1,000 = 1%.

Accuracy counts the fraction of predictions that are correct:

The always-negative classifier gets 990 / 1,000 = 99%. Keep this baseline in the comparison: a large accuracy percentage alone is not evidence of useful detection.

Separate the two kinds of mistakes

A false negative is a missed positive: a defective item called ordinary. A false positive is a false alarm: an ordinary item called defective. Correctly detected positives are true positives; correctly dismissed negatives are true negatives.

Compare the baseline with an illustrative detector that catches 8 of the 10 defects and raises 20 false alarms. These are fixed example outcomes, not a claim about a trained model.

Recall asks how many actual positives we found: 8 / 10 = 80%. Precision asks how many alerts were true positives: 8 / (8 + 20) ≈ 28.6%. The detector has lower accuracy, 97.8%, but it performs the detection task the baseline ignores.

That does not automatically make it good enough. Most of its alerts are false alarms. If review is expensive or capacity is limited, that matters. When there are no alerts, precision has a zero denominator and is undefined; libraries may report a chosen fallback value. State that convention rather than interpreting it as measured alert quality.

Decide which mistakes matter for this task

Suppose a missed defect costs 20 evaluation units and a false alarm costs 1. The baseline costs 10 × 20 = 200 units. The detector costs 2 × 20 + 20 × 1 = 60. Under those stated costs, its lower accuracy is an acceptable tradeoff.

If each miss and false alarm instead costs one unit, the baseline makes 10 mistakes and the detector makes 22. The preferred rule changes with the objective. Class counts alone cannot supply those costs.

You might instead require a minimum recall, compare precision at a fixed review budget, or report each class’s recall with balanced accuracy. The evaluation chapter explains how these measurements differ. Choose what success means before choosing a balancing method.

Why the model can struggle, and what imbalance does not prove

With equal per-row loss weights, 990 negative training examples contribute many more terms than 10 positive examples. Reducing mistakes on the majority may therefore dominate the training objective. The default classification threshold can also yield few positive predictions even when the score ranks positives usefully.

But rare does not mean impossible to learn. If a reliable feature cleanly separates the classes, even an imbalanced dataset can be classified well. Conversely, ten positive examples may be too few to represent several positive subtypes, regardless of how often we copy them.

Inspect the minority examples, their labeling quality, overlap with other classes and coverage of the situations expected in use. More representative independent examples can supply information that resampling existing rows cannot. Also, a minority class is not automatically an outlier population: class labels and geometric unusualness answer different questions.

Three changes that act at different places

See random oversampling, random undersampling, class weights and threshold moving for their separate mechanisms. Synthetic methods such as SMOTE add assumptions about what valid new feature vectors look like.

Preserve the population you need to evaluate

Split original observations before resampling. Inside cross-validation, resample only each training fold; keep validation observations independent and representative of the intended workload. Otherwise copies or synthetic neighbors can leak information across the boundary, and a balanced test set can hide the alert volume seen under natural prevalence.

For independent observations, stratified splitting helps preserve class proportions. It does not create more rare events, and group or time constraints still take priority. Always inspect the actual positive count in each evaluation partition.

References and further reading

• Imbalanced-learn: problem statement — how unequal class representation interacts with the learned decision function.
• Scikit-learn: classification metrics — confusion counts, precision, recall and class-aware evaluation.
• Scikit-learn: decision thresholds — separating score prediction from the action selected using that score.
• Imbalanced-learn: resampling leakage — why sampling belongs inside the training boundary.