Confusion Matrix, ROC-AUC, PR-AUC, and Log-Loss
No single metric describes every part of a classifier. Evaluation has three separate questions:
- Ranking: are positive rows usually scored above negative rows?
- Decision: at one threshold, how many positives are found and how many false alarms occur?
- Probability: do values such as $0.2$ and $0.8$ behave like real probabilities?
ROC-AUC and average precision measure ranking. Confusion-matrix rates measure one threshold. Log-loss and the Brier score measure probabilities. A model can perform well in one family and poorly in another.
The Confusion Matrix and Its Rates
At a fixed threshold, every row belongs to one of four groups:
| Count | Meaning |
|---|---|
| True positive (TP) | Predicted positive, actually positive |
| False positive (FP) | Predicted positive, actually negative |
| False negative (FN) | Predicted negative, actually positive |
| True negative (TN) | Predicted negative, actually negative |
Precision starts from the predicted positives and asks what fraction were correct. Recall starts from the actual positives and asks what fraction were found. Recall is also called sensitivity or the true-positive rate (TPR).
$$\text{precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\qquad \text{recall}=\text{TPR}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}.$$Specificity, or the true-negative rate (TNR), is the fraction of actual negatives rejected correctly. The false-positive rate (FPR) is the remaining fraction of negatives.
$$\text{TNR}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}},\qquad \text{FPR}=\frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}}=1-\text{TNR}.$$Common one-number summaries combine these counts:
$$\text{accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{n},\qquad \text{balanced accuracy}=\frac{\mathrm{TPR}+\mathrm{TNR}}{2},$$ $$F_1=\frac{2\,\mathrm{Prec}\,\mathrm{Rec}}{\mathrm{Prec}+\mathrm{Rec}},\qquad F_\beta=(1+\beta^2)\frac{\mathrm{Prec}\,\mathrm{Rec}}{\beta^2\mathrm{Prec}+\mathrm{Rec}},$$ $$\mathrm{MCC}=\frac{\mathrm{TP}\cdot\mathrm{TN}-\mathrm{FP}\cdot\mathrm{FN}}{\sqrt{(\mathrm{TP}{+}\mathrm{FP})(\mathrm{TP}{+}\mathrm{FN})(\mathrm{TN}{+}\mathrm{FP})(\mathrm{TN}{+}\mathrm{FN})}}.$$- Balanced accuracy gives the positive and negative classes equal weight, regardless of their sizes.
- $F_1$ is high only when both precision and recall are high. $F_\beta$ places more emphasis on recall when $\beta>1$ and more on precision when $\beta<1$.
- Matthews correlation coefficient (MCC) uses all four cells and ranges from $-1$ to $1$. Its denominator is zero for some constant predictions, so that case needs an explicit software convention.
All these values depend on the threshold. They describe one operating point, not every possible use of the score.
Why Accuracy Misleads Under Imbalance
When $99\%$ of rows are negative, a model that predicts negative for every row gets $99\%$ accuracy and $0\%$ recall. The high accuracy comes entirely from the majority class.
Recall and specificity expose the two classes separately. Balanced accuracy gives this model $(0+1)/2=0.5$. The MCC denominator is zero because the prediction never changes, which marks the classifier as degenerate rather than successful.
The ROC Curve and the AUC
The receiver operating characteristic (ROC) curve plots TPR against FPR while the threshold moves from strict to loose. It begins at $(0,0)$ when nothing is predicted positive and ends at $(1,1)$ when everything is predicted positive. Admitting a positive row moves the curve upward; admitting a negative row moves it right.
The area under this curve is ROC-AUC. A random ranking has expected AUC $0.5$, and a perfect ranking has AUC $1$. Its most useful meaning is pairwise:
$$\mathrm{AUC}=\Pr\big(s_i>s_j\mid y_i=1,\,y_j=0\big)=\frac{1}{PN}\sum_{i:y_i=1}\ \sum_{j:y_j=0}\Big[\mathbf{1}(s_i>s_j)+\tfrac12\mathbf{1}(s_i=s_j)\Big],$$Here, $P$ and $N$ are the numbers of positive and negative rows. ROC-AUC is the probability that a random positive receives a higher score than a random negative, with a tie receiving half credit.
Only order matters. Any strictly increasing transformation preserves ROC-AUC even if it destroys probability calibration. ROC-AUC is also unchanged by a pure change in class prevalence when the score distributions within each class stay the same. It measures ranking, not the number of false alarms produced at a chosen threshold.
▶ Building the ROC Staircase and Accumulating the AUC
Ten scored examples, five positive and five negative, sorted by score. Each step lowers the threshold to admit the next example, stepping up for a positive and right for a negative, and adds the trapezoid to the area. Values are computed at runtime.
The example has five positives and five negatives, giving $25$ positive-negative pairs. The positive receives the higher score in $21$ pairs, so $\mathrm{AUC}=21/25=0.84$. The same value appears as the geometric area under the staircase.
The Precision-Recall Curve and Average Precision
FPR divides false positives by all actual negatives. When negatives are abundant, a small FPR can still represent many false alarms. For example, $100$ false positives among $10{,}000$ negatives gives $\mathrm{FPR}=1\%$. If only $50$ true positives were found, precision is just $50/(50+100)=33.3\%$.
The precision-recall (PR) curve makes this visible by plotting precision against recall. Average precision (AP) summarizes the ranked list by adding precision whenever recall increases:
$$\mathrm{AP}=\sum_{k}\big(R_k-R_{k-1}\big)\,P_k,$$A random ranking has expected precision equal to the positive prevalence. With $1\%$ positives, its baseline AP is near $0.01$, not $0.5$. This dependence on prevalence is why PR results should be compared only on datasets with the same class balance.
The PR curve can move down when a false positive is admitted and up when a true positive is admitted, creating a sawtooth shape. Average precision uses the stepwise sum above; trapezoidal PR-AUC uses a different interpolation and can give a different number. In the animation, the prevalence baseline is $0.5$ and AP is $0.853$.
▶ Sweeping Thresholds Through the Precision-Recall Curve
The same ten scores are admitted from highest to lowest. Each step places the threshold at the next score, recomputes the confusion matrix, plots the resulting precision-recall point, and adds an average-precision rectangle only when recall increases.
Probabilistic Metrics: Log-Loss and the Brier Score
ROC-AUC sees only order, and confusion metrics see only the final class. Neither checks whether $0.8$ behaves like an $80\%$ chance. Log-loss and the Brier score use the probability values directly:
$$\text{log-loss}=-\frac{1}{n}\sum_{i=1}^{n}\big[y_i\ln p_i+(1-y_i)\ln(1-p_i)\big],\qquad \text{Brier}=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2.$$Both are strictly proper scoring rules: in expectation, the best report is the true probability rather than a strategically inflated or softened value. Lower scores are better.
- Log-loss has no upper bound. Assigning probability near $0$ to an event that occurs creates a very large penalty.
- Brier score is squared probability error and stays between $0$ and $1$ for binary outcomes. It is less dominated by one confidently wrong prediction.
A model can have excellent ROC-AUC but poor log-loss if its ordering is good and its probabilities are distorted. That pattern points toward a calibration problem.
Sampling Uncertainty and Confidence Intervals
Every test metric is uncertain because the test set is finite. AUC intervals commonly use DeLong's method or a bootstrap. Precision, recall, and specificity can use Wilson, exact, or bootstrap intervals; the simple Wald interval is unreliable near $0$, near $1$, or with small counts.
The resampling unit must match the independent unit. Repeated rows from one patient, account, or time period are resampled together rather than as independent rows. Two models evaluated on the same cases are also resampled as a pair.
A confidence interval does not undo test-set tuning. If the test set chooses the threshold or winning model, the reported interval is too optimistic. Rare-event uncertainty is driven mainly by the number of positives, so even a large test set can give a wide recall or AP interval when positives are scarce.
Three Lenses and When Each Misleads
The required metrics follow from the intended use. Many evaluations need at least one view from each family:
| Lens | Examples | Sees | Blind to |
|---|---|---|---|
| Ranking | ROC-AUC, AP, PR-AUC | The ordering of positive and negative scores | Probability calibration and the chosen operating threshold |
| Thresholded | Precision, recall, specificity, $F_1$, MCC, accuracy | The realized errors at one chosen cut | Behavior at every other threshold |
| Probabilistic | Log-loss, Brier score | The quality of the probability values | Whether one chosen decision rule meets its operational target |
Accuracy alone can hide minority failure. AUC alone can hide miscalibration. One thresholded metric alone can hide behavior at other cuts.
For multiclass problems, macro averaging computes a metric per class and gives every class equal weight. Micro averaging pools all class decisions first, so frequent classes dominate. The averaging rule must accompany the reported value because macro and micro results can differ sharply.
Edge Cases and the Implementation
- Tied scores. ROC-AUC gives half credit when a positive and negative have equal scores.
- Missing class. ROC and PR metrics are undefined when a test split has no positives or no negatives.
- PR summaries. Average precision and trapezoidal PR-AUC use different interpolation rules and need not match.
- Unstated threshold. Precision, recall, specificity, $F_1$, MCC, and accuracy are incomplete without the threshold that produced them.
The C++ metrics section implements the confusion matrix, rank-sum AUC, ranking curves, and probabilistic scores. A dedicated evaluation and calibration chapter is also planned.