Neel Mishra
← All Posts
Deep Learning · Data Imbalance in Deep Learning · Class Imbalance and Resampling

Threshold Moving

September 2026 · 15 min read

A classifier’s score and its final action are separate objects. Threshold moving changes the action rule while leaving the fitted score model unchanged. It can reduce missed positives, limit false alarms or meet another operating objective.

This chapter uses the same ten ranked cases as the evaluation chapter. We first select a threshold from validation outcomes. Later, we derive a cost-based cutoff for scores that can be interpreted as probabilities.

State the rule, including equality

Let $s(x)$ be the score for an observation $x$, with higher scores indicating stronger evidence for the positive class. Choose a threshold $t$ and write $\hat y$ for the predicted label:

$$\hat y=\begin{cases}1,&s(x)\ge t,\\0,&s(x)<t.\end{cases}$$

We include a score exactly equal to the threshold. Other code may use a strict comparison; that difference matters when scores are tied. A threshold of 0.5 is a common default for probability predictions, but arbitrary decision scores have no universal cutoff at 0.5.

Lowering the threshold can only add alerts to this fixed score list. Recall cannot decrease, and the number of false positives cannot decrease. Precision need not change monotonically: the next group admitted might contain more positives or more negatives.

Choose against a stated validation objective

The validation scores, from highest to lowest, are [0.95, 0.90, 0.80, 0.70, 0.60, 0.55, 0.40, 0.30, 0.20, 0.10]. Their true labels are + − + − − + − − − −. These scores establish a ranking; we have not established that they are calibrated probabilities.

Assign cost 4 to a missed positive and cost 1 to a false alarm; correct decisions cost zero. At threshold 0.80, we detect two positives, miss one and raise one false alarm. Total cost is 4 × 1 + 1 × 1 = 5. At 0.55, we detect all three positives with three false alarms, costing 3.

Move the cutoff across a fixed ranking

A + is an actual positive; a − is an actual negative. Highlighted cases trigger an alert.

0.95 +Alert
0.90 −Alert
0.80 +Alert
0.70 −Pass
0.60 −Pass
0.55 +Pass
0.40 −Pass
0.30 −Pass
0.20 −Pass
0.10 −Pass
CURRENT OUTCOMES2 detected · 1 missed1 false alarm
CURRENT COST5 units4 × 1 miss + 1 false alarm
Lowest validation cost: 3 units at threshold 0.55.
False-alarm cost stays at 1. The search includes every distinct score and the no-alert decision. If costs tie, it keeps the highest candidate threshold, producing fewer alerts.

Changing the miss cost can change the preferred threshold. With miss cost 1, detecting only the highest-scoring case costs 2; accepting more alerts has no lower cost in this example. Lowering the threshold is a tradeoff, not an automatic improvement.

For a finite validation set, predictions change only when the threshold crosses a distinct score. Evaluate those score cutoffs and one cutoff above the maximum, which predicts everything negative. Under our ≥ rule, a cutoff at the minimum score already includes the all-positive decision.

For cost 4, any threshold greater than 0.40 and at most 0.55 gives the same predictions here. The displayed 0.55 is a representative of that interval, not a uniquely precise parameter. Tied scores enter as a group because the model cannot rank them internally.

If the objective is a capacity limit or a minimum recall, first reject thresholds that violate the constraint, then compare the feasible choices. Sometimes no threshold satisfies all requirements. Randomly breaking ties or allowing partial review is a separate policy that needs its own evaluation.

Tune on validation predictions, then freeze the rule

01Fit the score modelUse the training partition, including any training-only sampling, weights and preprocessing.
02Select using validationScore independent validation cases. Compare cutoffs under the chosen costs or constraints.
03Evaluate the frozen systemApply the selected model and threshold to the untouched test population. Report counts, costs and alert volume.
A threshold is a fitted decision parameter. The ten cases above illustrate the calculation; they are too small to establish a reliable deployment threshold.

Cross-validation can supply held-out predictions when data are limited. Scikit-learn’s TunedThresholdClassifierCV automates threshold tuning; its default objective is balanced accuracy, so set a scorer that matches your actual goal. With cv="prefit", the data used to choose the threshold must be separate from the data that trained the supplied classifier.

Changing the threshold does not change ROC AUC, AP or the underlying scores. It moves the operating decision along the existing curves. Refitting a model, changing class weights or recalibrating scores can change the score scale, so do not assume a threshold selected for one fitted system transfers unchanged to another.

When the score really is a probability

Now suppose $p$ estimates the probability that this case is positive in the intended population. Let $C_{\mathrm{FP}}$ be the cost of a false alarm and $C_{\mathrm{FN}}$ the cost of a miss, both positive, with zero cost for correct decisions.

Predicting positive is wrong only when the case is negative, with probability $1-p$. Predicting negative is wrong only when the case is positive, with probability $p$. Their expected costs are therefore

$$\begin{aligned}\text{cost of predicting positive}&=(1-p)C_{\mathrm{FP}},\\\text{cost of predicting negative}&=pC_{\mathrm{FN}}.\end{aligned}$$

Choose positive when its expected cost is no larger. Rearranging $(1-p)C_{\mathrm{FP}}\le pC_{\mathrm{FN}}$ gives

$$p\ge\frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.$$
The expected cost of predicting negative is four times p, and the cost of predicting positive is one minus p. They cross at p equal to 0.20.
With false-alarm cost 1 and miss cost 4, the probability cutoff is 1 / 5 = 0.20. Below it, negative has lower expected cost; above it, positive does.

This 0.20 rule assumes meaningful probabilities for the population being served. It is not a replacement for the empirical 0.55 cutoff on the earlier arbitrary scores. Check probability calibration, and account for a review-capacity constraint separately if one exists.

References and further reading