Popular Loss Functions
A loss function is the objective that a learning algorithm actually optimizes. Accuracy, F1, AUC, RMSE, and calibration error may be the metrics reported to stakeholders, while training optimizes a chosen loss. For gradient-based methods, its derivatives or subgradients supply a signal for changing the model.
This series collects common machine-learning objectives and explains why each one has its particular shape. The prediction-loss pages derive the best constant under squared error, absolute error, binary log loss, Poisson loss, and Huber loss. The entropy and divergence pages connect probability, coding, uncertainty, classification, trees, calibration, and distribution comparison. The Triplet Loss pages develop relative-distance learning for retrieval and verification.
Loss versus Metric
A metric answers the evaluation question: how good is the final model for the task? A loss answers the training question: how should the model be updated after one batch? The two are related but not identical. Accuracy is easy to understand, but it is flat almost everywhere, so it does not provide useful gradients. Cross-entropy is less directly interpretable to non-specialists, but it changes smoothly as probabilities improve or degrade.
Prediction losses and their best constants
Begin with the comparison and interactive loss explorer, then open a loss for its formula, correction target, and derivation of the best constant.
Squared Error: Why the Best Constant Is the Mean
Derive the mean as the best constant under squared error, connect it to residual corrections, and inspect how a large target changes the objective.
Absolute Error: Why the Best Constant Is a Median
Derive the median using slopes and subgradients, explain the interval of optimal constants for even samples, and compare it with squared error.
Binary Log Loss: From Class Frequency to Initial Log Odds
Derive the best constant probability and its log-odds score, with a class-imbalance example and explicit all-zero/all-one boundary cases.
Poisson Loss: Why the Best Constant Is a Log Mean
Derive the Poisson loss and optimal constant count, distinguish mean count from log-mean score, and handle zero counts and unequal exposures.
Huber Loss: Finding a Constant by Balancing Clipped Residuals
Derive the fixed-threshold Huber location equation, solve a worked example, and explain bisection, flat minima, and the role of the threshold.
Entropy, divergences, and metric learning
Entropy Foundations
Entropy as expected surprise, measured in bits, with an animation that turns a probability distribution into average code length.
Binary Entropy
The full binary entropy curve, why uncertainty peaks at p = 0.5, and how class imbalance changes impurity.
Cross-Entropy and KL Divergence
Why classification models minimize cross-entropy, and how excess code length becomes KL divergence.
Information Gain
Entropy in decision trees: parent entropy, weighted child entropy, gain, and why pure splits are rewarded.
Maximum Entropy and Calibration
Maximum-entropy distributions, softmax intuition, label smoothing, and why high confidence should be earned.
KL Divergence
KL as extra surprise when a model distribution disagrees with the real distribution, why direction matters, and where KL appears in ML objectives.
Jensen-Shannon Divergence
A symmetric, bounded, smoothed KL comparison used for distribution comparison, drift monitoring, and GAN intuition.
Earth Mover's Distance
Optimal transport intuition, the one-dimensional Wasserstein formula, and why geometry matters when distributions shift.
Triplet Loss Foundations
Anchor-positive-negative geometry, margin constraints, distance choices, normalized embeddings, and active triplets.
Triplet Geometry and Gradients
Exact squared-distance gradients, active hinge updates, margin geometry, normalization, scale, and collapse.
Triplet Mining and Batches
Hard and semi-hard negatives, structured batches, vectorized distances, masks, and false-negative safeguards.
Triplet Training and Evaluation
Retrieval metrics, verification, safe splits, collapse diagnostics, production behavior, and alternative objectives.
Future Loss Families
The same framework can later expand to quantile loss, hinge loss, focal loss, ranking losses, and supervised contrastive losses. Each new series should connect the formula to its gradients, sampling assumptions, evaluation target, and practical failure modes.