Triplet Loss Geometry, Margin, and Gradients
Triplet loss turns one ranking requirement into geometry. Around each anchor, the positive should lie inside a smaller distance while the negative should lie beyond that distance plus a margin.
This page uses squared Euclidean distance:
$$L=\left[\lVert a-p\rVert_2^2-\lVert a-n\rVert_2^2+\alpha\right]_+.$$The Margin Boundary
For fixed anchor $a$ and positive $p$, the triplet becomes inactive when
$$\lVert a-n\rVert_2^2\ge \lVert a-p\rVert_2^2+\alpha.$$In two dimensions, this condition places the negative outside a circle centered at the anchor. The circle radius is
$$r_{\text{required}}=\sqrt{\lVert a-p\rVert_2^2+\alpha}.$$The positive does not need to collapse onto the anchor. It only needs to be closer than the negative by the chosen margin.
Gradient of an Active Triplet
When the hinge term is positive, the loss inside the brackets is differentiable:
$$L_{\text{active}}=(a-p)^\top(a-p)-(a-n)^\top(a-n)+\alpha.$$The three embedding gradients are
$$\frac{\partial L}{\partial a}=2(n-p),\qquad \frac{\partial L}{\partial p}=2(p-a),\qquad \frac{\partial L}{\partial n}=2(a-n).$$Gradient descent subtracts these values:
- The positive moves toward the anchor.
- The negative moves away from the anchor.
- The anchor moves toward the positive and away from the negative in one combined update.
The anchor gradient does not contain $a$ after simplification because its two $2a$ terms cancel. Its direction depends on the line from the positive to the negative.
▶ Gradient Descent on One Active Triplet
Each step applies the exact squared-distance gradients with learning rate $\eta=0.08$. Updates stop when the margin condition is satisfied.
One Update by Hand
The animation starts from
$$a=(0,0),\qquad p=(1,0.4),\qquad n=(0.8,-0.1),\qquad \alpha=0.5.$$The distances and loss are
$$d_{ap}=1.16,\qquad d_{an}=0.65,\qquad L=1.16-0.65+0.5=1.01.$$The active gradients are
$$g_a=(-0.4,-1.0),\qquad g_p=(2.0,0.8),\qquad g_n=(-1.6,0.2).$$With $\eta=0.08$, the updated embeddings are
$$a'=(0.032,0.080),\qquad p'=(0.840,0.336),\qquad n'=(0.928,-0.116).$$The positive moved closer, the negative moved farther, and the anchor moved between those two goals.
Inactive Triplets
When the bracketed term is negative, the hinge returns zero and the gradient is zero:
$$d_{an}\ge d_{ap}+\alpha\quad\Longrightarrow\quad L=0,\quad \nabla L=0.$$This protects already-correct triplets from being pushed indefinitely. It also creates the main sampling problem: random training can become dominated by easy triplets that contribute no update.
Squared Versus Unsquared Euclidean Distance
| Distance | Positive-term gradient with respect to $a$ | Behavior |
|---|---|---|
| $\lVert a-p\rVert_2^2$ | $2(a-p)$ | Gradient magnitude grows with distance |
| $\lVert a-p\rVert_2$ | $\frac{a-p}{\lVert a-p\rVert_2}$ | Unit direction away from coincidence; needs an epsilon near zero |
Squared distance emphasizes far violations. Unsquared distance gives a more uniform pull but has an unstable denominator when two embeddings coincide.
Backpropagation Through Normalization
For normalized embedding $\bar z=z/\lVert z\rVert_2$, the Jacobian is
$$\frac{\partial \bar z}{\partial z} =\frac{1}{\lVert z\rVert_2}\left(I-\bar z\bar z^\top\right).$$The matrix $I-\bar z\bar z^\top$ removes the radial component of a gradient. Training therefore moves the normalized direction along the unit sphere rather than increasing its norm.
A small epsilon in the normalization denominator protects against a zero raw vector. Embedding norms should still be monitored because a nearly zero pre-normalized vector can amplify gradients.
What the Margin Controls
- Small margin: many triplets become easy early, reducing training signal.
- Large margin: many triplets remain active, including contradictory or impossible constraints.
- Normalized space: the maximum squared distance is $4$, which places a hard geometric limit on feasible margins.
The margin is a hyperparameter chosen with held-out retrieval or verification performance. Training loss alone prefers constraints that are easier to satisfy and does not identify the margin best suited to deployment.
What the Loss Cannot Identify
Euclidean triplet loss depends only on pairwise distances. Translating every embedding by the same vector or rotating the entire space leaves every loss unchanged. Reflection also preserves the objective.
Scaling is different. Multiplying all embeddings by $c$ multiplies squared distances by $c^2$ while leaving the fixed margin unchanged. Without normalization or norm control, scale becomes part of how the model satisfies triplets.
The Exact-Collapse Trap
If $a=p=n$, every squared distance is zero and each triplet has loss $\alpha$. Yet all three distance gradients are also zero because every pairwise difference is zero. Exact collapse is therefore a stationary point for the naive squared-distance objective, even though its loss is not minimal.
Random initialization, informative batches, normalization, auxiliary objectives, or proxy-based formulations help avoid this state. The final page returns to collapse diagnostics in full.