Angle-Based Step Lengths: Probe the Gradient Field
Before you start: Gradient directions and step sizes.
Two nearby gradients can reveal how quickly a landscape bends. The angle-based method in Mishra and Kumar’s paper makes a small move perpendicular to the current gradient, measures the gradient again, and uses the angle between the two directions to estimate a travel distance. It is not simply the angle between gradients from two consecutive training iterations.
A setting where the triangle is exact
For f(x,y)=(x²+y²)/2, start at X₁=(r,0) with r>0. The negative gradient points left toward the origin. Probe at X₂=(r,h); its negative gradient points toward the same origin at an angle θ with tanθ=h/r. Therefore hcotθ=r: the estimated travel distance equals the distance to the optimum in this spherical quadratic.
With r=2 and h=0.2, θ=arctan(0.1)≈5.71° and d=2. This exact construction explains the intuition. In a general high-dimensional nonlinear landscape, the gradient rays need not intersect, and a randomly chosen tangent direction probes only one part of the geometry. The formula then acts as a heuristic step estimate.
Construct a perpendicular direction
Draw a vector z and subtract its projection onto the unit gradient direction $\widehat g$: $p=z-(z^\top\widehat g)\widehat g$. Normalize p if its norm is sufficiently large; otherwise draw again. In one dimension there is no nonzero perpendicular direction, so this construction needs at least two dimensions or an explicitly different fallback.
import numpy as np
def perpendicular_unit(g, rng):
g = np.asarray(g, dtype=float)
norm = np.linalg.norm(g)
if len(g) < 2 or norm == 0:
raise ValueError("Need dimension >= 2 and a nonzero gradient")
unit = g / norm
for _ in range(100):
z = rng.normal(size=g.shape)
p = z - np.dot(z, unit) * unit
if np.linalg.norm(p) > 1e-10:
return p / np.linalg.norm(p)
raise RuntimeError("Could not construct a stable probe")The paper’s complete step rule adds a heuristic
Algorithm 1 adds a small ε to the measured angle before computing hcotθ. It also keeps an exponential moving average of the estimated distance and doubles the current distance when it falls below that average. These operations are part of the proposed rule, not consequences of the exact triangle derivation. Reproductions should state initialization and numerical safeguards explicitly.
The distance d multiplies a normalized negative gradient. An equivalent ordinary learning rate would be d/‖g₁‖, which changes with gradient norm. Confusing distance with a multiplier on the unnormalized gradient changes the algorithm. The sideways probe size h is a separate hyperparameter and does not itself equal the main learning rate.
Small angles are a difficult measurement problem
As θ approaches zero, cotθ grows without bound. Adding ε prevents one exact singularity but does not guarantee a reasonable step. Clamp the cosine argument to [−1,1] before arccos, check finite values, handle zero gradients, and consider an explicit maximum distance or acceptance test in a safeguarded variant. Label such safeguards as implementation choices if they differ from the paper.
For stochastic training, use the same minibatch and controlled stochastic layers for the two gradient evaluations when trying to measure local geometry. Otherwise their angle mixes curvature with sampling noise. The second gradient also adds compute. Compare validation quality versus wall-clock cost, not only the number of outer updates.
Start where the expected answer is known
Test the spherical quadratic, an anisotropic quadratic, and a nonconvex surface. On the sphere, the unsafeguarded geometric estimate should recover r up to numerical error. On other surfaces, inspect accepted loss changes and probe-direction sensitivity. Then compare against tuned SGD, momentum, and Adam under the same data and compute accounting.
Check your understanding: Why is d not the same as the usual learning rate η?
The update uses d times a unit direction. In θ′=θ−η∇f, the distance is η‖∇f‖, so η=d/‖∇f‖ when the gradient is nonzero.
Sources and further reading
- Mishra & Kumar: angle based dynamic learning rate for gradient descent — Perpendicular probing, the cotangent distance, and the moving-average heuristic.