Optimizer Geometry: From Gradient Descent to AdamW
Before you start: Gradients and gradient descent.
A gradient tells you which direction locally increases the objective fastest under the ordinary Euclidean geometry. It does not tell you how far to travel. On a narrow valley, one coordinate can tolerate a large step while another becomes unstable. Optimizers differ in how they use history and scale to choose that motion.
A stability condition you can derive
For $f(x)=\tfrac12ax^2$ with $a>0$, gradient descent gives $x_{t+1}=(1-\eta a)x_t$. Convergence requires $|1-\eta a|<1$, or $0<\eta<2/a$. Larger curvature permits a smaller stable step. For a positive-definite quadratic with Hessian H, the largest eigenvalue controls this upper bound.
In f(x,y)=(x²+10y²)/2, a single learning rate must respect the curvature 10 along y. The shallow x direction then progresses relatively slowly. Feature scaling, preconditioning, and adaptive methods all try to make the optimization geometry easier, though they do so differently.
Momentum remembers directions
This exponential-average convention smooths gradients. Some implementations omit the factor 1−β and absorb it into the learning rate; the two conventions need matched scaling before comparison. Consistent directions accumulate, while rapidly alternating components partially cancel. Momentum can accelerate progress but can also overshoot; it does not remove the need to tune the step size.
Adam scales coordinates using their recent squared gradients
Squares, division, and square roots are coordinatewise. The bias corrections compensate for moments initialized at zero. On the first step with a nonzero gradient, the corrected moments are g and g², so the update is approximately −ηsign(g), ignoring ε. Adam therefore changes the direction relative to raw gradient descent; it is not just smoothing the same vector.
The second moment is an uncentered squared-gradient average, not the variance unless the mean is also accounted for. It is not the Hessian. It changes sensitivity to coordinate scaling and gradient history, but does not encode full cross-coordinate curvature. Two parameters with correlated gradients still have separate denominators.
AdamW separates shrinkage from the adaptive gradient
This displayed AdamW-style step applies weight shrinkage separately from the loss gradient. Adding λθ to the gradient instead sends that term through Adam’s moments and coordinate scaling. Those operations are generally different. Specify which parameters decay; biases and normalization parameters are often handled separately according to the training recipe.
| Method | Persistent state per parameter | Main control |
|---|---|---|
| SGD | None beyond the parameter | Learning rate |
| Momentum | One first-moment buffer | Learning rate and momentum |
| Adam/AdamW | First and second moments | Rates, moment decay, epsilon, decay policy |
Compare optimization fairly
Use the same initialization, data order, objective, and batch definition. Tune each optimizer on validation under a declared budget. Plot objective versus steps and versus wall-clock time: a costly step can win per iteration and lose per second. Across stochastic runs, show variability rather than selecting the best seed.
Monitor update norm relative to parameter norm, gradient norm, loss spikes, and validation quality. A stable training loss does not prove useful generalization. If changing the optimizer also changes batch size, precision, or regularization, report those changes instead of attributing everything to the update rule.
Check your understanding: Does Adam’s second-moment buffer estimate the Hessian?
No. It tracks coordinatewise squared gradients. Curvature methods use derivatives or approximations with a different mathematical meaning.
Sources and further reading
- Kingma & Ba: Adam — First and second moment estimates with bias correction.
- Loshchilov & Hutter: decoupled weight decay — Why weight decay and an L2 penalty differ under adaptive updates.