Numerical Optimization: Reading Guide
Connect update geometry, curvature, games, adaptive step lengths, and communication-efficient matrix optimizers.
Reading order
Understand curvature through a quadratic, derive momentum and Adam updates, and separate adaptive scaling from decoupled weight decay.
Derive Newton and damped Gauss–Newton from local models, solve a numerical least-squares example, and understand singular curvature and acceptance tests.
See why descent–ascent can rotate, derive the rank-one update in the Gauss–Newton GAN paper, and inspect the limits of its local model.
Explain the perpendicular probe in angle-based gradient descent, derive its spherical example, and handle small angles and stochastic-gradient noise.
Derive the polar update from a spectral-norm constraint, distinguish it from elementwise sign, and understand practical orthogonalization and parameter groups.
Trace the paper’s distributed algorithm, distinguish sign aggregation from averaging, and calculate communication without confusing one-bit information with an integer collective.
How to use these notes
Start with the worked example in each chapter. Calculate the answer before changing the interactive figure, then use the self-check to test the distinction the chapter teaches. The numerical examples are deliberately small enough to audit by hand; they are teaching examples, not production benchmarks.
Follow the prerequisites when a concept is unfamiliar. Related notes link to existing foundations so each chapter can concentrate on its own question.