Optimizers in Deep Learning: From SGD to Matrix Geometry
Eleven detailed chapters with worked updates, original figures, and interactive experiments.
Backpropagation computes a gradient. The optimizer decides what to do with it. This series follows that decision from a single learning rate to memory, coordinate-wise scales, and matrix geometry. Every chapter connects the equations to a small calculation and an experiment you can inspect.
Eleven chapters, one progression
Review 1D and multivariate Taylor expansions, and prove why the negative gradient is a descent direction and why the gradient meets every contour tangent at $90^\circ$.
Derive the mini-batch update, see where sampling noise comes from, and explore learning-rate stability.
Unroll the momentum buffer, distinguish velocity from displacement, and see why narrow valleys are difficult.
Accumulate squared gradients per coordinate, equalize sparse and frequent features, and see why monotonic decay stalls.
Track two running averages and derive a coordinate-wise update from gradient and step history.
Understand squared-gradient memory, epsilon, and the difference between normalization and curvature.
Build Adam from its moments, audit the first update, and distinguish AdamW from L2 regularization.
Understand singular directions, momentum, polar updates, and which parameters belong in Muon.
Derive the cubic iteration, inspect Muon’s quintic approximation, and watch singular values evolve.
Derive matrix preconditioners, understand inverse fourth roots, and inspect their memory and computation.
Rotate into Shampoo’s eigenbasis, apply Adam, and understand why changing coordinates requires care.
Separate the two momentum coefficients, inspect sign changes, and understand update size and weight decay.
What changes between optimizers?
| Family | Question it answers | Methods |
|---|---|---|
| One global scale | How far should we step along the gradient? | SGD |
| Directional memory | Which directions persist across steps? | Momentum |
| Coordinate-wise adaptation | How large has each coordinate’s gradient or update been? | Adagrad, Adadelta, RMSprop, Adam |
| Matrix structure | How should we treat coupled row, column, and singular directions? | Muon, Shampoo, SOAP |
| Sign of a filtered gradient | Which way should each coordinate move? | Lion |
Notation that stays consistent
$w_t$ is the parameter after step $t$; $g_t$ is evaluated at $w_{t-1}$. A matrix parameter is $W$, and its gradient is $G$. The learning rate is $\eta$. Squares, square roots, and divisions on vectors are element-wise; matrix products are explicitly identified. A zero-initialized buffer means its value before the first update is zero.
How to use the experiments
Try the suggested experiment before changing everything at once. The first-order labs use the same two-dimensional quadratic dataset, seeded sampling, and exact gradients. Matrix labs expose the small matrices used in every calculation. These are controlled teaching experiments, not neural-network benchmarks or a ranking of optimizers.
Each chapter includes a static figure and worked values, so the argument also works without JavaScript. Interactive plots have text readouts and explicit step controls. No experiment starts playing automatically.
Before you start
You need partial derivatives, vector addition, and the idea of a training loss. The matrix chapters introduce the needed singular-value and eigenvalue geometry. For broader context, see the numerical optimization notes.