Regularization in Deep Learning: Reading Guide
A connected guide with step-by-step derivations, worked numerical examples, geometric figures, and interactive experiments.
Regularization biases training toward solutions that generalize beyond the training set. Depending on the mechanism, it acts on parameter norms, stochastic activation masks, synthetic inputs, soft target distributions, optimization trajectories, input Jacobians, or worst-case neighborhoods.
Reading order
Derive proportional L2 shrinkage, L1 soft-thresholding via subgradients, and the elastic-net proximal update.
Separate coupled L2 from AdamW’s decoupled weight decay, project onto max-norm balls, and bound amplification via spectral normalization.
Derive inverted dropout’s $1/q$ scaling, its exact equivalence to feature-scaled L2 in linear regression, and DropConnect’s covariance.
Compare element, channel (SpatialDropout), block (DropBlock), and residual-branch (Stochastic Depth) masks.
Contrast invariance with equivariance, derive Mixup’s linear cross-entropy decomposition, and compute clipped CutMix label weights.
Derive the $p_k-q_k$ logit gradient, the finite optimal logit gap under label smoothing, and the $T^2$ gradient scaling in knowledge distillation.
Derive gradient descent’s spectral filter $1-(1-\eta s)^t$, compare it with ridge regression’s $s/(s+\lambda)$, and inspect checkpoint selection.
Expand expected noisy loss to second order, connect consistency losses to the Jacobian Frobenius norm, and contrast R1 with WGAN-GP.
Derive closed-form $L_\infty$ (FGSM) and $L_2$ worst-case perturbations, trace PGD, and step through Sharpness-Aware Minimization’s two-pass update.
Core notation at a glance
- Objectives: $L(w)=\frac1N\sum_{i=1}^N\ell_i(w)$ is the average data loss, $J(w)$ adds an explicit penalty with strength $\lambda$, and $q=1-p$ is the dropout keep probability.
- Connected series: Read alongside the optimizer series and the weight initialization series.