Weight Initialization in Deep Learning: Reading Guide
A connected guide with step-by-step derivations, worked examples, figures, and interactive experiments.
Weight initialization sets the starting point of optimization. Before a single gradient step is taken, it must break symmetry across hidden units, keep forward activations and backward gradients from vanishing or exploding across depth, and match the architecture’s activations.
Reading order
See why identical hidden units stay identical, derive the per-layer second-moment multipliers $n_{\rm in}s_W^2$ (forward) and $n_{\rm out}s_W^2$ (backward), and watch scale compound through depth.
Derive fan-in ($s_W^2=1/n_{\rm in}$), fan-out ($s_W^2=1/n_{\rm out}$), Xavier ($s_W^2=2/(n_{\rm in}+n_{\rm out})$), and He ($s_W^2=2/n_{\rm in}$) variances, and convert them to uniform bounds via $a=\sqrt{3}\,s_W$.
Core notation at a glance
- Layer widths: $n_{\rm in}$ and $n_{\rm out}$ count the inputs and outputs connected to each unit in $W_l\in\mathbb R^{n_{\rm out}\times n_{\rm in}}$.
- Moments: Weight variance $s_W^2 = \operatorname{Var}(W_{l,ij}) = \mathbb E[W_{l,ij}^2]$ (standard deviation $s_W$), preactivation variance $\sigma_z^2 = \operatorname{Var}(z_{l,i}) = \mathbb E[z_{l,i}^2]$, activation second moment $q_l = \mathbb E[h_{l,i}^2] = \operatorname{Var}(h_{l,i}) + (\mathbb E[h_{l,i}])^2$ (or $q = \mathbb E[a_j^2]$), and backward gradient second moment $\tilde q_l = \operatorname{Var}(g_{l,i}) = \mathbb E[g_{l,i}^2]$ are tracked explicitly because non-centered activations like ReLU have $\mathbb E[h_{l,i}] > 0$.
- Connected series: Pair these notes with the optimizer series and the regularization series.