Xavier, LeCun, and He Initialization: Count the Inputs and Outputs

September 2026 · 4 min read + interactive lab

Derive LeCun (fan-in), fan-out, Xavier, and He variances from forward and backward sums, convert variance into uniform bounds, and count connections in dense and convolutional layers.

Fan-based initialization sets a layer’s weight variance $s_W^2=\operatorname{Var}(W_{l,ij})$ by counting how many terms each matrix multiplication sums: LeCun preserves forward scale ($q_l=q_{l-1}$), fan-out preserves backward gradient scale ($\tilde q_{l-1}=\tilde q_l$), Xavier (Glorot) balances the two, and He (Kaiming) doubles the variance when ReLU discards the negative half.

1. Forward sums ($n_{\rm in}$) vs. backward sums ($n_{\rm out}$)

Consider layer $l$ with $z_l=W_lh_{l-1}$ and $h_l=\phi(z_l)$ in the linear or small-signal $\tanh$ regime ($\phi(z)\approx z$, $\phi'(z)\approx 1$), where $W_l\in\mathbb R^{n_{\rm out}\times n_{\rm in}}$ has independent zero-mean weights $\mathbb E[W_{l,ij}]=0$ of variance $s_W^2=\operatorname{Var}(W_{l,ij})=\mathbb E[W_{l,ij}^2]$, independent of incoming activations $a_j=h_{l-1,j}$ with second moment $q_{l-1}=\mathbb E[h_{l-1,j}^2]$:

• Forward pass (LeCun / fan-in): Each output neuron $i\in\{1,\dots,n_{\rm out}\}$ takes the dot product of row $i$ of $W_l$ with $h_{l-1}$, summing over its fan-in of $n_{\rm in}$ incoming connections: $z_{l,i}=\sum_{j=1}^{n_{\rm in}}W_{l,ij}h_{l-1,j}$. Because the weights have zero mean and are independent, all cross terms ($j\ne k$) vanish in expectation, and each of the $n_{\rm in}$ squared terms contributes $\mathbb E[(W_{l,ij}h_{l-1,j})^2]=\mathbb E[W_{l,ij}^2]\,\mathbb E[h_{l-1,j}^2]=s_W^2\,q_{l-1}$:
  $$q_l=\mathbb E[h_{l,i}^2]\approx\mathbb E[z_{l,i}^2]=\sum_{j=1}^{n_{\rm in}}\mathbb E[W_{l,ij}^2]\,\mathbb E[h_{l-1,j}^2]=\sum_{j=1}^{n_{\rm in}}s_W^2\,q_{l-1}=(n_{\rm in}\,s_W^2)\,q_{l-1}.$$
  Requiring the forward second moment to stay constant ($q_l=q_{l-1}$) sets the forward multiplier $c=n_{\rm in}\cdot\operatorname{Var}(W)=1$, giving LeCun (fan-in) initialization:
  $$\begin{aligned}n_{\rm in}\cdot\operatorname{Var}(W)&=1\\\operatorname{Var}(W)=s_{W,\text{fan-in}}^2&=\frac{1}{n_{\rm in}}\\\sqrt{\operatorname{Var}(W)}=s_W&=\sqrt{\frac{1}{n_{\rm in}}}=\frac{1}{\sqrt{n_{\rm in}}}.\end{aligned}$$
• Backward pass (fan-out): During backpropagation, the chain rule routes upstream gradients $g_l=\partial\ell/\partial h_l$ backward via the transpose $g_{l-1}=W_l^\top\delta_l\approx W_l^\top g_l$. Whereas the forward pass sums across the $n_{\rm in}$ columns of row $i$, the backward pass sums down the $n_{\rm out}$ rows of column $j$—the input unit’s fan-out (how many downstream outputs unit $j$ feeds into):
  $$g_{l-1,j}=\sum_{i=1}^{n_{\rm out}}W_{l,ij}\,g_{l,i}\implies\tilde q_{l-1}=\mathbb E[g_{l-1,j}^2]=\sum_{i=1}^{n_{\rm out}}\mathbb E[W_{l,ij}^2]\,\mathbb E[g_{l,i}^2]=(n_{\rm out}\,s_W^2)\,\tilde q_l.$$
  Requiring backward gradient second moments to keep their scale ($\tilde q_{l-1}=\tilde q_l$) sets the backward multiplier $c_{\rm back}=n_{\rm out}\cdot\operatorname{Var}(W)=1$, giving fan-out initialization:
  $$\begin{aligned}n_{\rm out}\cdot\operatorname{Var}(W)&=1\\\operatorname{Var}(W)=s_{W,\text{fan-out}}^2&=\frac{1}{n_{\rm out}}\\\sqrt{\operatorname{Var}(W)}=s_W&=\sqrt{\frac{1}{n_{\rm out}}}=\frac{1}{\sqrt{n_{\rm out}}}.\end{aligned}$$

When $n_{\rm in}=n_{\rm out}$, both rules match. When $n_{\rm in}\ne n_{\rm out}$, fan-in scaling ($s_W^2=1/n_{\rm in}$) keeps the forward multiplier $c=1$ but scales backward gradient moments by $c_{\rm back}=n_{\rm out}/n_{\rm in}$, whereas fan-out scaling ($s_W^2=1/n_{\rm out}$) keeps $c_{\rm back}=1$ but scales forward moments by $c=n_{\rm in}/n_{\rm out}$.

2. Xavier and He initialization: balancing fans and ReLU

• Xavier (Glorot) initialization: To compromise when $n_{\rm in}\ne n_{\rm out}$ in linear or small-signal $\tanh$ layers, Glorot and Bengio set the average of the forward multiplier $c=n_{\rm in}\,s_W^2$ and backward multiplier $c_{\rm back}=n_{\rm out}\,s_W^2$ equal to $1$ (the harmonic mean of $1/n_{\rm in}$ and $1/n_{\rm out}$):
  $$\frac{n_{\rm in}\,s_W^2+n_{\rm out}\,s_W^2}{2}=1\implies s_{W,\text{Xavier}}^2=\frac{2}{n_{\rm in}+n_{\rm out}},\qquad s_{W,\text{Xavier}}=\sqrt{\frac{2}{n_{\rm in}+n_{\rm out}}}.$$
• He (Kaiming) initialization (ReLU and Leaky ReLU): As derived in Chapter 1, ReLU ($\phi(z)=\max(0,z)$) zeros out the negative half of a symmetric preactivation distribution, cutting the second moment in half ($q_l=\frac12\mathbb E[z_{l,i}^2]=(\frac12 n_{\rm in}s_W^2)q_{l-1}$ forward, and $\mathbb E[\phi'(z_l)^2]=\frac12$ backward). Setting $\frac12 n_{\rm in}s_W^2=1$ (or $\frac{1+\alpha^2}{2}n_{\rm in}s_W^2=1$ for Leaky ReLU with negative slope $\alpha$) gives:
  $$\frac12\,n_{\rm in}\,s_W^2=1\implies s_{W,\text{He}}^2=\frac{2}{n_{\rm in}},\qquad s_{W,\text{He}}=\sqrt{\frac{2}{n_{\rm in}}}\qquad\left(\text{or }s_W^2=\frac{2}{(1+\alpha^2)n_{\rm in}}\text{ for Leaky ReLU}\right).$$

3. Converting variance to uniform bounds: how $a=\sqrt{3}\,s_W$ translates to weight init and $\sqrt{6}$

Our signal-propagation equations in Sections 1 and 2 only depend on two numbers: $\mathbb E[W_{l,ij}]=0$ and $\operatorname{Var}(W_{l,ij})=s_W^2$. They do not require the weights to be Gaussian—we can draw $W_{l,ij}$ from either a Normal distribution or a Uniform distribution, provided both have mean $0$ and variance $s_W^2$:

• Normal weight initialization ($W_{l,ij}\sim\mathcal N(0,s_W^2)$): A Gaussian is parameterized directly by its standard deviation $s_W=\sqrt{s_W^2}$, so we pass $s_W$ straight to the sampler (e.g., $s_{W,\text{Xavier}}=\sqrt{\frac{2}{n_{\rm in}+n_{\rm out}}}$ or $s_{W,\text{He}}=\sqrt{\frac{2}{n_{\rm in}}}$).
• Uniform weight initialization ($W_{l,ij}\sim\operatorname{Uniform}(-a,a)$): A uniform sampler is parameterized by its interval endpoints $[-a,+a]$, where every weight in $[-a,a]$ has constant density $p(w)=\frac{1}{a-(-a)}=\frac{1}{2a}$. What bound $a$ must we choose so that $\operatorname{Uniform}(-a,a)$ has the target variance $s_W^2$? Integrating $w^2p(w)$ over $[-a,a]$ gives:
  $$\operatorname{Var}(W_{l,ij})=\mathbb E[W_{l,ij}^2]=\int_{-a}^a w^2\,\frac{1}{2a}\,dw=\frac{1}{2a}\left[\frac{w^3}{3}\right]_{-a}^a=\frac{1}{2a}\left(\frac{a^3-(-a)^3}{3}\right)=\frac{1}{2a}\cdot\frac{2a^3}{3}=\frac{a^2}{3}.$$
  Setting this uniform variance $\frac{a^2}{3}$ equal to the target variance $s_W^2$ solves for the uniform endpoint $a$:
  $$\frac{a^2}{3}=s_W^2\implies a^2=3\,s_W^2\implies a=\sqrt{3}\,s_W\approx 1.732\,s_W.$$
  What $a=\sqrt{3}\,s_W$ means in practice: If you naively initialized weights from $\operatorname{Uniform}(-s_W,+s_W)$, their variance would only be $s_W^2/3$—three times too small. Because a uniform box on $[-a,a]$ has no tails beyond $\pm a$, its endpoints must stretch out to $\pm\sqrt{3}\,s_W$ to achieve variance $s_W^2$.
• How $a=\sqrt{3}\,s_W$ produces $\sqrt{6}$ in Xavier and He Uniform: Both $s_{W,\text{Xavier}}=\sqrt{\frac{2}{n_{\rm in}+n_{\rm out}}}$ (where $2$ comes from averaging $n_{\rm in}$ and $n_{\rm out}$) and $s_{W,\text{He}}=\sqrt{\frac{2}{n_{\rm in}}}$ (where $2$ compensates for ReLU halving the second moment) already have a $\sqrt{2}$ in the numerator. Plugging $s_W$ into $a=\sqrt{3}\,s_W$ multiplies $\sqrt{3}$ by $\sqrt{2}$ to give $\sqrt{3\times 2}=\sqrt{6}$:
  $$\begin{aligned}\text{LeCun Uniform }\left(s_W=\sqrt{\tfrac{1}{n_{\rm in}}}\right):&\quad a=\sqrt{3}\,s_W=\sqrt{3}\cdot\sqrt{\frac{1}{n_{\rm in}}}=\sqrt{\frac{3\times 1}{n_{\rm in}}}=\sqrt{\frac{3}{n_{\rm in}}}\implies W_{l,ij}\sim\operatorname{Uniform}\!\left(-\sqrt{\frac{3}{n_{\rm in}}},\;+\sqrt{\frac{3}{n_{\rm in}}}\right),\\[6pt]\text{Xavier Uniform }\left(s_W=\sqrt{\tfrac{2}{n_{\rm in}+n_{\rm out}}}\right):&\quad a=\sqrt{3}\,s_W=\underbrace{\sqrt{3}}_{\text{uniform factor}}\cdot\underbrace{\sqrt{\frac{2}{n_{\rm in}+n_{\rm out}}}}_{s_{W,\text{Xavier}}}=\sqrt{\frac{6}{n_{\rm in}+n_{\rm out}}}\implies W_{l,ij}\sim\operatorname{Uniform}\!\left(-\sqrt{\frac{6}{n_{\rm in}+n_{\rm out}}},\;+\sqrt{\frac{6}{n_{\rm in}+n_{\rm out}}}\right),\\[6pt]\text{He Uniform }\left(s_W=\sqrt{\tfrac{2}{n_{\rm in}}}\right):&\quad a=\sqrt{3}\,s_W=\underbrace{\sqrt{3}}_{\text{uniform factor}}\cdot\underbrace{\sqrt{\frac{2}{n_{\rm in}}}}_{s_{W,\text{He}}}=\sqrt{\frac{6}{n_{\rm in}}}\implies W_{l,ij}\sim\operatorname{Uniform}\!\left(-\sqrt{\frac{6}{n_{\rm in}}},\;+\sqrt{\frac{6}{n_{\rm in}}}\right).\end{aligned}$$

4. Lab: compare expected and measured amplification

Test how $n_{\rm in}$, $n_{\rm out}$, and the variance rule affect a single rectangular linear layer:

5. Fan counts for dense and convolutional layers

• Dense layer (weight shape $[n_{\rm out}, n_{\rm in}]$): $n_{\rm in}$ is the second dimension and $n_{\rm out}$ is the first.
• Conv2d layer (weight shape $[C_{\rm out}, C_{\rm in}/G, k_h, k_w]$ with $G$ groups): Each output pixel sums over a $k_h\times k_w$ spatial patch across $C_{\rm in}/G$ input channels, so
  $$n_{\rm in}=\frac{C_{\rm in}}{G}k_hk_w,\qquad n_{\rm out}=\frac{C_{\rm out}}{G}k_hk_w.$$

# PyTorch Linear stores weight as [out_features, in_features]
torch.nn.init.xavier_uniform_(layer.weight, gain=1.0)
if layer.bias is not None:
    torch.nn.init.zeros_(layer.bias)

# He (Kaiming) fan-in normal initialization for a ReLU layer
torch.nn.init.kaiming_normal_(layer.weight, mode="fan_in", nonlinearity="relu")

# Explicit LeCun normal (used with SELU or forward-preserving linear layers)
torch.nn.init.normal_(layer.weight, mean=0.0, std=1.0 / math.sqrt(layer.in_features))

6. Questions worth answering

Sources

Glorot and Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks; He et al., Delving Deep into Rectifiers; LeCun et al., Efficient BackProp; Klambauer et al., Self-Normalizing Neural Networks; PyTorch initialization documentation.