Vanishing and Exploding Gradient Problems Across Activation Functions

October 2026 · 2 min read

1. Forward pass

2. Backward recurrence with $\phi'\!\bigl(z^{(l)}\bigr)$

With $\delta^{(l)} = \frac{\partial \mathcal{L}}{\partial z^{(l)}}$:

3. Unrolling $\delta^{(l)}$ across $L$ layers: minimum bound $m^L$ and maximum bound $M^L$

1. Unroll across all $L$ layers: Repeating $\delta^{(l)} = \Bigl(\bigl(W^{(l+1)}\bigr)^{\top} \delta^{(l+1)}\Bigr) \odot \phi'\!\bigl(z^{(l)}\bigr)$ from layer $L$ back to layer $0$ multiplies $L$ weight-and-derivative factors together:
   $$\bigl|\delta^{(0)}\bigr| = \underbrace{\bigl|W^{(1)}\phi'(z^{(0)})\bigr|}_{\text{Layer } 1} \cdot \underbrace{\bigl|W^{(2)}\phi'(z^{(1)})\bigr|}_{\text{Layer } 2} \cdots \underbrace{\bigl|W^{(L)}\phi'(z^{(L-1)})\bigr|}_{\text{Layer } L} \cdot \bigl|\delta^{(L)}\bigr|.$$
2. Isolate the activation function ($\bigl|W^{(l+1)}\bigr| = 1$): With unit-scale weights so the weights themselves neither shrink nor amplify the signal, only the $L$ activation derivatives remain:
   $$\bigl|\delta^{(0)}\bigr| = \bigl|\phi'(z^{(0)})\bigr| \cdot \bigl|\phi'(z^{(1)})\bigr| \cdots \bigl|\phi'(z^{(L-1)})\bigr| \cdot \bigl|\delta^{(L)}\bigr|.$$
3. Plug in the minimum bound $m$ and maximum bound $M$: If the activation derivative at every layer satisfies $m \le \bigl|\phi'(z^{(l)})\bigr| \le M$, replacing each of the $L$ factors by $m$ on the left and $M$ on the right sandwiches the gradient between $m^L$ and $M^L$:
   $$\underbrace{m \cdot m \cdots m}_{L\text{ layers}} \bigl|\delta^{(L)}\bigr| = m^L \bigl|\delta^{(L)}\bigr| \;\le\; \bigl|\delta^{(0)}\bigr| \;\le\; \underbrace{M \cdot M \cdots M}_{L\text{ layers}} \bigl|\delta^{(L)}\bigr| = M^L \bigl|\delta^{(L)}\bigr|.$$