Plugging Minimum and Maximum Derivative Bounds Across Activation Functions

October 2026 · 6 min read

In the recurrence derivation, unrolling $\delta^{(l)} = \Bigl(\bigl(W^{(l+1)}\bigr)^{\top} \delta^{(l+1)}\Bigr) \odot \phi'\!\bigl(z^{(l)}\bigr)$ across $L$ layers with unit-scale weights ($\bigl|W^{(l+1)}\bigr|=1$) showed that if an activation function’s derivative satisfies $m \le \bigl|\phi'(z)\bigr| \le M$, then the backward gradient $\bigl|\delta^{(0)}\bigr|$ at layer $0$ is sandwiched between $m^L\bigl|\delta^{(L)}\bigr|$ and $M^L\bigl|\delta^{(L)}\bigr|$:

1. Why you need both bounds: the story of $m^L$ and $M^L$

• What the maximum bound $M = \max_z \bigl|\phi'(z)\bigr|$ tells you (best-case active region): $M^L$ is the ceiling on gradient flow when every neuron stays at the steepest point of $\phi(z)$:
  • If $M < 1$, then $M^L \to 0$: gradients must vanish even in the best possible case (e.g. Sigmoid with $M = \frac{1}{4}$).
  • If $M = 1$, then $M^L = 1$: the active region preserves gradients under unit-scale weights, and only explodes ($(|w|M)^L = |w|^L \to \infty$) when weight scale $|w| > 1$.
  • If $M > 1$, then $M^L \to \infty$: the activation function itself has an overshoot peak that can amplify and explode gradients across depth even when $|w| = 1$ (e.g. SELU, GELU, SiLU).
• What the minimum bound $m = \min_z \bigl|\phi'(z)\bigr|$ tells you (worst-case saturation & dead neurons): Even when $M \ge 1$ looks safe, pre-activations rarely stay at the peak—large weights, unnormalized sums, or negative bias drift push $z^{(l)}$ into flat tails, negative half-lines, or zero-slope troughs where $\bigl|\phi'(z^{(l)})\bigr|$ drops to its floor $m$:
  • Exact zero floor ($m = 0$ on a region or point): Because the $L$-layer gradient is a product $\prod_{l=0}^{L-1} \bigl|\phi'(z^{(l)})\bigr|$, even one layer hitting $m = 0$ turns the entire product into $0$ (permanent dead neurons in ReLU for $z < 0$, or exact gradient cancellation at the interior troughs of GELU and SiLU).
  • Asymptotic tail floor ($m_{\rm tail}(z) \to 0$ as $|z| \to \infty$): When pre-activations saturate at $|z| = c \gg 1$ (Sigmoid, Tanh, ELU, SELU), the effective floor in that tail is $m_{\rm tail}(c) \ll 1$, driving $m_{\rm tail}(c)^L \to 0$ exponentially across depth.
  • Non-zero leaky floor ($m = \alpha > 0$): Lifting the minimum slope from $0$ to $\alpha$ (Leaky ReLU) keeps the lower bound $\alpha^L > 0$ strictly positive so negative neurons can still receive gradients and recover.

2. Plugging $m$ and $M$ in for every activation function

1. Sigmoid: $\phi(z) = \sigma(z) = \frac{1}{1+e^{-z}}$

With derivative $\phi'(z) = \sigma(z)\bigl(1-\sigma(z)\bigr) = \frac{1}{4} - \bigl(\sigma(z)-\frac{1}{2}\bigr)^2$:

• Minimum bound: $m \to 0$ as $|z| \to \infty$, with tail floor $m_{\rm tail}(z) \approx e^{-|z|}$ for $|z| \gg 1$.
• Maximum bound: $M = \frac{1}{4} = 0.25$ (attained at $z = 0$).

Cause & effect: Because $M = \frac{1}{4} < 1$, even the upper bound vanishes exponentially ($(1/4)^{10} \approx 9.5 \times 10^{-7}$). Worse, because $\sigma(z) \in (0,1)$ is strictly positive, its non-zero mean pushes next-layer pre-activations into the $|z| \gg 1$ tail where the lower bound $e^{-L|z|} \to 0$ takes over. Trying to compensate by scaling weights to $|w| > 4$ (so $|w|M > 1$) causes active paths near $z=0$ to explode as $(|w|/4)^L \to \infty$ while saturated neurons stay frozen.

2. Tanh: $\phi(z) = \tanh(z)$

With derivative $\phi'(z) = 1 - \tanh^2(z)$:

• Minimum bound: $m \to 0$ as $|z| \to \infty$, with tail floor $m_{\rm tail}(z) \approx 4e^{-2|z|}$ for $|z| \gg 1$.
• Maximum bound: $M = 1$ (attained at $z = 0$).

Cause & effect: Raising the ceiling from $\frac{1}{4}$ to $M = 1$ stops vanishing at the origin ($1^L = 1$). However, whenever pre-activations drift past $|z| > 2$, the derivative falls onto the exponential tail floor $\bigl(4e^{-2|z|}\bigr)^L \to 0$ (vanishing gradients); conversely, if weight spectral norms exceed $|w| > 1$ near $z \approx 0$, the upper bound scales as $(|w|\cdot 1)^L = |w|^L \to \infty$ (exploding gradients).

3. ReLU: $\phi(z) = \max(0, z)$

With piecewise-constant derivative $\phi'(z) = 0$ for $z < 0$ and $\phi'(z) = 1$ for $z > 0$:

• Minimum bound: $m = 0$ (attained on the entire negative half-line $z < 0$).
• Maximum bound: $M = 1$ (attained on the entire positive half-line $z > 0$).

Cause & effect: Active positive paths sit at the upper bound $M^L = 1^L = 1$ without any tail saturation, which solves Sigmoid/Tanh vanishing—but also removes any saturation brake, so weights exceeding He scaling explode along the positive ray as $(|w|\cdot 1)^L \to \infty$. Meanwhile, any neuron pushed into $z < 0$ hits the exact lower bound $m = 0$, zeroing out its gradient forever (Dying ReLU).

4. Leaky ReLU and PReLU: $\phi(z) = \max(\alpha z, z)$

With derivative $\phi'(z) = \alpha$ for $z < 0$ and $\phi'(z) = 1$ for $z > 0$ (for $0 < \alpha < 1$):

• Minimum bound: $m = \alpha$ (for $z < 0$).
• Maximum bound: $M = 1$ (for $z > 0$; or $M = \alpha$ if a learned PReLU slope grows past $\alpha > 1$).

Cause & effect: Lifting the minimum bound from $m = 0$ to $m = \alpha > 0$ prevents dead neurons because $\alpha^L > 0$ never hits hard zero. However, if $\alpha$ is very small (e.g. $\alpha = 0.01$), a path of $L$ negative neurons still vanishes at rate $0.01^L$; in PReLU, if learned $\alpha > 1$, the upper bound becomes $\alpha^L \to \infty$ and explodes.

5. ELU and SELU: $\phi(z) = \lambda z$ ($z \ge 0$), $\lambda\alpha(e^z - 1)$ ($z < 0$)

With derivative $\phi'(z) = \lambda$ for $z \ge 0$ and $\phi'(z) = \lambda\alpha e^z$ for $z < 0$ (where ELU has $\lambda = 1$, and SELU has $\lambda \approx 1.0507, \alpha \approx 1.6733$):

• Minimum bound: $m \to 0$ as $z \to -\infty$, with negative-tail floor $m_{\rm tail}(z) = \lambda\alpha e^{-|z|}$.
• Maximum bound: $M = 1$ for ELU; $M = \lambda\alpha \approx 1.7581$ as $z \to 0^-$ and $M_+ = \lambda \approx 1.0507 > 1$ on the entire positive ray $z > 0$ for SELU.

Cause & effect: Deep negative inputs ($z \ll -2$) decay along the lower bound $e^{-L|z|} \to 0$. Conversely, SELU deliberately sets its positive slope to $\lambda \approx 1.0507 > 1$ so positive units expand variance to offset negative-tail contraction under LeCun initialization ($s_W^2 = 1/n_{\rm in}$); if paired with He initialization ($s_W^2 = 2/n_{\rm in}$), the upper bound explodes by $(2\lambda^2)^L \approx 2.21^L$.

6. GELU ($\phi(z) = z\Phi(z)$) and SiLU / Swish ($\phi(z) = z\sigma(\beta z)$)

By the product rule, $\phi'(z) = \Phi(z) + z\Phi'(z)$ for GELU and $\phi'(z) = \sigma(\beta z) + \beta z\sigma(\beta z)\bigl(1-\sigma(\beta z)\bigr)$ for SiLU/Swish:

• Minimum bound ($m = 0$): Unlike monotone activations, $\phi'(z)$ dips below zero on $(-\infty, 0)$, crossing $\phi'(z^*) = 0$ at an exact interior trough ($z^* \approx -0.7518$ for GELU; $\beta z^* \approx -1.2785$ for SiLU/Swish) and decaying to $0$ as $z \to -\infty$. At the origin $z = 0$, both have $\phi'(0) = \frac{1}{2}$.
• Maximum bound ($M > 1$): On $(0, \infty)$, the positive term $z\phi_{\rm gate}'(z) > 0$ pushes the slope strictly above $1$, peaking at $M_{\rm GELU} = 1 + \frac{1}{\sqrt{2\pi e}} \approx 1.1289$ (at $z = \sqrt{2}$) and $M_{\rm SiLU} \approx 1.0998$ (at $\beta z \approx 2.40$).

Cause & effect: Both functions exhibit three distinct regimes: (1) exact zero-gradient death ($m = 0$) at the trough $z^*$ and in the deep negative tail, (2) $(1/2)^L$ vanishing if pre-activations collapse around $z = 0$, and (3) $M^L > 1$ amplification (up to $1.1289^L$ or $1.0998^L$) when activations cluster in the positive overshoot zone $z \in [1, 2.5]$.

7. SwiGLU: $\phi(u, v) = \operatorname{SiLU}(u) \odot v$

Because SwiGLU multiplies two learned projections $u = W_{\rm gate}x$ and $v = W_{\rm up}x$, its partial derivatives with respect to the two branches are $\frac{\partial \phi}{\partial u} = v \odot \operatorname{SiLU}'(u)$ and $\frac{\partial \phi}{\partial v} = \operatorname{SiLU}(u)$:

• Minimum bound ($m = 0$): Attained whenever either branch vanishes ($v = 0$, $u = 0$, $u \approx -1.2785$, or $u \to -\infty$).
• Maximum bound ($M \propto \max(|u|, |v|) \to \infty$): Because each branch’s derivative is multiplied by the other branch’s activation magnitude, $M$ has no finite ceiling—it scales linearly with $\|x\|$.

Cause & effect: Small initial weights trigger a double-zero vanishing bottleneck (near $u, v \approx 0$, both branches suppress each other’s gradients), whereas large outlier features ($|u|, |v| \gg 1$) grow quadratically in the forward pass ($\phi(u,v) \approx u \odot v \propto \|x\|^2$) and push the backward upper bound $M^L \to \infty$ without Pre-RMSNorm and down-projection scaling.

8. Softmax: $p_i = \frac{e^{z_i/T}}{\sum_k e^{z_k/T}}$

Replacing the scalar derivative $\phi'(z)$ with the $K \times K$ Jacobian $J = \frac{1}{T}\bigl(\operatorname{diag}(p) - pp^{\top}\bigr)$ whose entries are $J_{ij} = \frac{1}{T}p_i(\delta_{ij} - p_j)$:

• Minimum bound ($m = 0$): $J\mathbf{1} = 0$ always has a zero eigenvalue along the uniform direction $\mathbf{1}$, and when one logit dominates ($p_c \to 1$, $p_{j \ne c} \to 0$), every entry of $J$ collapses to $0$.
• Maximum bound ($M = \frac{1}{2T}$ in spectral norm, $\frac{1}{4T}$ per diagonal entry): Attained when probability mass splits evenly between competing classes.

Cause & effect: Large logit gaps—such as unscaled attention dot products $q^{\top}k$ without $\frac{1}{\sqrt{d_k}}$—push $p_c \to 1$ onto the lower bound $m = 0$, freezing attention weights. Conversely, small temperatures $T \to 0$ blow up the upper bound $M = \frac{1}{2T} \to \infty$, exploding gradients whenever two logits are nearly tied.

3. Summary table: $m$, $M$, and $[m^L, M^L]$ across all activations