Activation Functions: A Reading Guide
An activation changes what a neural-network layer passes forward and how gradients travel backward. This series explains both sides, with short derivations, comparable curves, and a small number of calculations.
Read from scalar functions to learned gates
See where an activation sits, why linear layers collapse, and how its slope affects learning.
Understand bounded activations, derive their slopes, and separate hidden-layer saturation from probability outputs.
Prevent exponential overflow with the two-branch sigmoid identity, preserve tail precision in log-sigmoid, and derive stable binary cross-entropy from logits.
Follow ReLU through its gradient, then see how fixed and learned negative slopes change the behavior.
Unpack the Gaussian CDF and product-rule derivative for GELU, then compare ELU and SELU's exponential negative branch and self-normalizing conditions.
Explain sigmoid-weighted inputs, derive their slopes, and see how Swish scales the transition.
Build SwiGLU from SiLU, trace both branches and their gradients, and compare feed-forward blocks at equal parameter budgets.
Explain vector-valued softmax, stable computation, its coupled derivatives, and the connection to cross-entropy.
Build a smooth leaky activation from a desired slope, check its mathematics and numerics, and plan a fair training comparison.
Choose according to the job
| Job | Relevant chapter |
|---|---|
| Understand why a layer needs nonlinearity | Foundations |
| Bound a scalar or predict a binary probability | Sigmoid and tanh; Stable sigmoid |
| Keep a simple positive branch and control negative gradients | ReLU family |
| Use a smooth hidden activation or study self-normalization | GELU and SELU; SiLU and Swish |
| Build a gated transformer feed-forward block | SwiGLU |
| Predict mutually exclusive class probabilities | Softmax |
| Develop and evaluate a custom function | Design guide |
| Diagnose vanishing or exploding gradients by activation | Vanishing & exploding gradients subseries |
Subseries: Vanishing and exploding gradient problems
Derive the $L$-layer backward recurrence with $\phi'(z^{(l)})$ and plug in the minimum bound $m$ and maximum bound $M$ for every activation function in this series:
Derive the backward recurrence relation with $\phi'(z^{(l)})$ and sandwich the $L$-layer gradient between $m^L$ and $M^L$.
Plug the minimum slope $m$ and maximum slope $M$ of Sigmoid, Tanh, ReLU, Leaky ReLU/PReLU, ELU/SELU, GELU, SiLU/Swish, SwiGLU, and Softmax into $[m^L, M^L]$ to see the exact cause and effect of vanishing and exploding gradients.
Advantages and disadvantages at a glance
Compare the property you need, rather than treating one activation as best for every layer. Each linked chapter gives the function-specific advantages and disadvantages.
| Function or family | Advantages | Disadvantages |
|---|---|---|
| Sigmoid / tanh | Bounded outputs: sigmoid suits binary probabilities or retention gates; tanh also allows signed values. | Both tails saturate, weakening their local gradients. A bounded output does not guarantee calibration or zero mean. |
| Stable sigmoid / log-sigmoid | Two-branch evaluation and log-space losses avoid $e^{-z}$ overflow and $\log(0)$ underflow across extreme logits. | Requires branch selection or raw logits; pre-computed probabilities still lose tail precision. |
| ReLU family | Simple positive branch. ReLU gives exact zeros; leaky and learned variants retain a negative branch. | ReLU can block negative-side gradients. Variants need a slope setting or learned parameters. |
| GELU / ELU / SELU | GELU gives smooth Gaussian input weighting; ELU and SELU offer an exponential negative branch and a self-normalizing recipe. | Negative-tail gradients can still become tiny. Exact vs. approximate GELU differ, and SELU's normalization claim depends on the network setup. |
| SiLU / Swish | Smooth self-gating retains moderate negative inputs; Swish can tune the transition. | Extra nonlinear computation, small negative-tail gradients, and no guaranteed training advantage. |
| SwiGLU | One learned feature branch can control another through multiplication. | Adds a projection at fixed width. Equal-budget comparisons need a narrower intermediate layer. |
| Softmax | Normalizes competing scores into a distribution across classes or attention positions. | Forces competition, so it does not suit independent labels. Probabilities can still be poorly calibrated. |
| Custom smooth leaky function | Direct control of the negative slope and transition width. | Adds settings and computation; mathematical guarantees do not establish better training. |
Keep the symbols consistent
$z$ is the input to a scalar activation, $f(z)$ its output, and $f^\prime(z)$ its slope. $\sigma$ denotes sigmoid, while $\Phi$ is the standard normal cumulative distribution function. In the SwiGLU chapter, $x$ is a vector and $\odot$ means coordinate-by-coordinate multiplication.
Hidden activations create features. Output activations enforce the form of a prediction. Softmax couples a vector of scores; SwiGLU combines learned branches. Those distinctions matter more than collecting a long list of names.
Reproduce the curves and calculations
The five figures are calculated from the displayed formulas. Download the calculation script and its pinned requirements to regenerate them and check derivatives, softmax, and SwiGLU. The numerical values are available separately. No training benchmark results are implied by the plots.
Continue alongside weight initialization and optimizers: activation shape, starting scale, and parameter updates work together.