Why Neural Networks Need Activation Functions
A neuron first combines its inputs. An activation then decides how that combined value passes to the next layer. To understand any activation, follow two quantities: the value going forward and the gradient coming back.
Start with one neuron
Let $x_1,\ldots,x_d$ be the inputs, $w_1,\ldots,w_d$ their learned weights, and $b$ a learned bias. The weighted sum is called the preactivation, $z$. The activation function $f$ turns it into an output $a$:
The activation sees $z$, after the weights have mixed the inputs. A negative $z$ is simply a number the layer produced; it does not mean an invalid input or a wrong answer.
Why another linear layer is not enough
Consider two scalar layers with no nonlinearity. The first produces $a=w_1x+b_1$; the second produces $y=w_2a+b_2$. Substitute the first into the second:
This is still one straight-line rule: a new slope and a new intercept. The same collapse happens for a stack of affine matrix layers. More layers alone have not made the input-output relationship nonlinear.
Insert ReLU, $f(z)=\max(0,z)$, and the result can bend where the preactivation crosses zero. Different neurons can bend at different locations. Combining them lets the network represent more complicated functions.
The slope also controls learning
Let $L$ be the loss. Backpropagation tells this neuron how a change in its output $a$ would affect $L$. The chain rule carries that information through the activation:
$f^\prime(z)$ is the activation's slope at the current input. A slope near zero weakens this local gradient; a slope above one amplifies it. A negative slope reverses its sign. The complete gradient also depends on weights, other layers, and residual paths.
This is why each chapter shows both the activation curve and its derivative. A pleasant-looking output curve is only half the story.
Separate hidden features from output requirements
A hidden activation helps build useful features. An output activation gives the prediction the right form. An unconstrained regression output can use the identity $f(z)=z$. A binary probability can use sigmoid. Mutually exclusive class probabilities can use softmax across the class scores.
These jobs have different requirements. A hidden value does not have to be a probability, positive, or bounded. Likewise, a sigmoid output being between zero and one does not establish that its probabilities are calibrated.
Advantages and disadvantages
| Aspect | Advantages | Disadvantages |
|---|---|---|
| Nonlinear hidden layers | Allow bends and other nonlinear relationships that a stack of affine layers alone cannot express. | The chosen activation can saturate or block gradients. Extra expressive power does not make optimization easy. |
| Output constraints | Give outputs a useful form, such as binary probabilities, a class distribution, or a signed bounded state. | A wrong constraint limits what the model can predict. Normalized outputs alone do not guarantee calibrated probabilities. |
| Choice of shape | Lets you choose properties such as exact zeros, smooth transitions, or a negative gradient path. | Changes activation and gradient statistics. Initialization, computation, and the rest of the architecture still need to match. |
What to inspect in the next chapters
Look for the output range, behavior on negative inputs, slope near zero, and behavior far from zero. Then ask what the formula costs to compute and how it interacts with initialization. The importance of studying activation and gradient scales together is developed in Glorot and Bengio's initialization study.
Start with sigmoid and tanh: their bounded outputs make saturation easy to see.