Deep Learning · Activation Functions · Part 2 of 9

Sigmoid and Tanh: Bounded Outputs and Saturation

October 2026 · 6 min read

Sigmoid and tanh both turn an unbounded input into a bounded output. That is useful when a value must stay within a range. The tradeoff appears in the tails: large changes in the input eventually produce very small changes in the output.

Sigmoid maps a score between zero and one

$$\sigma(z)=\frac{1}{1+e^{-z}}.$$

$z$ is the score entering the activation. At zero, the denominator is 2, so the output is $1/2$. Positive inputs move the output toward 1; negative inputs move it toward 0. The limits are approached rather than reached for finite real inputs in exact arithmetic.

A binary classifier can interpret $\sigma(z)$ as its modeled probability of the positive class. With several independent labels, apply a sigmoid to each label's score; those probabilities need not sum to one.

Tanh includes negative outputs

$$\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}.$$

Tanh ranges from −1 to 1 and passes through zero. It is odd: changing the input's sign changes the output's sign. Thus symmetric input distributions produce zero-mean outputs. It does not force arbitrary data to have zero mean.

Outputs for the same three inputs. Values rounded to four decimals.
Function	−1	0	1
Sigmoid	0.2689	0.5000	0.7311
Tanh	-0.7616	0.0000	0.7616

How sigmoid and tanh are related

Multiply the numerator and denominator of $\tanh(z)$ by $e^{-z}$ and split the numerator as $2-(1+e^{-2z})$. Tanh is an affine rescaling of sigmoid evaluated at $2z$:

$$\begin{aligned}\tanh(z)&=\frac{e^z-e^{-z}}{e^z+e^{-z}}=\frac{1-e^{-2z}}{1+e^{-2z}}\\[4pt]&=\frac{2}{1+e^{-2z}}-1=2\sigma(2z)-1.\end{aligned}$$

Solving the same equation for $\sigma$ gives the inverse relationship, expressing sigmoid as a shifted and halved tanh:

$$\sigma(z)=\frac{1+\tanh(z/2)}{2}.$$

This explains why both curves share the same S-shaped geometry with different centers and scales: doubling the input ($2z$) compresses the horizontal transition by a factor of two, while $2(\cdot)-1$ stretches the output range from $(0,1)$ to $(-1,1)$ and shifts the midpoint at $z=0$ from $1/2$ to $0$.

Why both functions saturate

Differentiating sigmoid gives $e^{-z}/(1+e^{-z})^2$. Rewrite the two factors using $\sigma(z)$ and $1-\sigma(z)$:

$$\sigma^\prime(z)=\sigma(z)(1-\sigma(z)).$$

Its largest slope is $1/4$, at zero. Near either output limit, one factor approaches zero, so the derivative becomes small. For tanh, differentiation gives:

$$\tanh^\prime(z)=1-\tanh^2(z)=4\,\sigma^\prime(2z).$$

The chain rule on $2\sigma(2z)-1$ multiplies by $2$ from the outer scale and another $2$ from the inner $2z$, so $\tanh^\prime(0)=4\cdot(1/4)=1$ is four times the peak slope of sigmoid, while still approaching zero in both tails. Saturation means this flattening of the curve. Through many hidden layers, repeated small derivative factors can weaken gradients; weights and residual connections also affect the final result.

Sigmoid and tanh outputs and derivatives: both flatten in the tails; their slopes at zero are 0.25 and 1. — Read the lower panel at the same input as the upper panel. A flat output region has a small derivative. Select the figure for a larger view.

Why sigmoid still works at a binary output

For ordinary binary cross-entropy with target $y$ and probability $p=\sigma(z)$, differentiation with respect to the logit simplifies to $\partial L/\partial z=p-y$. The loss and sigmoid derivatives combine; multiplying by an extra $p(1-p)$ afterward would be a mistake.

Use a loss that accepts logits directly, such as BCEWithLogitsLoss, to compute this combination stably. Hidden sigmoid layers still have their own sigmoid factors in backpropagation.

Advantages and disadvantages

Compare the benefits of a bounded output with the cost of saturation.
Function	Advantages	Disadvantages
Sigmoid	Smooth, bounded between zero and one, and useful for binary probabilities or retention gates.	Both tails have small derivatives; even its largest slope is only 0.25. Outputs are always positive, and being in range does not guarantee calibration.
Tanh	Allows bounded positive and negative values, maps zero to zero, and has slope 1 at zero. Symmetric inputs give zero-mean outputs.	Both tails saturate, so hidden-layer gradients can weaken. It does not center arbitrary inputs or directly produce probabilities.

The sigmoid saturation warning concerns its local derivative. At a binary output, the cross-entropy combination above gives $p-y$; it should not receive an extra sigmoid derivative.

When these shapes fit the job

Use sigmoid when the modeled output or a retention gate must lie between zero and one. Tanh suits signed bounded state updates, as in recurrent models. For a deep hidden stack, inspect saturation before choosing either; Glorot and Bengio investigate this interaction with initialization.

Before moving to piecewise-linear activations, see how stable sigmoid and log-sigmoid avoid floating-point overflow on large negative logits.