Deep Learning · Activation Functions · Part 6 of 9

SiLU and Swish: Smooth Self-Gating

October 2026 · 6 min read

GELU weights an input using a Gaussian CDF. SiLU uses sigmoid. In both cases, the input determines its own weight, which is why this pattern is called self-gating.

SiLU multiplies an input by its sigmoid

$$\operatorname{SiLU}(z)=z\,\sigma(z),\qquad\sigma(z)=\frac{1}{1+e^{-z}}.$$

For a large positive input, the sigmoid is close to one, so the input largely passes through. A moderately negative input receives a smaller positive weight and remains negative. Very negative inputs approach zero from below.

SiLU means sigmoid linear unit. Swish describes the family with an extra scale $\beta$:

$$\operatorname{Swish}_{\beta}(z)=z\,\sigma(\beta z).$$

SiLU is Swish with $\beta=1$. For $\beta\geq0$, increasing $\beta$ sharpens the transition. At $\beta=0$ the function is $z/2$; as $\beta$ grows without bound, the outputs approach ReLU. The Swish paper studies this family, including fixed and learned choices.

The sigmoid weight also changes

Use the product rule on $z\,\sigma(\beta z)$. Differentiating the input gives the sigmoid term. Differentiating the sigmoid contributes its derivative and a factor $\beta$ from the chain rule:

$$\begin{aligned}f^\prime(z)&=\sigma(\beta z)\\&\quad+\beta z\,\sigma(\beta z)[1-\sigma(\beta z)].\end{aligned}$$

For SiLU, set $\beta=1$. Keeping only $\sigma(z)$ would miss the change in the weight. The full derivative can be negative in part of the negative region and slightly exceed one elsewhere.

Outputs for the same three inputs. Values rounded to four decimals.
Function	−1	0	1
SiLU / Swish-1	-0.2689	0.0000	0.7311
Swish (beta=2)	-0.1192	0.0000	0.8808

SiLU and Swish with beta 2 outputs and slopes. Both have smooth negative regions and different transition sharpness. — Increasing $\beta$ sharpens the transition and changes both the output and derivative. The plot is a formula comparison, not a ranking of training performance. Select the figure for a larger view.

Advantages and disadvantages

SiLU is the beta=1 member of Swish; the extra scale parameter controls transition sharpness.
Function	Advantages	Disadvantages
SiLU	Smooth self-gating retains moderate negative inputs and approaches the identity on the positive side, without an extra shape parameter.	Needs a sigmoid and multiplication. Its negative tail still has small gradients, and the non-monotonic dip means slopes need not stay between zero and one.
Swish	For $\beta>0$, offers the same smooth behavior with control over transition sharpness. The shape can be fixed or learned.	Adds a setting or learned parameter. At $\beta=0$ it is linear, so it supplies no nonlinearity; for positive $\beta$, negative-tail gradients still become small.

Check computation as well as shape

Use a numerically stable sigmoid primitive. SiLU and Swish need a sigmoid and multiplication, with an extra scaling multiply when $\beta\neq1$. Actual speed depends on available fused kernels, precision, tensor sizes, and hardware, so operation counts alone are not a timing result.

Both are hidden-feature transformations. Their outputs are not probabilities and need not lie between zero and one. A smooth non-monotonic curve is a design choice to test, not a guarantee of better accuracy.

Next, SwiGLU uses SiLU on one learned branch to modulate a separate learned branch.