Deep Learning · Activation Functions · Part 7 of 9

SwiGLU: One Learned Branch Controls Another

October 2026 · 7 min read

SiLU changes each scalar using that scalar's own value. SwiGLU builds a larger operation: one projection of the input produces values, while another produces the factors that multiply those values. This lets one learned feature control another.

Follow the two branches

Use a row vector $x$ with $d$ features and an intermediate width $m$. For clarity, omit biases. Compute a gate preactivation $g$ and a value vector $v$:

$$g=xW_g,\qquad v=xW_u.$$

Apply SiLU to the gate branch, then multiply corresponding coordinates. The symbol $\odot$ means elementwise multiplication:

$$h=\operatorname{SiLU}(g)\odot v.$$

This gated intermediate vector is the SwiGLU operation. A complete feed-forward block then projects it back to the input width:

$$y=hW_d.$$

Shapes using row vectors. Leading batch and sequence dimensions are preserved.
Quantity	Shape	Job
$x$	$[\ldots,d]$	Input features
$W_g,W_u$	$d\times m$ each	Two projections
$g,v,h$	$[\ldots,m]$	Intermediate features
$W_d$	$m\times d$	Output projection

What does one coordinate do?

Suppose the projections have already produced the following gate and value entries. Each row is one coordinate of the same intermediate vector:

Compute SiLU(g), then multiply it by v. Rounded to four decimals.
$g_i$	$v_i$	SiLU$(g_i)$	$h_i$
−1	2	−0.2689	−0.5379
0	−3	0	0
2	0.5	1.7616	0.8808

Here “gate” means a multiplicative control. SiLU can be negative and can exceed one, so the factor can reverse a value's sign or amplify it. It is not a probability of retaining that feature.

How GLU, GEGLU, and SwiGLU differ

Keep the projections and multiplication fixed, and change the gate function:

$$\begin{aligned}\operatorname{GLU}&:\quad \sigma(g)\odot v,\\\operatorname{GEGLU}&:\quad \operatorname{GELU}(g)\odot v,\\\operatorname{SwiGLU}&:\quad \operatorname{SiLU}(g)\odot v.\end{aligned}$$

The transformer formulation here uses Swish with $\beta=1$, which is SiLU. Shazeer's GLU-variants paper compares these gated feed-forward blocks. Its empirical improvements belong to the tested setup; they are not a theorem that one gate always wins.

A library GLU operator may take one tensor of width $2m$ and split it into two branches. Two separate projections can likewise be packed into one larger projection. The branch meanings and resulting multiplication stay the same.

Both branches receive a gradient

For one coordinate, $h_i=\operatorname{SiLU}(g_i)v_i$. Hold the other branch fixed when taking each partial derivative:

$$\frac{\partial h_i}{\partial v_i}=\operatorname{SiLU}(g_i),$$

$$\frac{\partial h_i}{\partial g_i}=v_i\operatorname{SiLU}^\prime(g_i).$$

The value branch receives the gate factor. The gate branch receives the value times the SiLU slope. Backpropagation then continues through both projection matrices. A smooth gate does not guarantee a nonzero gradient in every coordinate.

Compare equal parameter budgets

A conventional bias-free MLP with intermediate width $r$ has two matrices and $2dr$ parameters. SwiGLU has three matrices and $3dm$. Matching those counts gives:

$$3dm=2dr\quad\Longrightarrow\quad m=\frac23r.$$

If the conventional width is $r=4d$, the matched gated width is $m=8d/3$. With $d=768$, widths 3072 and 2048 both give 4,718,592 matrix parameters. Real widths may be rounded for hardware. Equal parameter counts do not ensure equal latency or activation memory.

Advantages and disadvantages

These are gated blocks, not scalar activations. All three use a gate branch and a value branch, so they share the extra projection cost at fixed width. The table compares the gate choices:

Learned gates add multiplicative feature interactions, with different bounds and gradient behavior.
Block	Advantages	Disadvantages
GLU	A learned sigmoid gate gives a bounded, positive retention factor between zero and one.	The sigmoid gate can saturate, weakening its local derivative. Its factor cannot amplify a value above its original magnitude or reverse its sign.
GEGLU	A smooth GELU gate can reverse or amplify value entries instead of only attenuating them.	Requires GELU's CDF or approximation. A tiny gate factor can weaken the value-branch gradient; smoothness does not prevent small derivatives.
SwiGLU	A smooth SiLU gate lets one learned feature control another and supports negative or amplifying factors.	Uses three matrices in a full block instead of the conventional two. Matching parameters requires reducing width, and either branch can still receive a weak gradient.

A bounded gate factor is not automatically a probability, and these shape differences are not an accuracy ranking. Compare validation quality, runtime, and memory at matched budgets rather than assuming the extra branch is a free improvement.

Keep the computation visible

from scipy.special import expit

def swiglu(x, w_gate, w_up, w_down):
    g = x @ w_gate
    v = x @ w_up
    gate = g * expit(g)  # SiLU, computed with stable sigmoid
    return (gate * v) @ w_down

This NumPy-compatible sketch returns the whole feed-forward block. The broader transformer MLP guide places it inside a model. Next, softmax handles a different job: turning a vector of scores into competing probabilities.