Neel Mishra
← All Posts
Deep Learning · Popular Videos · Umar Jamil· Part 11 · Chapter 15

Pipeline Schedules: GPipe, 1F1B and Zero Bubble

September 2026 · 23 min read · Distributed Training from First Principles

Two schedules with an identical bubble can differ eightfold in memory. The bubble fraction is decided by how many microbatches you run; peak activation memory is decided by when you run their backward passes. Scheduling is the art of separating those two questions.

The problem GPipe leaves behind

The previous note derived the bubble fraction $(P-1)/(m+P-1)$ and showed that shrinking it needs many microbatches — $m\ge 9(P-1)$ for a 10% bubble. It also flagged the cost: under GPipe's all-forward-then-all-backward ordering, stage 0 holds activations for all $m$ microbatches simultaneously.

$$\text{GPipe peak activations}\ \propto\ m.$$

So the two goals are in direct conflict. Raise $m$ to kill the bubble and you raise memory by the same factor. At $P=8$ and $m=64$ that is 64 microbatches of activations on stage 0, which no accelerator will hold.

The resolution is that nothing requires all forwards to precede all backwards. A microbatch's backward pass can run as soon as its forward has reached the last stage and come back. Running it earlier frees its activations earlier.

1F1B: same bubble, bounded memory

The one-forward-one-backward schedule has three phases:

1

Warm-up. Stage $p$ runs $P-1-p$ forward passes to fill the pipe. Earlier stages run ahead; the last stage runs none.

2

Steady state. Each stage strictly alternates: one forward, one backward, one forward, one backward. Every backward immediately frees the activations of the microbatch it consumed, so memory stops growing.

3

Cool-down. The remaining backward passes drain out.

Count the microbatches in flight at stage $p$: it is $P-p$, so the worst case is stage 0 with $P$. That is the whole result:

$$\boxed{\text{1F1B peak activations}\ \propto\ P,\quad\text{independent of }m.}$$
The bubble is unchanged. 1F1B reorders operations but does not remove any idle slot — the pipe still fills and drains once, so the fraction is still $(P-1)/(m+P-1)$. What changes is that $m$ is now free to grow without a memory penalty. At $P=8,m=64$ it holds 8 microbatches instead of 64: an eightfold reduction for no cost at all.

This is why 1F1B, not GPipe, is the default in every serious framework. GPipe remains the clearer teaching example, and the two are identical in throughput.

forward backward idle
bubble fraction
peak microbatches on stage 0
memory versus GPipe
bubble if interleaved, $v=2$
Both schedules occupy exactly the same total time. Only the ordering of the backward passes differs, and with it the number of live activation sets.

Interleaved stages

1F1B fixed memory but left the bubble alone. To attack the bubble itself, give each device several non-contiguous chunks of the model instead of one contiguous block. With $v$ chunks per device — Megatron calls $v$ the virtual pipeline size — the pipeline has $vP$ logical stages running on $P$ physical devices, and the fill and drain overlap more finely:

$$\text{bubble}=\frac{P-1}{v\,m+P-1}.$$
Configuration$v=1$$v=2$$v=4$
$P=4$, $m=8$27.3%15.8%8.6%
$P=8$, $m=8$46.7%30.4%17.9%
$P=8$, $m=16$30.4%17.9%9.9%
The cost is communication. A microbatch now crosses device boundaries $v$ times more often, so point-to-point traffic multiplies by $v$. Pipeline traffic is small to begin with, so this is often affordable — but it is exactly the wrong trade if the pipeline spans a slow network, which is usually why the pipeline exists.

Zero bubble: split the backward pass

Every schedule so far treats the backward pass as one indivisible unit. It is not. From the FLOPs note, backward consists of two independent matmuls:

B — the input gradient

$$\bar X=\bar Y W^\top$$

On the critical path. The previous stage cannot start its own backward until this arrives.

W — the weight gradient

$$\bar W=X^\top\bar Y$$

Nothing downstream depends on it. It only has to be finished before the optimizer step at the end of the batch.

The insight. $W$ is deferrable work of exactly the right shape to fill a bubble. Schedule $B$ promptly to keep the pipeline flowing, and drop $W$ computations into the idle slots that fill and drain would otherwise waste. Since each is about half the backward cost, there is enough deferrable work to fill nearly the whole bubble.

Handling the fill and drain differently gives a family of schedules: heuristic variants that reduce the bubble by roughly a third to a half, and a fully optimized variant that eliminates it almost entirely at the price of extra memory and a more complex optimizer step.

Complexity is the real cost. Zero-bubble schedules need per-stage instruction sequences, careful gradient-buffer lifetime management, and a synchronization at the step boundary. They are worth it at large $P$ where the bubble is expensive, and rarely worth it at $P=2$ or $4$.

Choosing a schedule

ScheduleBubblePeak activationsP2P trafficUse when
GPipe$\dfrac{P-1}{m+P-1}$$O(m)$baselineteaching; almost never in production
1F1B$\dfrac{P-1}{m+P-1}$$O(P)$baselinethe sensible default
Interleaved 1F1B$\dfrac{P-1}{vm+P-1}$$O(vP)$$v\times$bubble hurts and the interconnect has headroom
Zero bubblenear zero$O(P)$ plus gradient buffersbaselinelarge $P$, and you can afford the machinery

One more lever sits outside the schedule entirely: activation recomputation. Storing only each stage's inputs and re-running the forward pass during backward makes GPipe's memory almost independent of $m$ too, at a 33% compute surcharge. It composes with any schedule, and combining 1F1B with selective recomputation is the usual production answer.

Takeaway

1F1B gives you GPipe's throughput with memory proportional to $P$ instead of $m$ — take it unconditionally. Interleaving trades communication for a smaller bubble. Zero-bubble schedules exploit the fact that the weight gradient has no downstream dependency and can therefore be deferred into the idle slots.

Check yourself