Pipeline Parallelism: Minimizing Bubble Overhead

In 2021, NVIDIA's Megatron-LM team faced a concrete engineering wall while training GPT-3-scale models across thousands of A100 GPUs. The standard fix for pipeline bubbles — pack in more microbatches per pipeline flush — worked mathematically, but every extra microbatch meant every stage had to keep another forward pass's activations sitting in GPU memory until its matching backward pass finally arrived. At the model sizes they were training, that memory was already the scarcest resource in the building. They could not simply "add more microbatches" their way out of the bubble problem. This chapter is about what they did instead — and about a more recent line of work that asks whether pipeline bubbles need to exist at all.

You should already know the basic GPipe (Huang et al., NeurIPS 2019) picture: split a model into p sequential stages across p devices, split a training batch into m microbatches, and stream the microbatches through the stage pipeline like cars through a car wash. Because stage 1 can't start on microbatch 1 until stage 0 finishes it, and the last stage can't finish backward on the last microbatch until every earlier stage has passed its gradient back, every device sits idle during a warm-up ramp and a cool-down drain. For uniform stages, the standard result is that the total wall-clock time is (m + p − 1)·(t_f + t_b), versus an ideal, bubble-free m·(t_f + t_b), where t_f and t_b are the per-stage forward and backward times. That gives the familiar bubble fraction (p − 1)/(m + p − 1), which shrinks toward zero as m grows. What that formula does not tell you is what growing m actually costs — and that omission is exactly where the interesting engineering lives.

The hidden cost the bubble formula doesn't show

In GPipe's original schedule, every device finishes all of its forward passes before starting any of its backward passes. That ordering is what makes the bubble-fraction formula so clean, but it has a brutal side effect: a device cannot discard an activation until the matching backward pass consumes it, and none of the backward passes start until every forward pass on every device is done. So every device must hold activations for all m microbatches simultaneously at peak. Shrinking the bubble fraction by cranking up m — the "obvious" fix — makes this memory problem strictly worse. You are trading a time cost you can measure directly against a memory cost that, once it exceeds your GPU's HBM, doesn't degrade gracefully — it crashes the job. This is the wall Megatron-LM's engineers hit, and it's the reason production pipeline schedules abandoned GPipe's naive ordering entirely.

1F1B: identical bubbles, a fraction of the memory

PipeDream (Narayanan et al., SOSP 2019) introduced the fix that Megatron-LM adopted as its default: the one-forward-one-backward (1F1B) schedule. Instead of running all forwards before any backward, each device alternates — run one forward, then, as soon as a matching backward gradient becomes available, run one backward, freeing that activation immediately. Concretely, device d (0-indexed, out of p stages) runs a warm-up phase of p − d forward passes to fill the pipeline, then enters a steady state alternating exactly one forward and one backward per step, then drains with the remaining backward passes.

The remarkable thing is what does not change: the total makespan and hence the bubble fraction are identical to GPipe's — still (p − 1)/(m + p − 1). Reordering when a device runs forward versus backward doesn't change how much total idle time the pipeline structure forces; the warm-up and cool-down ramps are a function of p and m alone. What changes is memory: because each backward pass now runs as soon as its dependency is satisfied rather than being deferred to the very end, a device never needs to hold more than p − d activations in flight — bounded by pipeline depth, not by microbatch count.

Concretely, for p = 4 stages and m = 8 microbatches: GPipe's naive ordering forces the first stage to hold all 8 microbatches' worth of activations before its first backward pass runs. 1F1B bounds that same stage to 4 — exactly p, completely independent of m. Double m to 16 and GPipe's peak memory doubles to 16; 1F1B's stays at 4. This is the actual payoff of 1F1B, and it's not visible anywhere in the bubble-fraction formula.

Interleaving: shrinking the bubble itself

1F1B fixes memory but leaves the bubble fraction untouched — it's still governed by p − 1 idle steps relative to m productive ones. Megatron-LM's SC21 paper (Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM," 2021) attacks the bubble fraction itself with a second idea: the interleaved (virtual pipeline) schedule. Instead of giving each of the p devices one contiguous block of layers, split the model into p·v chunks and assign each device v non-adjacent chunks, round-robin. Device d now holds chunk d, chunk d + p, chunk d + 2p, and so on, up to v chunks total.

Why does this help? Each chunk is now only 1/v the size of what the device held before, so a single forward or backward pass through one chunk takes only t_f/v or t_b/v. The pipeline's warm-up and cool-down ramps are still p − 1 "hops" deep, but each hop is now v times shorter, because the device only has to wait for a thin chunk of neighboring work, not a full 1/p slice of the model, before it can proceed. Deriving this from first principles: total work per device across the whole run is still fixed at m·(t_f + t_b) (each microbatch still needs a full forward and backward through that device's share of the model, however it's chopped up), but the bubble time shrinks to (p − 1)·(t_f + t_b)/v. That gives:

total time = (t_f + t_b) · [m + (p − 1)/v],   bubble fraction = (p − 1) / (v·m + p − 1)

Setting v = 1 recovers the plain 1F1B/GPipe formula exactly — a useful sanity check that this is a genuine generalization, not a different model. Notice also that the formula depends on v and m only through their product v·m: doubling the number of virtual stages has mathematically the same bubble-shrinking effect as doubling the microbatch count, even though the two routes have completely different memory and communication consequences — a fact worth remembering when you're deciding which knob to turn.

Worked example

Take p = 4 devices, m = 8 microbatches, forward time t_f = 1 unit, backward time t_b = 2 units per stage (backward is roughly twice the FLOPs of forward, a standard approximation since it computes gradients with respect to both activations and weights). Compare three interleaving depths:

Every entry checks two ways — the closed-form fraction (p−1)/(vm+p−1) and the direct ratio of bubble time to total time — and they agree at every row, e.g. for v=2: 3/(16+3) = 3/19 = 0.1579 matches 4.5/28.5 = 0.1579. Going from no interleaving to v=4 nearly cuts the bubble fraction to a third, and the total pipeline time drops from 33 to 26.25 units — a 20% wall-clock reduction from a scheduling change alone, no extra hardware.

That gain is not free. Splitting each device's work into v chunks means each chunk boundary is a point-to-point send/receive to a neighboring device, so the number of communication events per microbatch multiplies by roughly v. On a single node with NVLink, that extra traffic is nearly free. Across nodes connected by ordinary Ethernet or a congested fabric, it can dominate — which is exactly why production configurations pick a modest v (2 or 4 is typical) rather than maximizing it. There is also a memory cost: because a device now has v separate warm-up ramps running concurrently instead of one, its peak number of buffered activations rises somewhat above plain 1F1B's p — still independent of m, but with a larger constant, since a device must keep more, smaller in-flight chunks alive simultaneously before each is consumed.

Beyond scheduling tricks: can the bubble be zero?

1F1B and interleaving both accept that some idle time is structurally forced by the dependency chain between forward and backward passes. Zero Bubble Pipeline Parallelism (Qi, Wan, Huang, and Lin, ICLR 2024) challenges that assumption by attacking the dependency itself. The key observation: a backward pass is conventionally treated as one atomic unit, but it actually does two distinct jobs — computing the gradient with respect to the layer's input activations (call it B), which the upstream stage genuinely needs before it can run its own backward pass, and computing the gradient with respect to that layer's weights (call it W), which only that device's own optimizer step will ever consume.

B is a hard dependency: stage d − 1 is blocked until stage d finishes B and ships the gradient backward. W has no such consumer outside the device that computed it. Once you split backward into B and W, W becomes a free-floating chunk of work that the scheduler can drop into any idle slot in the pipeline — including slots that used to be bubbles — without blocking any other device's progress. The zero-bubble schedules (ZB-H1, ZB-H2 in the paper) place W chunks precisely into what would otherwise have been warm-up or cool-down idle time, driving the bubble fraction toward zero as the schedule's flexibility increases. The cost moves from wasted time to two other places: extra activation memory (deferred W chunks mean some intermediate state has to stay alive longer than in 1F1B), and correctness bookkeeping — because weight updates from deferred W chunks can, in edge cases, land after a device has technically moved on to a later training step, the scheduling has to guarantee the optimizer still sees numerically correct, ordered updates. This is a fundamentally different lever from interleaving: interleaving shrinks the bubble by making each idle wait shorter; zero-bubble scheduling fills the idle time with genuinely useful work that was always available but never placed there.

A misconception worth correcting

Because 1F1B reproduces GPipe's exact bubble-fraction formula, (p − 1)/(m + p − 1), it is tempting to conclude that switching from GPipe's ordering to 1F1B is cosmetic — same formula, same performance, no real reason to bother. This is wrong, and the error is mistaking "same wall-clock time" for "same resource cost." The two schedules take identical total time because the warm-up/cool-down structure is fixed by p and m, not by execution order. But the peak activation memory each schedule demands is not the same: GPipe forces every stage to hold activations for all m in-flight microbatches, while 1F1B bounds that to p, the pipeline depth, regardless of how large m grows. In practice this is the difference between a training run that fits on your GPUs and one that doesn't — or between being forced to use a small m (and therefore accept a large bubble fraction) versus being free to push m up specifically to shrink the bubble, because memory is no longer the limiting factor. The bubble-fraction formula answers "how much time is wasted"; it says nothing about "how much memory is spent," and for large models the second question is often the one that decides whether the job runs at all.

Active recall

Attempt these before reading the answers.

1. Why does 1F1B achieve exactly the same bubble fraction as GPipe, despite reordering when each device runs its forward and backward passes?
2. For p = 6 devices, m = 12 microbatches, t_f = 1, t_b = 2: compute the bubble fraction and total time under plain 1F1B. Then suppose the team switches to an interleaved schedule with v = 3. Recompute the bubble fraction, the total time, and state what happens to peak activation memory.
3. Suppose backward actually cost the same as forward (t_b = t_f, not 2t_f) for some architecture. Does the bubble fraction change? Does the absolute wall-clock time change? Justify both answers from the formulas.
4. Starting from p = 4, m = 8, v = 2, the team doubles the microbatch count to m = 16 while keeping v = 2. What happens to the bubble fraction? Does peak activation memory under the interleaved 1F1B schedule grow the way it would under plain GPipe?
5. In zero-bubble pipeline parallelism, why can the weight-gradient computation (W) be scheduled into arbitrary idle slots, while the input-activation-gradient computation (B) cannot?
6. A cluster splits pipeline stages across physical nodes connected by standard Ethernet rather than NVLink. Why might this team choose v = 2 over v = 8, even though the bubble-fraction formula says v = 8 shrinks the bubble more?

Answers

1. The bubble fraction is a consequence of the pipeline's warm-up and cool-down ramps — the p − 1 steps it takes to fill the pipeline and the p − 1 steps it takes to drain it — which are determined purely by pipeline depth p and microbatch count m, not by the order in which a single device interleaves its own forward and backward work. 1F1B changes only when a device's backward pass runs relative to its own future forward passes (freeing activation memory earlier); it does not change how many total device-idle steps the dependency chain across stages forces. Total time (m + p − 1)(t_f + t_b) and hence the bubble fraction are unaffected.

2. Plain 1F1B (v=1): total = (12 + 5)(1+2) = 17 · 3 = 51; ideal = 12 · 3 = 36; bubble = 15; fraction = 15/51 = 5/17 ≈ 29.4%. With v=3: total = 3 · (12 + 5/3) = 3 · 13.667 = 41; ideal stays 36; bubble = 5; fraction = 5/41 ≈ 12.2% (cross-checks against the closed form (p−1)/(vm+p−1) = 5/(36+5) = 5/41). So the bubble fraction drops from 29.4% to 12.2% and total time falls from 51 to 41 units — a genuine 19.6% wall-clock improvement. Peak activation memory: under plain 1F1B it was bounded by p = 6, independent of m = 12. Under the interleaved schedule it remains independent of m — it does not grow the way GPipe's would if m increased — though the constant bound sits somewhat above 6, since each device now has 3 concurrent chunk pipelines to keep activations alive for.

3. The bubble fraction does not change. The formula (p−1)/(m+p−1) (or its interleaved generalization) has no t_f or t_b terms at all — they cancel because both the total time and the ideal time scale by the same factor (t_f + t_b), and the fraction is a ratio of the two. What does change is the absolute wall-clock time: with t_b = t_f = 1 instead of t_b=2, for p=4, m=8, total time becomes (11)(1+1) = 22 instead of 33 — faster in real seconds — but the bubble still eats exactly 3/11 ≈ 27.3% of it, unchanged.

4. Bubble fraction: (p−1)/(vm+p−1) = 3/(2·16+3) = 3/35 ≈ 8.6%, down from 15.8% at m=8 — shrinking further, as expected, since increasing m always dilutes the fixed warm-up/cool-down cost. (Note this matches the earlier v=4, m=8 result exactly, because the formula depends on m and v only through their product, and 2×16 = 4×8 = 32 in both cases.) Peak activation memory: under interleaved 1F1B, memory stays bounded by p and v, not by m — doubling m from 8 to 16 does not double the buffered-activation count the way it would under naive GPipe, where peak memory equals m directly and would jump from 8 to 16. This is precisely the property that lets teams grow m to shrink the bubble without paying a proportional memory tax.

5. B produces the gradient with respect to the layer's input — the exact quantity the upstream (earlier) pipeline stage needs to continue running its own backward pass. Skipping or delaying B stalls every stage behind it in the dependency chain, so it must run in its normal position. W produces the gradient with respect to that layer's own weights; nothing outside that device — no upstream stage, no downstream stage — ever consumes it. Its only consumer is that device's own optimizer step at the end of the training iteration, so it can be computed at any point before then, including inside what would otherwise be idle bubble time.

6. Interleaving with depth v multiplies the number of point-to-point activation/gradient transfers between devices roughly by v, because the model is now chopped into p·v pieces instead of p, and each piece boundary is a cross-device communication. NVLink-connected GPUs within a node have enough bandwidth that this extra traffic barely registers, so large v is close to free. Across nodes on Ethernet, bandwidth and latency are far more constrained, so the communication overhead from v=8 can easily exceed the compute time saved by the smaller bubble, making the pipeline network-bound instead of compute-bound. Choosing v=2 accepts a smaller bubble-fraction improvement in exchange for keeping communication volume low enough that the network doesn't become the new bottleneck — the mathematically optimal v depends on interconnect bandwidth, not on the bubble formula alone.

Think About It

Think about this: How would you explain pipeline parallelism: minimizing bubble overhead to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind pipeline parallelism: minimizing bubble overhead, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.