Distributed Training: Scaling Deep Learning Across GPUs

Why One GPU Runs Out of Room

Every March, CBSE evaluators face a problem of scale: crores of answer scripts have to be checked in about six weeks. No single examiner reads every script — that would take years. Instead, boards distribute papers across thousands of examiners, each using the identical marking scheme, each grading a different pile. Periodically, moderation meetings pull examiners together to compare notes so that one examiner's harsh marking and another's generous marking don't quietly diverge into two different standards. This is not a decorative comparison — it is, almost exactly, the mechanism this chapter teaches. When you train a neural network too large or too slow to fit on one processor, you replicate the "marking scheme" (the model's weights) across many workers, split the "papers" (the training batch) between them, and periodically force the workers to agree on how much each weight should change.

The real constraint is memory and time, not ambition. A single NVIDIA A100 GPU carries 80 GB of memory. GPT-2 (2019) had about 1.5 billion parameters; GPT-3 (2020) had 175 billion. Training with the Adam optimizer in mixed precision — the industry-standard recipe — costs roughly 16 bytes of GPU memory per parameter: 2 bytes for an fp16 copy of the weight, 2 bytes for its fp16 gradient, and 4+4+4 bytes for an fp32 master weight plus Adam's two running-average buffers (momentum and variance). For a 175-billion-parameter model that is 175,000,000,000 × 16 bytes = 2,800,000,000,000 bytes ≈ 2.8 TB — just for optimizer state, before a single activation tensor is stored. Spread across 80 GB GPUs with zero redundancy, that alone needs at least 2800 ÷ 80 = 35 GPUs. One GPU was never in the running. The question this chapter answers is: given many GPUs, exactly how do you split the computation, and what does it cost you to keep them consistent?

Data Parallelism: Same Model, Different Data

The simplest and by far the most common strategy mirrors the CBSE analogy directly. Every GPU holds a complete, identical copy of the model. The global training batch is split into shards — one shard per GPU. Each GPU runs its own forward pass and backward pass independently, on its own shard, producing its own gradient tensor. If the GPUs stopped here and each applied its own gradient, the four replicas would drift apart after one step, exactly like examiners who never attend a moderation meeting: same starting rubric, different final standards. So before any weight is updated, the GPUs must synchronize — average their gradients — so that every replica applies the identical update and stays byte-for-byte identical to every other replica. This synchronization step is called an all-reduce, and how efficiently it is implemented determines whether adding GPUs actually makes training faster.

Synchronizing Replicas: Ring All-Reduce

The naive way to average N gradient tensors is to send them all to one GPU, sum them there, and broadcast the result back — but that GPU's network link becomes a bottleneck as the group grows, since it alone handles all incoming and outgoing traffic. The standard fix, used by NCCL (NVIDIA's collective communication library) inside PyTorch's DistributedDataParallel and TensorFlow's MirroredStrategy, is the ring all-reduce. The P GPUs are arranged in a logical ring. Each GPU's gradient tensor is cut into P equal chunks. The algorithm runs in two phases, each taking P−1 steps:

Reduce-scatter: at each step, every GPU sends one chunk to its ring-neighbor and adds an incoming chunk from its other neighbor into its own copy of that chunk. After P−1 steps, each GPU holds the fully-summed value for exactly one chunk (a different chunk on each GPU).

All-gather: the GPUs pass those completed chunks around the same ring for another P−1 steps, this time simply overwriting rather than adding, until every GPU holds every fully-summed chunk.

The diagram below shows this ring for P = 4 GPUs, using small toy gradient vectors so every number can be checked by hand.

Trace it step by step. Each GPU starts with a 4-element gradient chunked into 4 pieces of 1 element each:

Start:
GPU0: [1, 2, 3, 4]      GPU1: [5, 6, 7, 8]
GPU2: [9, 10, 11, 12]   GPU3: [13, 14, 15, 16]

Reduce-scatter, step 1 (each GPU sends chunk i, adds into neighbor):
GPU0: [1, 2, 3, 20]     GPU1: [6, 6, 7, 8]
GPU2: [9, 16, 11, 12]   GPU3: [13, 14, 26, 16]

Reduce-scatter, step 2:
GPU0: [1, 2, 29, 20]    GPU1: [6, 6, 7, 28]
GPU2: [15, 16, 11, 12]  GPU3: [13, 30, 26, 16]

Reduce-scatter, step 3 (done — each GPU now fully owns one chunk):
GPU0: [1, 32, 29, 20]   <- index 1 fully summed: 32
GPU1: [6, 6, 36, 28]    <- index 2 fully summed: 36
GPU2: [15, 16, 11, 40]  <- index 3 fully summed: 40
GPU3: [28, 30, 26, 16]  <- index 0 fully summed: 28

All-gather, step 1 (send the completed chunk, overwrite on arrival):
GPU0: [28, 32, 29, 20]  GPU1: [6, 32, 36, 28]
GPU2: [15, 16, 36, 40]  GPU3: [28, 30, 26, 40]

All-gather, step 2:
GPU0: [28, 32, 29, 40]  GPU1: [28, 32, 36, 28]
GPU2: [15, 32, 36, 40]  GPU3: [28, 30, 36, 40]

All-gather, step 3 (done — every GPU now identical):
GPU0: [28, 32, 36, 40]  GPU1: [28, 32, 36, 40]
GPU2: [28, 32, 36, 40]  GPU3: [28, 32, 36, 40]

Check the totals directly: position 0 should be 1+5+9+13 = 28 ✓, position 1 should be 2+6+10+14 = 32 ✓, position 2 should be 3+7+11+15 = 36 ✓, position 3 should be 4+8+12+16 = 40 ✓. Every GPU performed exactly 6 send/receive operations (2(P−1) with P=4), moving one element each time — which is where the general formula comes from: for a tensor of total size S split across P GPUs, the total data each GPU transmits over the full ring all-reduce is 2(P−1)/P × S. To turn the summed gradient into the averaged gradient every optimizer step actually needs, divide by P:

gpu_grads = [
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12],
    [13, 14, 15, 16],
]
avg_grad = [sum(vals) / len(gpu_grads) for vals in zip(*gpu_grads)]
print(avg_grad)

Tracing it: zip(*gpu_grads) transposes the four lists into column tuples (1,5,9,13), (2,6,10,14), (3,7,11,15), (4,8,12,16); summing and dividing by 4 gives 28/4=7.0, 32/4=8.0, 36/4=9.0, 40/4=10.0. The printed output is exactly [7.0, 8.0, 9.0, 10.0] — the identical averaged gradient that every GPU now applies to its identical weight copy, keeping all replicas in lockstep. In real training frameworks, this is triggered automatically:

import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

dist.init_process_group(backend="nccl")
local_rank = dist.get_rank()
torch.cuda.set_device(local_rank)

model = MyTransformer().to(local_rank)
model = DDP(model, device_ids=[local_rank])
# model, optimizer, and my_shard_of_data are assumed already constructed

for batch in my_shard_of_data:
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()   # DDP fires ring all-reduce here, bucket by bucket,
                       # overlapped with the still-running backward pass
    optimizer.step()  # every replica applies the identical averaged gradient

Worked Example: Does Doubling GPUs Halve Training Time?

Take a 500-million-parameter model training in fp32, so its gradient tensor is 500,000,000 × 4 bytes = 2,000,000,000 bytes = 2 GB (decimal GB, matching how network bandwidth is quoted). Suppose 8 GPUs are connected by 200 Gbps InfiniBand, which is 200 ÷ 8 = 25 GB/s. Using the ring formula, the data each GPU must transmit is:

2 × (P − 1) / P × S = 2 × 7/8 × 2 GB = 3.5 GB per GPU

At 25 GB/s, communication takes 3.5 ÷ 25 = 0.14 s = 140 ms. Now suppose each GPU's forward-plus-backward pass on its shard takes 300 ms. If communication were not overlapped with computation at all, one training step costs 300 + 140 = 440 ms. Compare this to what a single GPU would need to process the same total global batch serially — 8 shards, one after another — at 300 ms per shard: 8 × 300 = 2400 ms. The actual speedup from 8 GPUs is:

speedup = 2400 / 440 ≈ 5.45×
efficiency = 5.45 / 8 ≈ 68%

Eight GPUs deliver roughly 5.45× the throughput of one — not 8×. In practice, DDP overlaps the all-reduce with the backward pass (gradients for the last layers finish first and start syncing while earlier layers are still being differentiated), which recovers much of that 140 ms and pushes real efficiency higher than this pessimistic estimate. But the overlap is never total, and on slower interconnects (Ethernet instead of InfiniBand, or GPUs split across separate machines) or with more GPUs in the ring, the communication term grows relative to compute and efficiency drops further.

Common Misconception

The mistake nearly every student makes on first encountering this topic is assuming "N GPUs means N× faster training." The 68% efficiency figure above is the direct counter-example: doubling GPU count from 1 to 8 gave 5.45×, not 8×, purely because of the ring all-reduce's non-zero communication cost. This is not an implementation flaw to be patched away — it is a structural fact about distributed systems, the same reason a group project doesn't finish in 1/N the time just because you added N−1 teammates: coordination has a cost that scales with the group. The correct mental model is that adding GPUs helps until communication overhead catches up with the compute you saved, and the interconnect quality (NVLink within one server, versus InfiniBand or Ethernet across servers) is what decides how many GPUs you can add before diminishing returns dominate. It is also why data parallelism by itself cannot rescue a model too large to fit in one GPU's memory in the first place — replicating a model that doesn't fit does not make it fit, it only fails on every GPU simultaneously instead of one.

Beyond Data Parallelism: Tensor and Pipeline Parallelism

When a model's weights alone exceed a single GPU's memory — recall the 175-billion-parameter model needing 2.8 TB of optimizer state and at least 35 GPUs just to hold it — data parallelism cannot help, since every replica still needs the full model. Two other splitting strategies address this directly.

Tensor (intra-layer model) parallelism, used by Megatron-LM, cuts individual weight matrices apart. For a linear layer computing Y = X·W, split W column-wise across two GPUs as W = [W1 | W2]; GPU A computes X·W1 and GPU B computes X·W2, each producing half the output columns — no GPU ever needs to hold the full weight matrix in memory. Megatron-LM alternates this "column-parallel" split with a complementary "row-parallel" split in the very next linear layer of a transformer block, arranged so the two GPUs only need to exchange (all-reduce) a single activation tensor once per block, rather than after every matrix multiply. Because this exchange happens after every layer, tensor parallelism needs the fastest possible link — it is normally kept within one server over NVLink (900 GB/s-class bandwidth), never stretched across a slower network.

Pipeline (inter-layer) parallelism instead assigns whole contiguous blocks of layers to different GPUs — GPU 0 holds layers 1–8, GPU 1 holds layers 9–16, and so on — and activations flow GPU-to-GPU like an assembly line. The naive version leaves every GPU except the one currently active sitting idle, called a pipeline bubble. GPipe's fix is to split each mini-batch into M smaller micro-batches and feed them into the pipeline back-to-back, so while GPU 1 processes micro-batch 1, GPU 0 is already starting micro-batch 2. The fraction of time still lost to the bubble is (P − 1) / (M + P − 1), where P is the number of pipeline stages. With P = 4 stages and M = 8 micro-batches, the bubble fraction is 3 / 11 ≈ 27.3% — still substantial. Push M up to 32 micro-batches and it falls to 3 / 35 ≈ 8.6%, showing why production pipelines use many small micro-batches rather than few large ones.

Real large-model training systems (Megatron-LM, DeepSpeed) combine all three axes at once — commonly called 3D parallelism — matching each split to the interconnect available at that boundary: tensor-parallel within a server over NVLink, pipeline-parallel across servers, and data-parallel across whole replica groups on top. DeepSpeed's ZeRO optimizer adds a fourth idea that composes with data parallelism rather than replacing it: instead of every data-parallel GPU holding its own full 16-bytes-per-parameter optimizer state, ZeRO shards that state across the data-parallel group itself, so the 2.8 TB figure for a 175B model gets divided by however many GPUs are in the group rather than duplicated on each one.

Active Recall

Attempt each question before reading its answer.

Q1. A model has 2 billion parameters, trained mixed-precision with Adam (16 bytes/parameter for optimizer state). Each GPU has 24 GB of memory. What is the minimum number of GPUs needed to shard the optimizer state alone, with zero redundancy?

Q2. A gradient tensor of total size 1.2 GB is ring-all-reduced across P = 6 GPUs. Using 2(P−1)/P × S, how much data does each GPU transmit in total?

Q3. Using the CBSE-examiner analogy, explain in one sentence each how data parallelism and tensor parallelism differ.

Q4. Why can adding more GPUs to a data-parallel job produce sub-linear rather than linear speedup?

Q5. Each GPU in a 4-GPU data-parallel job takes 250 ms to compute its shard's forward+backward pass. Ring all-reduce communication (unoverlapped) takes 40 ms. Compute the speedup over an equivalent single-GPU serial run, and the resulting efficiency.

Q6. True or false: "Pipeline parallelism removes the need for data parallelism once a model is large enough." Justify your answer.

Answers.

A1. Total optimizer memory = 2,000,000,000 × 16 = 32,000,000,000 bytes = 32 GB. One 24 GB GPU cannot hold it (24 < 32); two GPUs give 48 GB of combined capacity, which is enough. Minimum is 2 GPUs.

A2. 2 × (6−1)/6 × 1.2 GB = 2 × 5/6 × 1.2 = 10/6 × 1.2 = 2.0 GB. Each GPU transmits 2 GB total across the full ring all-reduce.

A3. Data parallelism gives every examiner (GPU) the same complete marking scheme (model) but a different pile of scripts (batch shard) to grade independently. Tensor parallelism instead splits the marking of a single script itself — one examiner checks certain questions, another checks the rest of the same script — so no single GPU needs to hold the entire model.

A4. Because synchronizing gradients (or activations, for pipeline/tensor parallelism) costs communication time that does not shrink to zero as GPU count grows; once that communication time becomes comparable to or larger than the compute time it is overlapped with, adding more GPUs yields progressively smaller returns, as shown numerically in the worked example (68% efficiency at P = 8, not 100%).

A5. Equivalent single-GPU serial time = 4 × 250 = 1000 ms. Actual 4-GPU step time = 250 + 40 = 290 ms. Speedup = 1000/290 ≈ 3.45×. Efficiency = 3.45/4 ≈ 86.2%.

A6. False. Pipeline parallelism solves the memory problem (fitting layers that don't fit on one GPU) but not the throughput problem of processing a large dataset quickly — production systems like Megatron-LM and DeepSpeed run pipeline-parallel and tensor-parallel groups internally, then replicate that entire group with data parallelism to scale further, which is exactly the 3D-parallelism combination described above.

Think About It

Think about this: How would you explain distributed training: scaling deep learning across gpus to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind distributed training: scaling deep learning across gpus, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.