Picture an ML engineer at a Bengaluru startup fine-tuning a 1.3-billion-parameter Hindi language model on a single RTX 4090 — 24 GB of VRAM, the kind of card the team can actually afford, not an 80 GB A100 rented by the hour. The training script works perfectly for a batch size of 4. Push it to a batch size of 64, the size the paper they are replicating used to get stable, low-variance training, and the process dies with CUDA out of memory before a single optimizer step completes. The GPU has not become slower. It has simply run out of a specific, quantifiable resource: memory to hold the intermediate activations of every example in the batch, all at once, for the backward pass. Renting a bigger card is one option — and an expensive, recurring one in Indian cloud pricing. The other option costs nothing extra: gradient accumulation, a way to get the exact gradient a batch of 64 would have produced, using hardware that only has room for 4 at a time.
This chapter builds that technique from the definition of a mini-batch gradient upward, proves the accumulated gradient is numerically identical to the large-batch gradient, walks a full worked example by hand, traces real code, and corrects the misconception almost everyone forms about what this trick actually buys you.
Why batch size costs memory in the first place
In mini-batch gradient descent, one training step needs the gradient of the loss with respect to every trainable weight, averaged over the batch. To compute that gradient, backpropagation needs the intermediate activation of every layer, for every example in the batch, because the chain rule at each layer multiplies the incoming gradient by that layer's local Jacobian, which depends on the forward-pass values it saw. Those activations cannot be discarded after the forward pass — they are the very quantities the backward pass differentiates through — so they sit in GPU memory from the moment they are computed until the backward pass consumes them.
This is the part students usually miss: model weights and their gradients occupy a fixed amount of memory regardless of batch size — a 1.3B-parameter model has 1.3B weights and 1.3B gradient values whether you feed it 1 example or 1000. What scales with batch size is the activation memory: every layer's output, for every example, held simultaneously. Double the batch size and you roughly double the activation memory, while the weight and gradient memory stays flat.
As a rough order-of-magnitude illustration (implementation details shift this, so treat it as an engineering rule of thumb, not a measured number): mixed-precision training with the Adam optimizer typically costs about 2 bytes per parameter for FP16 weights, 2 bytes per parameter for FP16 gradients, and roughly 8 bytes per parameter for Adam's two FP32 moment buffers — around 12 bytes of fixed overhead per parameter before a single activation is stored. For 1.3 billion parameters, that is roughly 1.3 × 10⁹ × 12 bytes ≈ 15.6 GB, already gone from a 24 GB card before training has produced anything. What remains — under 9 GB — has to hold every activation for the whole batch. That is the wall the engineer hits, and it is a wall that grows linearly with batch size, not with model size.
The identity gradient accumulation exploits
The loss for a batch of N examples is almost always defined as the mean of the per-example losses:
L(w) = (1/N) · Σᵢ₌₁ᴺ ℓᵢ(w)
Differentiation is linear, so the gradient of a sum is the sum of the gradients, and a constant factor pulls straight out:
∇L(w) = (1/N) · Σᵢ₌₁ᴺ ∇ℓᵢ(w)
Now split the batch into K disjoint micro-batches of size M each, so N = K·M. Rewrite the sum over all N examples as K partial sums, one per micro-batch:
∇L(w) = (1/N) · Σₖ₌₁ᴷ ( Σᵢ ∈ micro-batch k ∇ℓᵢ(w) )
Each inner term, Σᵢ ∈ micro-batch k ∇ℓᵢ(w), is exactly M times the *mean* gradient of that micro-batch alone — the quantity a normal forward-backward pass on that micro-batch already computes. So if you run K separate forward-backward passes, one per micro-batch, scale each micro-batch's mean loss by 1/K before calling backward, and let the resulting gradients add into the same buffer instead of overwriting it, the sum you accumulate is mathematically identical to ∇L(w) computed on the full batch of N examples in one shot. No approximation, no averaging trick with hidden error — it is the same real number, up to ordinary floating-point rounding. That identity is the entire mechanism. Gradient accumulation is not an approximation to large-batch training; it is large-batch training, computed in installments.
Worked example: four points, two micro-batches
Take the smallest case that shows every step. A single-weight linear model ŷ = w·x, no bias, current weight w = 1. Four training points following the true relation y = 2x + 1:
x = [1, 2, 3, 4], y = [3, 5, 7, 9]
Use per-example squared error ℓᵢ = (w·xᵢ − yᵢ)², so the batch loss is the mean of these four values, and the per-example gradient is dℓᵢ/dw = 2·xᵢ·(w·xᵢ − yᵢ).
Full batch, computed directly (the target we must match). Predictions at w = 1 are [1, 2, 3, 4]; errors (ŷᵢ − yᵢ) are [1−3, 2−5, 3−7, 4−9] = [−2, −3, −4, −5]. Per-example gradients dℓᵢ/dw = 2xᵢ(ŷᵢ−yᵢ):
- i = 1: 2 · 1 · (−2) = −4
- i = 2: 2 · 2 · (−3) = −12
- i = 3: 2 · 3 · (−4) = −24
- i = 4: 2 · 4 · (−5) = −40
The true full-batch gradient is the mean of these four numbers: (−4 − 12 − 24 − 40) / 4 = −80 / 4 = −20. That is the number any correct implementation of gradient accumulation must reproduce.
Now split into two micro-batches of size 2 (K = 2, M = 2), which is what a memory-limited GPU forces. Following the standard recipe — scale each micro-batch's mean loss by 1/K before calling backward, and never zero the gradient buffer between micro-batches:
Micro-batch 1 = examples 1, 2. Mean gradient of this micro-batch alone (mean of the per-example gradients, exactly what one ordinary backward pass on these two points computes): (−4 + −12) / 2 = −8. Scaled by 1/K = 1/2, the contribution written into the gradient buffer is −8 × (1/2) = −4.
Micro-batch 2 = examples 3, 4. Mean gradient: (−24 + −40) / 2 = −32. Scaled by 1/2: −32 × (1/2) = −16.
Because the two backward passes accumulate rather than overwrite, the buffer after both micro-batches holds −4 + (−16) = −20 — exactly the full-batch gradient computed above, to the last digit. This is not a coincidence tuned for round numbers; it is the linearity identity from the previous section, verified by direct arithmetic on both sides.
Tracing the code
The same computation, expressed the way it is written in practice:
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
y = torch.tensor([3.0, 5.0, 7.0, 9.0])
w = torch.tensor(1.0, requires_grad=True)
micro_batch_size = 2
accumulation_steps = 2 # effective batch size = 4
for step in range(accumulation_steps):
start = step * micro_batch_size
end = start + micro_batch_size
xb, yb = x[start:end], y[start:end]
y_hat = w * xb
loss = ((y_hat - yb) ** 2).mean() / accumulation_steps
loss.backward() # adds into w.grad, does not overwrite
print(w.grad) # tensor(-20.)
# only now, after every micro-batch is folded in, does the model move
optimizer_step_would_use = w.grad.item() # -20.0
Trace it line by line against the hand computation above. On step = 0, xb, yb are examples 1 and 2; loss is their mean squared error (6.5) divided by 2, giving 3.25; calling .backward() computes d(3.25)/dw and writes it into w.grad, which is −4, matching the hand trace exactly. On step = 1, xb, yb are examples 3 and 4; the scaled loss is 10.25; the crucial detail is that PyTorch's .backward() adds the newly computed gradient into whatever is already sitting in .grad, rather than replacing it — this accumulate-by-default behaviour is precisely why optimizer.zero_grad() exists as a separate call in every training loop you have seen, and precisely why this technique is named "accumulation." After step 1, w.grad holds −4 + (−16) = −20. Only after the loop — after all K micro-batches have contributed — does the real training loop call optimizer.step() to actually move the weight, followed by optimizer.zero_grad() to reset the buffer to zero before the next effective batch begins. Calling optimizer.step() or zero_grad() inside the loop, once per micro-batch, is the single most common bug in a first implementation — it either updates the weight K times too often using partial gradients, or wipes out the accumulated sum before it is complete.
The diagram: memory budget and the accumulation loop
The misconception: accumulation does not make training faster
Almost every student who first sees this technique assumes it is a pure win with no downside — "run K small passes instead of 1 big pass, get the same gradient, so why wouldn't you always do this?" The gap is speed. A single true batch of N = 512 examples runs as one large matrix multiplication that saturates the GPU's parallel compute units efficiently. K = 64 sequential micro-batches of size 8 run 64 separate, smaller matrix multiplications, each under-using the GPU's parallelism, plus 64 separate rounds of Python/framework overhead for launching kernels. The total floating-point work is the same either way, but the wall-clock time for K small sequential passes is typically similar to or slightly worse than one large pass — never better. Gradient accumulation trades GPU memory for wall-clock time and framework overhead; it does not trade memory for nothing. What accumulation actually buys is access, not speed: it lets an engineer with a 24 GB card reach a gradient estimate — and the training stability, generalization behaviour, and learning-rate schedule tuned for large-batch training — that would otherwise be completely unreachable on that hardware, full stop, regardless of how long they were willing to wait. On a card with unlimited memory, gradient accumulation would offer nothing, because you would simply run the whole batch at once and get the identical gradient faster. The technique exists purely because memory, not compute, is the binding constraint for large models on affordable GPUs. One further correctness caveat worth carrying forward: this exact-equivalence proof assumes every layer's computation is independent per example. Layers with cross-example statistics, most notably Batch Normalization, compute their running mean and variance from whatever is in the current micro-batch, not the full effective batch — so a model with BatchNorm layers trained with accumulation is not perfectly identical to the same model trained on one true large batch, even though the weight gradient is. Layer Normalization and RMSNorm, used in almost all modern transformer-based LLMs, have no such issue, since they normalize within a single example rather than across the batch — one reason accumulation is the default large-model training technique today.
Choosing the accumulation schedule
In practice the recipe is: find the largest micro-batch size M that fits in memory by binary search (try M, halve on OOM, double if there is headroom), decide the effective batch size N the training recipe calls for, and set K = ⌈N / M⌉. Two bookkeeping details trip up almost every first implementation. First, learning-rate warmup schedules and logging are usually defined in units of optimizer steps, not micro-batches — a warmup of "500 steps" means 500 calls to optimizer.step(), i.e. 500 × K forward-backward passes, so miscounting which loop variable increments the step counter silently stretches or compresses the warmup period. Second, on multi-GPU setups using distributed data-parallel training, each micro-batch's backward pass ordinarily triggers a network-wide gradient synchronization (an all-reduce) across GPUs; doing that after every micro-batch instead of once per effective batch wastes bandwidth for no benefit, since only the final accumulated gradient is used. Production frameworks expose a context (PyTorch DDP's no_sync(), or automatically inside libraries like Hugging Face Accelerate and DeepSpeed) that skips synchronization on every micro-batch except the last one in each accumulation cycle — the memory-for-time trade this chapter describes, applied once more at the network level instead of the single-GPU memory level.
Active recall
Attempt these before reading the answers.
- A model trains with micro-batch size 16 and accumulation_steps = 8. What effective batch size does the optimizer actually see per update?
- Why must the per-micro-batch loss be divided by accumulation_steps before calling
.backward(), rather than left as the plain mean loss of that micro-batch? - A student writes a training loop that calls
optimizer.zero_grad()at the start of every micro-batch iteration, inside the accumulation loop. What breaks, concretely? - True or false, with justification: gradient accumulation trains a model in less wall-clock time than fitting the full large batch directly on a bigger GPU.
- For a model that uses BatchNorm layers, is the accumulated gradient after K micro-batches exactly equal to the gradient a true large batch of size N = K·M would have produced? Justify using the linearity argument from this chapter.
- You need an effective batch size of 2048 and the largest micro-batch that fits in memory is 24. What is the minimum number of accumulation steps, and what effective batch size does that setting actually realize?
Answers.
1. Effective batch size = micro_batch_size × accumulation_steps = 16 × 8 = 128. The optimizer applies one update per 128 examples' worth of accumulated gradient, even though no single forward pass ever holds more than 16 examples in memory.
2. Without dividing by accumulation_steps, each micro-batch's backward pass would write its own mean gradient (already correctly averaged over that micro-batch) into the buffer, and K of those means would sum to K times the true full-batch mean — the effective gradient magnitude, and therefore the effective learning rate, would scale up by a factor of K compared to what the chosen learning rate was tuned for, typically causing instability or divergence. Dividing each micro-batch's loss by K before calling backward is exactly what cancels this factor, as shown algebraically in the derivation: (1/N)Σᵢℓᵢ splits into K terms each already carrying a 1/N = 1/(KM) factor, which the per-micro-batch mean (a 1/M factor) only partially supplies — the remaining 1/K has to be applied explicitly.
3. Calling zero_grad() inside the loop wipes the buffer before the next micro-batch's gradient is added, so by the time optimizer.step() finally runs, w.grad holds only the last micro-batch's contribution, not the sum of all K. The optimizer silently takes a step using a fraction of the intended gradient (in the worked example, it would use only −16 instead of −20), so training proceeds — no crash, no error message — but on a systematically wrong, smaller effective batch than requested, which is exactly the class of bug the "measure, don't assume" discipline exists to catch: nothing about the log output announces that the effective batch size silently shrank.
4. False. As explained in the misconception section, the total floating-point work is identical either way, but K sequential small matrix multiplications under-use GPU parallelism and add K rounds of framework/kernel-launch overhead compared to one large matrix multiplication, so a true large batch on hardware with enough memory to hold it is normally faster or equal, never slower, than the accumulated equivalent. Accumulation buys memory headroom, not speed.
5. No, not exactly, and the reason is a scope limit on the linearity proof, not an error in it. The derivation ∇L(w) = (1/N)Σᵢ∇ℓᵢ(w) treats each ℓᵢ as depending only on example i's own forward pass. BatchNorm layers violate that assumption: their normalization statistics (mean and variance) are computed from whichever examples are physically present in the current micro-batch, so ℓᵢ computed inside a micro-batch of 24 is a different function of the weights than ℓᵢ computed inside a true batch of 2048 — the per-example loss itself changes, not just how the gradients are summed. Every operation with no cross-example dependency (linear layers, attention, LayerNorm, RMSNorm) is completely unaffected and the identity holds exactly for their contribution to the gradient.
6. Minimum accumulation steps K = ⌈2048 / 24⌉ = ⌈85.33⌉ = 86. That realizes an effective batch size of 86 × 24 = 2064, not exactly 2048 — the last micro-batch of the cycle would typically be trimmed to 8 examples (85 full micro-batches of 24 plus one of 8) to land on exactly 2048, or the target is relaxed to accept 2064 as close enough, depending on how strict the training recipe is about the exact number.
Think About It
Think about this: How would you explain gradient accumulation: training large models on small gpus to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind gradient accumulation: training large models on small gpus, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.