Mixed Precision Training: Float16 and Beyond

A Training Run That Quietly Stops Learning

A research group training a mid-sized transformer language model on a V100 cluster notices something strange around step 40,000. The loss curve, which had been falling smoothly, flattens. Not diverges — flattens, as if the model had simply stopped absorbing information from the data. Gradient norms logged every 100 steps are near zero for entire layers. There is no crash, no NaN, no error message. The run looks healthy on every dashboard except the one that matters: the model is not getting better. This is the signature failure of naive float16 training, and it is more insidious than the dramatic NaN-and-halt failure most tutorials show, because nothing alerts you. The optimizer keeps stepping, checkpoints keep saving, and the wasted GPU-hours keep accumulating until someone plots validation loss against a known-good FP32 baseline and finds a gap that should not exist.

The V100's tensor cores, introduced by NVIDIA in 2017, compute matrix multiplies in FP16 with FP32 accumulation — the hardware reason mixed precision training exists at all. But FP16 has a narrow floating-point exponent range, and small gradients simply vanish inside it before the optimizer ever sees them. This chapter works through exactly why that happens at the bit level, how the standard fix (loss scaling) operates as an automatic control loop rather than a single knob, and what changed when NVIDIA's A100 (2020) and H100 (2022) generations moved the field to bfloat16 and then to 8-bit floating point. If you have already studied mixed precision training's memory and throughput benefits, this chapter is the numerics underneath — the part that explains why the technique needs a scaling factor at all, and why "just use a smaller dtype" gets harder, not easier, as the dtype shrinks further.

What Sixteen Bits Actually Encode

IEEE 754 half precision (FP16) spends its 16 bits as 1 sign bit, 5 exponent bits, and 10 mantissa (fraction) bits, with an exponent bias of 15. A normal (non-subnormal) value is reconstructed as (-1)^sign × 1.mantissa × 2^(exponent_field - 15). Five exponent bits means only 30 usable exponent field values (1 through 30; 0 and 31 are reserved for subnormals and inf/NaN), giving a dynamic range from a smallest normal magnitude of 2⁻¹⁴ ≈ 6.10 × 10⁻⁵ up to a largest magnitude of roughly 65,504. Ten mantissa bits give about 3 decimal digits of precision wherever a value falls inside that range.

Compare that to FP32: 1 sign bit, 8 exponent bits (bias 127), 23 mantissa bits. The exponent range is enormous — down to 2⁻¹²⁶ ≈ 1.18 × 10⁻³⁸ — which is why gradients that are numerically tiny in FP32 (common in the first few layers of a deep network, or in attention logits after softmax saturation) are still comfortably representable there. Shrink the exponent field to 5 bits and that safety margin disappears. A gradient that FP32 stores without complaint can fall below FP16's smallest representable magnitude entirely, and rounds to exactly zero — not "loses some precision," but is erased.

The diagram below lays out the bit-field structure of FP32, FP16, BF16, and the two 8-bit floating-point formats introduced with Hopper-generation tensor cores, drawn to scale by bit width, together with how far each format's exponent field can reach on a log scale.

Two things should jump out. First, FP16 and FP8-E5M2 have identical exponent fields (5 bits, bias 15) — E5M2 was deliberately designed to inherit FP16's dynamic range exactly, trading two of FP16's ten mantissa bits away to make room for the extra sign/exponent bit that a full byte needs elsewhere. Second, on a log scale, the FP16/E5M2 exponent range and the FP8-E4M3 exponent range are both a sliver compared to FP32/BF16's span. That sliver is the entire reason mixed precision training needs a compensating mechanism at all.

Worked Example: Where a Gradient Goes Missing

Take a concrete gradient value that a real backward pass could produce for a weight several layers deep in a transformer: g = 4.5 × 10⁻⁶. This is smaller than FP16's smallest normal value (6.10 × 10⁻⁵), so it must be represented, if at all, as a subnormal — a number with the reserved exponent field 00000, where the value is mantissa/1024 × 2⁻¹⁴. The quantization step size for subnormals is therefore fixed at 2⁻¹⁴ / 1024 = 2⁻²⁴ ≈ 5.9605 × 10⁻⁸, regardless of the mantissa bits — every subnormal is a multiple of this one step.

To round g to the nearest representable subnormal, divide by the step: 4.5 × 10⁻⁶ / 5.9605 × 10⁻⁸ ≈ 75.497, which rounds to the integer 75. The representable value is 75 × 2⁻²⁴ = 75 / 16,777,216 ≈ 4.47035 × 10⁻⁶ — a relative error of about 0.66%, tolerable. But take a second, smaller gradient, g = 2 × 10⁻⁸: 2 × 10⁻⁸ / 5.9605 × 10⁻⁸ ≈ 0.3355, which rounds to 0. The gradient is not approximated — it is deleted. Every weight update that depended on that gradient contribution silently loses it, and if many gradients in a layer sit below roughly 2⁻²⁵ ≈ 2.98 × 10⁻⁸ (half a subnormal step, the round-to-zero threshold), that entire layer can stop learning while every other part of the pipeline reports normal operation.

Loss scaling works around this without touching the model's numerics at all. Because differentiation is linear, multiplying the scalar loss L by a constant S before calling backward multiplies every gradient in the computation graph by exactly the same S — ∂(S·L)/∂w = S · ∂L/∂w. Pick S = 2¹⁵ = 32,768 and the previously-vanishing gradient becomes 2 × 10⁻⁸ × 32,768 = 6.5536 × 10⁻⁴, which is comfortably inside FP16's normal range. After the backward pass, the optimizer divides the FP32 master-copy gradient by the same S before applying the update, so the scale factor never touches the actual weights — it only exists to keep the gradient tensor's exponents inside FP16's usable window during the backward pass itself.

The following function reproduces FP8-E4M3's rounding rule directly (used again below); the same round-to-nearest-mantissa logic is what FP16 applies to the loss-scaled gradient above, just with 10 mantissa bits instead of 3:

import math

def to_e4m3(x, bias=7, mantissa_bits=3):
    if x == 0:
        return 0.0
    sign = -1.0 if x < 0 else 1.0
    x = abs(x)
    e = math.floor(math.log2(x))
    frac = x / (2 ** e) - 1.0          # fractional part in [0, 1)
    m_int = round(frac * (2 ** mantissa_bits))
    if m_int == 2 ** mantissa_bits:    # mantissa rounded up to next power
        m_int = 0
        e += 1
    value = (1 + m_int / (2 ** mantissa_bits)) * (2 ** e)
    return sign * value

print(to_e4m3(9.4))   # e=3, frac=0.175, m_int=round(1.4)=1 -> 1.125 * 8 = 9.0

Tracing it by hand: e = floor(log2(9.4)) = floor(3.2327) = 3, so frac = 9.4/8 - 1 = 0.175. With 3 mantissa bits there are 8 possible fractions; round(0.175 × 8) = round(1.4) = 1, giving mantissa 1 + 1/8 = 1.125 and final value 1.125 × 2³ = 9.0. The function returns exactly 9.0, a 4.26% relative error against the true 9.4 — three mantissa bits simply cannot distinguish 9.4 from 9.0.

Loss Scaling as a Control Loop

Production frameworks (PyTorch's GradScaler, NVIDIA's Apex) do not use a single fixed scale factor picked by hand — they run a small automatic control loop, because the right scale changes as training progresses and gradient magnitudes shift. The algorithm, matching PyTorch's actual implementation logic, is:

# illustrative reimplementation of the GradScaler update rule
# (PyTorch's real defaults: init_scale=65536, growth_factor=2.0,
#  backoff_factor=0.5, growth_interval=2000)

def scaler_step(state, grads_have_inf_or_nan):
    if grads_have_inf_or_nan:
        state["scale"] *= state["backoff_factor"]
        state["growth_tracker"] = 0
        # optimizer.step() is skipped this iteration
    else:
        # optimizer.step() runs normally here
        state["growth_tracker"] += 1
        if state["growth_tracker"] == state["growth_interval"]:
            state["scale"] *= state["growth_factor"]
            state["growth_tracker"] = 0

Every step, the framework checks whether any gradient overflowed to inf (a scale set too high pushes some gradients past FP16's max of 65,504). If it did, that step's optimizer update is skipped entirely — not applied with corrupted gradients — and the scale is cut by the backoff factor. If gradients were clean, the optimizer steps normally and a counter increments; once that counter reaches growth_interval consecutive clean steps, the scale is grown, on the theory that if overflow hasn't happened in a long stretch, the scale can probably go higher and recover more small gradients from underflow.

Using smaller illustrative numbers (growth_interval=3, backoff_factor=0.5, growth_factor=2, init_scale=1024) instead of the real defaults, here is a six-step trace with a given sequence of clean/overflow outcomes:

Note that the backoff at step 4 is unconditional — it does not care how far into the growth cycle the tracker was; it always multiplies by backoff_factor and resets the tracker to zero. This asymmetry (fast, unconditional shrink; slow, conditional growth) is deliberate: the cost of one skipped optimizer step is negligible, but training on overflowed gradients for even one step can corrupt weights permanently.

Misconception: "BF16 Is Just a Better FP16"

A common misconception is that bfloat16 is a strict upgrade over FP16 — wider exponent range and therefore "safer," so it must also be at least as precise. It is not. BF16 keeps FP32's 8-bit exponent field, which is exactly why it avoids the underflow and overflow problems worked through above without needing loss scaling at all — a gradient of 2 × 10⁻⁸ lands nowhere near BF16's minimum normal value of 1.18 × 10⁻³⁸. But BF16 pays for that exponent width by cutting the mantissa to 7 bits, three fewer than FP16's 10.

The worst-case relative rounding error for a normalized floating-point format with m mantissa bits is bounded by 2⁻⁽ᵐ⁺¹⁾ (half the gap between adjacent representable values, relative to the value itself). For FP16, m=10, giving 2⁻¹¹ ≈ 0.0488%. For BF16, m=7, giving 2⁻⁸ ≈ 0.391% — exactly 8× coarser, since the mantissa bit counts differ by exactly 3 and 2³ = 8. BF16 solves the range problem and creates no new overflow risk, but every individual value it stores is rounded roughly eight times more coarsely than the same value in FP16. In practice this rounding noise usually averages out across millions of weight updates and BF16 remains the more robust default on hardware that supports it natively — but "more robust" is not "more precise," and a student who conflates the two will misdiagnose why a BF16 run and an FP16-with-loss-scaling run of the same model can converge to slightly different final losses.

Beyond Float16: BF16 in Production

Google Brain developed bfloat16 for Cloud TPUs, which have supported native bfloat16 matrix units since the TPU v2 generation, precisely to sidestep the loss-scaling machinery described above — TPU training pipelines at Google could skip the overflow-detection-and-retry control loop entirely because the format's range already matches FP32. NVIDIA followed with native BF16 tensor-core throughput starting on the A100 (Ampere, 2020), matching FP16's throughput while removing the dynamic-range failure mode. This is why most large language model pretraining runs published after roughly 2021 report BF16 rather than FP16 as their working precision: it removes an entire class of training-instability bugs — the silent-underflow failure from this chapter's opening scenario cannot happen in BF16, because the gradient would have to be smaller than roughly 10⁻³⁸ to vanish, a magnitude essentially never seen in practice.

Beyond BF16: FP8 and the Transformer Engine

Hopper-generation H100 GPUs (2022) added tensor cores that operate natively on 8-bit floating point, in the two formats diagrammed above: E4M3 (more mantissa, used for weights and activations, where quantization error matters more than range) and E5M2 (more exponent, used for gradients, which need FP16-like range). NVIDIA's Transformer Engine automates the choice of per-tensor scale factor needed to keep values inside each tensor's narrow representable band — with only 3 or 2 mantissa bits, a poorly chosen scale wastes most of the format's already-thin precision on values that are all clustered in one corner of the representable range.

Two scaling strategies are used in practice. Delayed scaling picks the scale for the current step from a running history of recent per-tensor maximum-absolute-values (the amax history), so the scale factor is available before the forward pass runs and no extra synchronization pass over the data is needed — the tradeoff is that the scale is always slightly stale, based on recent-past statistics rather than the current tensor. Current scaling computes the scale from the tensor's actual maximum in the same step, which is more accurate but requires an additional reduction pass over the tensor before the scaled matmul can run, adding latency. Micikevicius et al. (2022), in the joint NVIDIA/Arm/Intel proposal that standardized E4M3 and E5M2, reported that with per-tensor scaling of this kind, FP8 training of large transformer models could match FP32 baseline accuracy across a range of model sizes — the original mixed-precision paper, Micikevicius et al. (2018), had shown the analogous result for FP16 with loss scaling a few years earlier.

The precision cost is steep on paper: applying the same rounding-error formula as above, E4M3's m=3 gives a worst-case relative error bound of 2⁻⁴ = 6.25% — over 100 times coarser than FP16's 0.049%. This is why FP8 is used surgically rather than end-to-end: master weights and gradient accumulation still happen in FP32 or BF16, and only the matrix-multiply operands for specific layers are cast down to FP8 and back, tensor by tensor, guided by exactly the per-tensor scale factors Transformer Engine computes.

Stochastic Rounding: A Different Fix for the Same Disease

Everything above uses round-to-nearest: pick whichever representable value is closest. Gupta et al. (2015), working with 16-bit fixed-point rather than floating-point formats, showed a different fix for the same underlying problem — repeated small updates that each round to zero under round-to-nearest, silently cancelling learning. Stochastic rounding instead rounds up or down with probability proportional to how close the true value is to each neighboring representable value: a gradient that is 30% of the way from 0 to the smallest step rounds up to that step 30% of the time and down to zero 70% of the time. Any single update is noisier, but the rounding is now unbiased in expectation — across many updates, the average effect matches the true unrounded gradient rather than being clipped at zero every time. This is a genuinely different mechanism from loss scaling: loss scaling moves values into a range where round-to-nearest already works well; stochastic rounding changes the rounding rule itself so that even values below the smallest step still contribute, on average, over enough steps. The two are complementary rather than substitutes, and stochastic rounding sees the most use in the very low-precision regimes — FP8 and below — where scaling alone cannot recover enough dynamic range.

Active Recall

Attempt each question before reading its answer.

1. A run uses the loss-scaling trace from this chapter (growth_interval=3, backoff_factor=0.5, growth_factor=2, init_scale=1024) but a teammate changes growth_interval to 5, keeping the same sequence of outcomes (clean, clean, clean, overflow, clean, clean). Does the scale ever grow within these 6 steps, and what is the scale after step 6?
2. A gradient has true value g = 3 × 10⁻⁸. Does it flush to zero in FP16, or round to the smallest subnormal? Compute the relative error.
3. Using the ULP-bound formula 2⁻⁽ᵐ⁺¹⁾, compute the worst-case relative rounding error for FP8-E4M3 (m=3) and state how many times coarser it is than BF16's.
4. Why doesn't BF16 need loss scaling, even though its mantissa is shorter than FP16's?
5. What is the practical tradeoff between delayed scaling and current scaling in NVIDIA's Transformer Engine?
6. Using the to_e4m3 rounding method from the worked example, compute the E4M3 representation of x = 0.02 and its relative error.

Answers.

1. With growth_interval=5, the tracker needs 5 consecutive clean steps to trigger growth. Steps 1–3 are clean: tracker goes 0→1→2→3, scale stays 1024 (no growth yet, since 3 < 5). Step 4 overflows: this is independent of growth_interval — backoff always fires on overflow — so scale is cut to 1024 × 0.5 = 512 and the tracker resets to 0, regardless of the growth-interval change. Steps 5–6 are clean: tracker goes 0→1→2, still short of 5. So the scale never grows in this 6-step window (unlike the original trace, which grew once at step 3), and the final scale after step 6 is 512. This shows the two hyperparameters govern independent things: growth_interval only paces how fast the scale climbs back up; it has zero effect on how hard or how quickly it falls.

2. The FP16 subnormal step is 2⁻²⁴ ≈ 5.9605 × 10⁻⁸. 3 × 10⁻⁸ / 5.9605 × 10⁻⁸ ≈ 0.5033, which rounds to 1 (since it exceeds the 0.5 round-to-zero threshold), not 0. The gradient survives as 1 × 2⁻²⁴ ≈ 5.9605 × 10⁻⁸. Relative error: (5.9605 − 3) / 3 ≈ 98.7% — technically representable, but the rounding error is nearly as damaging as an outright flush to zero. This is the borderline case loss scaling exists to move away from entirely.

3. 2⁻⁽³⁺¹⁾ = 2⁻⁴ = 6.25%. BF16's bound was 2⁻⁸ = 0.391%. The ratio is 2⁻⁴ / 2⁻⁸ = 2⁴ = 16 — E4M3's worst-case rounding error is 16 times coarser than BF16's.

4. Loss scaling exists to fight underflow/overflow caused by a narrow exponent field, not to fight rounding (precision) error. BF16 keeps FP32's full 8-bit exponent field and bias-127 range, so a gradient would need to fall below roughly 10⁻³⁸ in magnitude — far smaller than any real training gradient — before it could underflow. The mantissa is shorter, which does cost precision (per the misconception section above), but precision loss and range loss are different failure modes, and BF16 only fixes the second one — which happens to be the one loss scaling was built to fix.

5. Delayed scaling computes the per-tensor scale from a history of past amax values, so the scale for the current step is already known before the forward/backward pass starts, avoiding an extra reduction pass over the tensor — but it uses slightly stale statistics. Current scaling computes the scale from the tensor's actual maximum in the same step, which is more accurate but requires an additional pass over the data first, adding latency to every step. Delayed scaling trades a small, usually tolerable amount of scaling accuracy for speed; current scaling trades speed for tighter accuracy.

6. e = floor(log2(0.02)) = floor(-5.6439) = -6. frac = 0.02 / 2⁻⁶ − 1 = 0.02 / 0.015625 − 1 = 1.28 − 1 = 0.28. m_int = round(0.28 × 8) = round(2.24) = 2. Value = (1 + 2/8) × 2⁻⁶ = 1.25 × 0.015625 = 0.0195312. Relative error: (0.02 − 0.0195312) / 0.02 ≈ 2.34% — smaller than the 9.4 example's error, because 0.28 happened to land close to a representable mantissa fraction (2/8 = 0.25) this time.

Think About It

Think about this: How would you explain mixed precision training: float16 and beyond to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind mixed precision training: float16 and beyond, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.