Weight Initialization Strategies: From Xavier to Kaiming

A UPI fraud-detection team builds a 10-layer fully connected network on top of transaction embeddings — merchant category, device fingerprint, time-of-day, velocity features — 256 units per hidden layer, ReLU activations throughout. Two engineers initialize it two different ways before the first training run. Engineer A samples every weight from N(0, 0.01²), the "small random numbers" rule of thumb from an early tutorial. Engineer B uses PyTorch's default kaiming_uniform_. Both networks have identical architecture, identical data, identical loss function. Engineer A's network never learns — loss stays flat at ln(2) ≈ 0.693 for the binary cross-entropy, epoch after epoch, as if the model were guessing. Engineer B's converges in the first few hundred steps. No code bug exists anywhere in either script. The entire difference is a single scalar: the standard deviation used to draw the initial weights. This chapter derives exactly why that scalar matters, traces the arithmetic that produces it, and shows why the "obviously safe" choice of small random numbers is, for a ReLU network, actively the wrong one.

Why the weights cannot start at a constant

Before asking how large the initial weights should be, rule out the two easiest guesses. Setting every weight in a layer to a single constant c (including c = 0) is fatal regardless of the value chosen. If two neurons in the same layer receive the same input and hold identical weights, they compute identical outputs, receive identical gradients during backpropagation, and are updated identically at every step, forever. A layer of 256 such neurons behaves as one neuron copied 256 times — the effective capacity of the layer collapses to width 1, no matter how many parameters it nominally has. This is the symmetry-breaking requirement: initial weights must be independently randomized so that neurons in the same layer diverge from the very first gradient step. Random initialization is necessary, but it does not by itself say anything about the scale of the randomness — and that scale is where Engineer A's network died.

How variance moves through a linear layer

Consider one fully connected layer: input vector x with n components, weight matrix W of shape (n, n), output (pre-activation) y = Wx, so each output unit is y_j = ∑_i x_i W_ji. Assume the components of x are mutually independent with zero mean, and each weight W_ji is drawn independently with zero mean and variance Var(w), independent of x. Because the mean of every term is zero, the cross terms in the variance of a sum vanish, leaving:

Var(y_j) = ∑_i Var(x_i W_ji) = ∑_i E[x_i²] · Var(w) = n · E[x²] · Var(w)

where E[x²] is the second moment of a single input component (equal to Var(x) when x is zero-mean). This one line is the entire mechanism this chapter is about: the variance of a layer's output scales with the fan-in n, the variance of the incoming signal, and the variance of the weights, multiplied together. If n · Var(w) is less than 1, every layer shrinks the signal; if it exceeds 1, every layer amplifies it. A 10-layer network compounds this factor 10 times, so even a mild per-layer shrinkage becomes catastrophic by depth: a factor of 0.5 per layer leaves 0.5¹⁰ ≈ 0.001 of the original variance — pre-activations so small that gradients computed from them (which scale with the same factor, by the chain rule) shrink to a scale far below what the optimizer's learning rate can resolve — a practical vanishing-gradient problem, not a literal floating-point underflow — and the optimizer sees no usable signal. This is exactly what stalled Engineer A's network, and it is precisely the failure mode both Xavier and Kaiming initialization are algebraic solutions to.

Xavier / Glorot initialization: solving for tanh and sigmoid

Xavier Glorot and Yoshua Bengio's 2010 AISTATS paper, "Understanding the difficulty of training deep feedforward neural networks," was written for networks using tanh or sigmoid activations. Near zero, both functions are locally linear (tanh(z) ≈ z for small z), so to first order the activation doesn't change the variance-propagation equation above — it just passes it through. Requiring that variance be preserved forward through the layer, n · Var(w) = 1, gives the fan-in rule:

Var(w) = 1 / fan_in

Glorot and Bengio also required the same preservation to hold for gradients flowing backward through the layer during backpropagation, which by an identical argument (transposing the roles of fan-in and fan-out) gives Var(w) = 1 / fan_out. A single layer generally cannot satisfy both exactly unless fan-in equals fan-out, so the paper's practical compromise averages the two constraints:

Var(w) = 2 / (fan_in + fan_out)

This is "Xavier normal" initialization: draw each weight from N(0, 2/(fan_in+fan_out)). The equally common "Xavier uniform" variant draws from U(-a, a); since a uniform distribution on [-a, a] has variance a²/3, matching that to 2/(fan_in+fan_out) gives a = √(6/(fan_in+fan_out)) — the exact bound used in PyTorch's xavier_uniform_ and Keras's GlorotUniform.

Why the same rule fails under ReLU

Xavier's derivation leans on one specific fact about tanh and sigmoid near the origin: they don't change the second moment of a zero-mean signal passing through them, to first order. ReLU does not have this property — it doesn't approximate the identity, it deletes half the distribution outright. For a pre-activation z that is symmetric around zero (which y = Wx is, by the same zero-mean argument as above, regardless of what its input x looked like — a sum of independent zero-mean terms is itself zero-mean), the second moment of relu(z) is exactly half that of z:

E[relu(z)²] = ∫_{z>0} z² f(z) dz

Since z²f(z) is an even function (symmetric about 0), the integral over the positive half is exactly half the integral over the whole real line, which is E[z²] = Var(z). So E[relu(z)²] = Var(z)/2 — ReLU throws away exactly half the signal's power, every single layer, by construction, not as an approximation. Chaining this with the variance-propagation equation from before, for a ReLU network the recursion for the pre-activation variance at layer l becomes:

Var(y_l) = fan_in · Var(w) · (Var(y_{l-1}) / 2)

Using Xavier's Var(w) = 1/fan_in here gives Var(y_l) = Var(y_{l-1})/2 — the signal's variance is halved at every layer, regardless of network width, purely because Xavier's derivation never accounted for ReLU zeroing half the distribution. This is the exact gap Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun closed in their 2015 ICCV paper, "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification."

Kaiming / He initialization: compensating for the missing half

The fix follows directly from the recursion above: to cancel the factor of 1/2 that ReLU introduces, double the weight variance so that fan_in · Var(w) · (1/2) = 1:

Var(w) = 2 / fan_in

This is Kaiming (He) normal initialization — draw weights from N(0, 2/fan_in). The uniform variant follows the same variance-matching arithmetic as before: U(-a, a) with a = √(6/fan_in), PyTorch's default kaiming_uniform_ for nn.Linear and nn.Conv2d layers. Note what changed and what didn't relative to Xavier: the structure is identical — a variance target divided by a fan count — but the constant in the numerator doubles (2 instead of 1) to pay for the ReLU nonlinearity's effect, and the denominator drops back to a single fan-in term (the backward-pass symmetry argument works out equivalently with fan-out under the same halving logic, which is why PyTorch exposes a mode='fan_out' option that preserves gradient variance instead of activation variance). PyTorch's API also exposes a nonlinearity and a argument that generalizes the "2" to a gain² term appropriate to the actual activation — gain = √2 for plain ReLU, gain = √(2/(1+a²)) for LeakyReLU with negative slope a, gain = 5/3 for tanh — because different activations delete or compress different fractions of the incoming variance.

Worked example: tracing ten layers under three initializations

Return to the fraud-detection network: 256 units per hidden layer, ReLU activation, 10 hidden layers, input drawn from N(0,1) per feature (256-dimensional). The code below simulates a forward pass through 10 such layers under three initialization schemes and measures the standard deviation of the pre-activation y at each layer, over a batch of 20,000 samples, so the empirical variance concentrates close to its true expectation.

import numpy as np
np.random.seed(0)

n = 256          # units per layer (fan_in = fan_out here)
batch = 20000    # samples, for stable variance estimates
depth = 10       # hidden layers

def run(std_fn):
    x = np.random.randn(batch, n)          # input, mean 0, Var = 1
    pre_stds = []
    for l in range(depth):
        std_w = std_fn(n)
        W = np.random.randn(n, n) * std_w  # weight matrix, mean 0
        y = x @ W.T                        # pre-activation (zero-mean)
        pre_stds.append(float(y.std()))
        x = np.maximum(y, 0)               # ReLU -> next layer's input
    return pre_stds

naive_std   = lambda n: 0.01
xavier_std  = lambda n: np.sqrt(1.0 / n)
kaiming_std = lambda n: np.sqrt(2.0 / n)

print("naive  :", [round(s, 4) for s in run(naive_std)])
print("xavier :", [round(s, 4) for s in run(xavier_std)])
print("kaiming:", [round(s, 4) for s in run(kaiming_std)])

Running this (output shown exactly as produced, rounded to 4 decimals):

naive  : [0.1599, 0.0182, 0.002, 0.0003, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
xavier : [0.9985, 0.7111, 0.5333, 0.42, 0.3056, 0.2413, 0.1864, 0.1223, 0.085, 0.0593]
kaiming: [1.4088, 1.3627, 1.3237, 1.3231, 1.2254, 1.2123, 1.1158, 1.1521, 1.1641, 1.2519]

These numbers match the closed-form recursion almost exactly. With std = 0.01, n · Var(w) = 256 · 0.0001 = 0.0256; the first layer alone (which sees the raw Gaussian input, not yet halved by a prior ReLU) already scales std by √0.0256 = 0.16 — matching the simulated 0.1599 — and every subsequent layer applies a further factor of √(0.0256/2) ≈ 0.113. By layer 6 the theoretical std is 2.97×10⁻&sup6; — vanishingly small relative to the learning rate and update scale a real optimizer uses, not a literal float32 underflow, but the same practical vanishing-gradient wall — this is what stalled Engineer A's network. With Xavier (std = √(1/256)), the per-layer factor after the first is exactly √0.5 ≈ 0.7071 — visible directly in the simulated sequence, each entry roughly 0.71× the one before it — a slow, ReLU-driven leak rather than a collapse, but still a leak: by layer 10 only 4.4% of the signal survives. With Kaiming (std = √(2/256)), the per-layer factor is exactly 256 · (2/256) · (1/2) = 1 — the recursion is a fixed point, and the simulated std hovers around 1.1–1.4 for all ten layers, the small fluctuations being ordinary sampling noise from a finite 20,000-sample batch and a single random weight draw per layer, not systematic drift.

The misconception: "Xavier is the general-purpose choice"

The most common error at this point is treating Xavier initialization as activation-agnostic — reasoning that since it was "the" solution to the vanishing/exploding problem, it's safe as a default for any network, ReLU included. This is exactly backwards from the historical record: Xavier's derivation was correct only under the assumption that the activation function locally preserves the second moment of a zero-mean signal, which holds for tanh and sigmoid near the origin but not for ReLU, which deletes exactly half the distribution's mass every layer by construction. Using Xavier's 1/fan_in variance on a deep ReLU network doesn't crash training the way Engineer A's std=0.01 did, but it does introduce a slow, compounding √0.5-per-layer decay that becomes serious well before 20 or 30 layers — which is precisely the depth regime He et al.'s 2015 paper targeted, since it was written specifically to make substantially deeper ReLU convolutional networks trainable. The correct mental model is: the initialization variance formula is a function of which nonlinearity follows the weight matrix, not a universal constant. Match the formula to the activation — Xavier for tanh/sigmoid, Kaiming for ReLU and its variants — rather than treating either as a default.

Beyond plain MLPs: fan mode, gain, and residual streams

Two refinements matter in production networks. First, PyTorch's kaiming_normal_ exposes a mode argument: 'fan_in' (the default, preserving forward-pass activation variance, as derived above) or 'fan_out' (preserving backward-pass gradient variance instead, useful when a network's width changes sharply across layers and one direction matters more than the other for stable training). It also exposes nonlinearity, which sets the gain constant appropriately — √2 for ReLU as derived here, a smaller √(2/(1+a²)) for LeakyReLU with slope a (since a leaky unit doesn't zero the negative half completely, it deletes less variance, so it needs less compensation), and 5/3 for tanh (an empirically-tuned correction for tanh's saturation away from the origin, where the local-linearity assumption Xavier relies on starts to break down).

Second, residual connections change the propagation equation itself. A plain feedforward layer replaces its input, x_l = f(W_l x_{l-1}), which is the recursion this whole chapter has been analyzing. A residual block instead adds to a running stream, h_l = h_{l-1} + F(h_{l-1}). If each block's output variance is initialized to be comparable to the stream's own variance (as Kaiming init would do by design), then stacking N such blocks means N roughly-independent contributions are summed into the stream, and variances of independent sums add: Var(h_N) ≈ Var(h_0) + N · Var(F(·)). Deep transformers and ResNets are exactly this shape, and this is why GPT-2 (Radford et al., 2019, "Language Models are Unsupervised Multitask Learners") scales the initial weights of each residual-branch output projection by a further factor of 1/√N, where N is the number of residual blocks in the model — without it, the residual stream's variance would grow with model depth even with per-layer Kaiming init handling the within-block propagation correctly, because Kaiming init was never designed to account for the outer addition. A separate line of work, layer-sequential unit-variance initialization from Mishkin and Matas's 2015 paper "All You Need Is a Good Init," sidesteps closed-form derivation entirely: it initializes with orthonormal weight matrices, then empirically rescales each layer's weights, one layer at a time on a forward pass over real data, until that layer's output variance measures exactly 1 — a data-driven fixed point rather than an analytic one, useful when a network mixes activations or layer types the closed-form derivations above don't cleanly cover.

Active recall

Attempt each question before reading its answer.

1. A linear layer has fan_in = 512, fan_out = 128, followed by tanh. Give its Xavier-normal weight std, and the bound a for Xavier-uniform.
2. The same layer is switched to use ReLU instead of tanh. Give its Kaiming (He) fan_in std, and state exactly how it compares numerically to the Xavier fan_in-only std (√(1/fan_in), ignoring fan_out).
3. In the ten-layer worked example, suppose the hidden width is shrunk from n = 256 to n = 64 everywhere, keeping Kaiming init, ReLU, and depth = 10. Does the variance-preservation property (pre-activation std staying roughly constant across depth) survive? What changes and what doesn't?
4. Why does initializing every weight to exactly 0 fail, even though the variance-propagation equation says Var(y) = n · 0 · Var(x) = 0 is at least a well-defined (if trivial) fixed point?
5. In a deep transformer with residual connections, why does GPT-2 scale each residual block's output-projection weights by an extra 1/√N at initialization, on top of ordinary per-layer Kaiming/Xavier init within each block?
6. A student writes bias = np.random.randn(n) * np.sqrt(2/fan_in) to initialize biases the "Kaiming way," matching the weights. Is this sound practice?

Answers.

1. Combined Xavier variance is 2/(fan_in+fan_out) = 2/640 = 0.003125, so std = √0.003125 ≈ 0.0559. For the uniform variant, a = √(6/640) = √0.009375 ≈ 0.0968, so weights are drawn from U(-0.0968, 0.0968).

2. Kaiming fan_in variance is 2/512 = 1/256, so std = 1/16 = 0.0625 exactly. The Xavier fan_in-only std is √(1/512) ≈ 0.04419. The ratio is 0.0625 / 0.04419 = √2 ≈ 1.4142 — Kaiming's std is exactly √2 times larger, which is precisely the compensation factor derived for ReLU halving the variance (doubling the weight variance means multiplying the weight std by √2).

3. The property survives exactly. The per-layer variance-recursion factor for Kaiming init is fan_in · Var(w) · (1/2) = fan_in · (2/fan_in) · (1/2) = 1 for any fan_in — the fan_in cancels algebraically, so the fixed-point property is width-independent by design. What does change: the concrete weight std itself, √(2/64) ≈ 0.1768 versus √(2/256) ≈ 0.0884 — exactly 2× larger, since std scales as 1/√n. A second, subtler effect: the derivation relies on summing many independent terms (a law-of-large-numbers argument), so with only 64 terms per neuron instead of 256, the empirical per-neuron variance across a finite batch concentrates less tightly around the theoretical value — a rerun of the simulation at n = 64 would still center near std ≈ 1.4 by layer 10, but with visibly larger layer-to-layer and neuron-to-neuron fluctuation than the n = 256 run above.

4. The variance equation only tracks the second moment of a hypothetical random signal — it says nothing about whether distinct neurons compute distinct functions. With every weight equal to 0 (or any other shared constant), every neuron in a layer receives the same gradient during backpropagation (by symmetry of the identical forward computation) and is therefore updated identically at every training step, indefinitely. A 256-unit layer initialized this way has the trainable capacity of a single unit, regardless of how "correct" its aggregate variance looks. Preserving variance is necessary for trainability but not sufficient — the weights must also be independently randomized to break the symmetry.

5. A residual block adds its output to the running stream rather than replacing it: h_l = h_{l-1} + F(h_{l-1}). Per-layer Kaiming/Xavier init inside F correctly preserves F's own output variance relative to its input, but says nothing about the outer addition. Since each block's contribution is roughly independent of the accumulated stream, variances of the sum add: after N blocks, Var(h_N) ≈ Var(h_0) + N · Var(F(·)), growing roughly linearly with depth even though every individual block is internally well-scaled. Scaling each block's output projection by 1/√N at init shrinks Var(F(·)) by a factor of N, so the total added variance across all N blocks stays O(Var(h_0)) regardless of how deep the model is — keeping the residual stream's scale stable at initialization independent of layer count.

6. No. The variance formulas derived in this chapter come from a specific structural argument: a weight matrix maps a distributed n-dimensional input through n independent random terms summed together, and the 1/fan_in or 2/fan_in scaling exists to counteract exactly that summation. A bias is a single per-neuron scalar added once, with no fan-in sum behind it — the derivation simply doesn't apply. Standard practice, used in PyTorch, Keras and virtually every published architecture, initializes biases to zero (or occasionally a small positive constant, e.g. to keep ReLU units initially active). Applying a Kaiming-scaled random draw to biases instead injects unnecessary, unmotivated asymmetric noise into every neuron's pre-activation before any input-dependent signal has even arrived.

Think About It

Think about this: How would you explain weight initialization strategies: from xavier to kaiming to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind weight initialization strategies: from xavier to kaiming, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.