Weight Initialization: Starting Right

Every Independence Day, an hour before the flag hoisting, a school's cultural committee runs the same ritual: the sound check. Someone taps the microphone, says "testing, testing, one two three," while a teacher stands at the far end of the ground with one arm raised, judging whether the Principal's voice will actually reach the last row near the cycle stand once the ceremony begins. Grounds this size usually relay the sound through two or three speakers strung along their length, each one picking up the signal from the one before it and pushing it further.

Get the levels wrong in one direction, and by the third speaker down the field, the opening line of the speech has faded into an inaudible mumble — the back rows watch lips move on the distant stage and hear almost nothing. Get it wrong in the other direction, and the instant the Principal raises their voice for the first "Good morning," the speakers shriek with the ear-splitting feedback whine that makes the whole ground wince. Notice what has, and hasn't, happened here. Nobody has said a single wrong word. The content of the speech doesn't matter yet. The entire event can be ruined or saved by a decision made before anyone speaks: how the sound system is set at the start.

A neural network is in exactly this position before training begins. It has no learned skill yet — no notion of what a fraudulent UPI transaction looks like, no notion of what digit is written in an image, nothing. All it has is a large collection of numbers sitting inside its layers, called weights, which training will spend thousands of steps adjusting. But those numbers cannot start out ; they need some starting values before the very first training step can even be computed. Choosing those starting values is called weight initialization, and, exactly like the sound check, it happens before the network has learned anything at all. Get it wrong, and — as the arithmetic in this chapter will show, worked out step by step — no amount of good training data or clever optimization can save the network. Get it right, and training has a real chance to begin.

What Exactly Gets Initialized?

Inside a neural network layer, every neuron performs the same two-step calculation. First, it takes a weighted sum of its inputs: multiply each incoming value by its own weight, add the results, then add one more number called the bias. This weighted sum is usually written z = w·x + b. Second, it passes that sum through an activation function — commonly ReLU, sigmoid, or tanh — to produce the neuron's output. The weights and biases together are the network's parameters: the only numbers training is allowed to change.

Before training starts, nobody knows the correct weights; finding them is the entire point of training. So the network needs some starting values, and the simplest option seems obvious: set every weight to zero. Zero feels neutral, unbiased, safe. It is, in fact, one of the most reliable ways to make a network completely untrainable, for a reason worth tracing carefully.

A Small Network to Watch Closely

To see exactly what goes wrong, and later exactly what goes right, it helps to work with real numbers rather than prose alone. Picture one layer from a small model that screens UPI transactions for fraud. It receives four standardized input features — standardized meaning each has been rescaled to have a mean of 0 and a standard deviation of 1, so values look like 1.2 or -0.8 rather than raw rupee amounts:

• x1 = 1.2 — the transaction amount, somewhat above average
• x2 = -0.5 — time since the account's last transaction, a shorter gap than usual
• x3 = 0.8 — a merchant risk score, somewhat above average
• x4 = -1.1 — a device-risk score, well below average (a familiar, low-risk device)

This layer has three ReLU neurons, each fully connected to all four inputs. Every neuron needs its own vector of four weights before it can compute anything at all. How should those twelve numbers be chosen?

The Trap of Sameness: Why Identical Weights Fail

Suppose every weight in this layer starts at zero. For any neuron, the weighted sum is z = 0×1.2 + 0×(-0.5) + 0×0.8 + 0×(-1.1) = 0. That holds for all three neurons, since every one of the four input features gets multiplied by zero and none of them can be told apart. All three produce the same pre-activation, z = 0, and after ReLU, the same output, a = 0.

That alone would be bad enough, but the deeper problem is not that the outputs happen to be zero right now. Because the three neurons received identical weights and identical inputs, they will also receive an identical gradient during backpropagation — each is equally "responsible" for the mistake the network just made, since each computed exactly the same thing. An identical gradient applied to an identical starting weight produces an identical updated weight. After one training step, the three neurons have moved off zero, but they remain exactly equal to one another. This repeats at every step, forever: three neurons that start out identical stay identical through the whole training run, computing the same function no matter how many epochs pass. A layer with three neurons trapped this way has, in effect, the representational power of a layer with one.

This is the symmetry-breaking problem, and zero is not actually special here; it is just the most common way people stumble into it. Set every weight to 0.1 instead of zero, and the same failure appears: z = 0.1×1.2 + 0.1×(-0.5) + 0.1×0.8 + 0.1×(-1.1) = 0.1×0.4 = 0.04, identical across all three neurons, because they are still computing the same function of the input. Any constant initialization, whatever the constant, produces the same lockstep behavior. What breaks the symmetry is not moving away from zero — it is making the starting weights different from one another, so that different neurons are nudged toward looking for different patterns from the very first gradient step. This is why weight initialization is random by design, not by carelessness: randomness is what guarantees no two neurons are born as clones.

The Trap of Scale: Vanishing and Exploding Signals

Randomness alone is not enough. A network with random weights that are far too small, or far too large, runs into a second problem that has nothing to do with symmetry: scale. This is easiest to see in a much deeper network — say, 50 layers, unremarkable by the standards of modern image-recognition and language models.

The mathematical shape of this problem is one you already know from your maths classes, even without having applied it to a neural network: compound interest. Money growing by a fixed percentage every year reaches P×(1+r)^n after n years, and small changes in r, compounded over enough years, produce enormous differences in the final amount. A signal passing through the layers of a network compounds the same way. If each layer scales the signal down by just 10% on average — a multiplying factor of 0.9 — then after 50 layers the signal has been multiplied by 0.9^50, which works out to roughly 0.005: the original signal survives at about half of one percent of its strength, effectively noise. If instead each layer scales the signal up by 10% — a factor of 1.1 — then 1.1^50 comes to roughly 117: the signal has exploded past a hundred times its starting size. Both outcomes are fatal in their own way. A network cannot learn from a signal that has vanished into rounding error, and it cannot learn from one that has overflowed into unstable, meaningless numbers. Only a per-layer factor close to exactly 1 keeps a deep signal intact, and that has to be built in from the first layer; there is no fixing it later.

Networks using sigmoid or tanh face a second, compounding reason scale matters: saturation. Both functions flatten out for large inputs — sigmoid squeezes everything toward 0 or 1, tanh squeezes everything toward -1 or 1 — and in that flat region their slope is nearly zero. The steepest either function ever gets is right around an input of zero: sigmoid's derivative peaks at exactly 0.25, and tanh's peaks at exactly 1. If weights are too large, the weighted sums feeding these activations swing far from zero, landing neurons in the flat, saturated region where the local slope is close to zero. Since backpropagation multiplies gradients by these slopes at every layer, a saturated neuron blocks the gradient from passing through — its forward output may be large, but the gradient needed to learn from it still vanishes. This is why sigmoid and tanh networks are notoriously sensitive to initialization scale in both directions: too small, and the signal itself shrinks to nothing layer after layer; too large, and the signal survives but the gradient does not.

ReLU, defined as ReLU(z) = max(0, z), sidesteps saturation on its positive side — its slope is exactly 1 for any positive input, however large, so it never flattens out the way sigmoid and tanh do. This is a major reason ReLU became the default choice for deep networks. But ReLU has its own wrinkle: it sets every negative pre-activation to exactly zero, discarding it completely. For weights centered at zero, roughly half of a layer's pre-activations will be negative for any given input, so roughly half its neurons switch off. That "roughly half" is not a minor detail — it is the exact number the next section builds its formula around.

Xavier/Glorot and He: The Formulas That Set the Scale

By 2010, researchers had a precise mathematical answer to the scale problem. Xavier Glorot and Yoshua Bengio, in a paper presented at the AISTATS conference, worked out how large a layer's weights should be so that the variance of the signal — a measure of how spread out the values are — stays roughly constant as it moves forward through the network, and stays roughly constant as the gradient moves backward through it too. Their answer depends on two numbers for any given layer:

• fan-in — how many values flow into each neuron of the layer, the number of connections it receives
• fan-out — how many neurons this layer feeds forward into, its width as seen from the next layer

Protecting the forward-flowing signal alone calls for a weight variance of 1/fan-in. Protecting the backward-flowing gradient alone calls for 1/fan-out. Since one layer usually cannot satisfy both exactly, the compromise Glorot and Bengio settled on — now called Xavier initialization, or Glorot initialization after its authors — is to average fan-in and fan-out into a single effective number and use its reciprocal:

• Normal form: weights drawn from N(0, 2/(fan-in + fan-out))
• Uniform form: weights drawn from Uniform(-limit, limit), where limit = sqrt(6/(fan-in + fan-out))

Both describe the same underlying variance; the 2 and the 6 differ only because a uniform distribution and a normal distribution convert a target variance into a range differently. This works well for sigmoid and tanh layers, whose behavior near zero is close to a straight line, which is what the derivation assumes.

ReLU breaks that assumption, because — as the previous section showed — it throws away roughly half its input. In 2015, Kaiming He and colleagues Xiangyu Zhang, Shaoqing Ren, and Jian Sun, then at Microsoft Research, published a variance formula built for exactly this case: to make up for ReLU discarding about half the signal's variance on its way out of every layer, double the variance going in. This is now called He initialization:

• Normal form: weights drawn from N(0, 2/fan-in)
• Uniform form: weights drawn from Uniform(-limit, limit), where limit = sqrt(6/fan-in)

He initialization uses fan-in alone, without averaging in fan-out, because the paper's central concern was keeping the forward-propagating signal well-scaled through very deep ReLU networks. That same paper reported one of the first published results to beat a well-known human-level benchmark on the ImageNet classification challenge — a 4.94% top-5 error rate against the 5.1% human-level estimate used in the paper — a result that a badly-scaled first layer would have made unreachable no matter how the rest of the network was designed.

Back to the Test Network: He Initialization in Action

Return to the three-neuron fraud-screening layer, where fan-in = 4. He initialization calls for a standard deviation of sqrt(2/4) = sqrt(0.5) ≈ 0.71. Suppose the random initializer draws these twelve numbers — in practice this happens automatically; here they are written out for inspection:

Neuron A: [ 0.62, -0.45,  0.88, -0.31]
Neuron B: [-0.71,  0.19, -0.24,  0.95]
Neuron C: [ 0.33,  0.77, -0.66, -0.12]

Every value sits within about one and a half standard deviations of zero, an unremarkable draw. (With only twelve numbers, their exact spread will not match the target of 0.71 precisely — that only settles down once a layer has hundreds or thousands of weights, by the same law of large numbers that makes a fair coin land closer to 50-50 heads the more times it is flipped.) Now trace the forward pass for input x = [1.2, -0.5, 0.8, -1.1], term by term for Neuron A:

z_A = (0.62)(1.2) + (-0.45)(-0.5) + (0.88)(0.8) + (-0.31)(-1.1)
    = 0.744 + 0.225 + 0.704 + 0.341
    = 2.014

Working through Neuron B and Neuron C the same way gives z_B = -2.184 and z_C = -0.385. Passing all three through ReLU:

a_A = max(0, 2.014)  = 2.014
a_B = max(0, -2.184) = 0
a_C = max(0, -0.385) = 0

Only Neuron A fired for this particular transaction. With just three neurons, a 1-in-3 split is well within normal variation, but the pattern behind it is exactly what He initialization is built around: across a full layer of, say, 256 neurons, close to half would typically switch on for any given input and half would switch off — precisely the "roughly half" that motivated the factor of 2 in the formula.

Now compare the wrong scale on both sides. Multiplying every weight above by 10, pushing the standard deviation to roughly 7, scales every weighted sum by the same factor of 10, since the calculation is a straight linear multiplication: z_A becomes 20.14, z_B becomes -21.84, z_C becomes -3.85. Numbers of that size, compounding across dozens of layers the way the compound-interest example showed, are exactly the exploding signal that makes training unstable or produces NaN ("not a number") errors outright. Scaling every weight down by a factor of 100 instead, pushing the standard deviation to roughly 0.007, shrinks the same sums to 0.02014, -0.02184, and -0.00385 — technically nonzero, technically not symmetric, but small enough that after a few more layers treated the same way, the signal becomes indistinguishable from zero. Getting the scale right is not a rounding preference. It is the difference between a signal that survives the trip through the network and one that does not.

Putting It Into Code

Modern deep learning frameworks turn correct initialization into a one-line choice rather than something computed by hand. In PyTorch:

import torch.nn as nn

layer = nn.Linear(784, 256)   # e.g. a flattened 28x28 image into a hidden layer
nn.init.kaiming_normal_(layer.weight, nonlinearity='relu')
nn.init.zeros_(layer.bias)

And in Keras:

from tensorflow import keras

layer = keras.layers.Dense(
    256, activation='relu', kernel_initializer='he_normal'
)

Notice that in both examples the bias is set to zero, and that this is completely fine, unlike setting the weights to zero. The symmetry-breaking argument earlier in this chapter depended on every neuron's weight vector being identical, which is what makes two neurons compute the same function of a varying input. A shared bias adds the same constant to every neuron, but as long as their weight vectors already differ, that constant does not make the neurons identical to each other; it only shifts each one's threshold slightly. That is why zero-bias is the unremarkable default, while zero-weight is not.

For the same 784-input, 256-neuron layer used above, the Xavier formula gives a uniform limit of sqrt(6/(784+256)) ≈ 0.076 and a normal standard deviation of sqrt(2/(784+256)) ≈ 0.044; the He formula for the same layer gives a normal standard deviation of sqrt(2/784) ≈ 0.051. All three are small numbers, and that is the point — as a layer's fan-in grows into the hundreds, each individual weight must shrink to match, so that the many small contributions summed together land back in a sensible range. Keras' Dense layer defaults to Xavier's uniform version even when no initializer is specified, which is why simple networks often train reasonably well "out of the box"; reaching for he_normal or he_uniform explicitly matters most once a network is deep and built from ReLU-family activations, which describes most modern architectures. In practice, careful initialization is usually paired with complementary techniques — batch normalization, and the skip connections popularized by the ResNet family of architectures — that further stabilize signal flow across very deep networks. None of those techniques remove the need to start sensibly; they make a good foundation even more robust, not unnecessary.

Back to the Sound Check

Return to the school ground on Independence Day morning. The teacher doing the sound check is not writing the Principal's speech and is not deciding what will be announced. Their job is narrower than that: make sure whatever is said can travel, at a stable and legible volume, from the stage to the last row, before the ceremony starts. It is a small job, and if it is skipped or done carelessly, no quality of speech-writing rescues the event, because the message never arrives intact.

Weight initialization occupies exactly that narrow, non-negotiable position in a neural network. It does not decide what the network eventually learns; the training data and gradient descent do that work across the thousands of steps still to come. What initialization decides is whether a learnable signal can make the trip through the network's layers at all — forward as activations, backward as gradients — without fading into rounding error or blowing up into instability, before a single training step has happened. Break the symmetry with randomness, so no two neurons are born as clones. Set the scale with Xavier for sigmoid and tanh layers, or He for ReLU and its relatives, so fan-in does the arithmetic instead of guesswork. Do the sound check properly, and training gets a fair chance to work. Skip it, and the most carefully designed architecture, the most painstakingly labelled dataset, and the cleverest optimizer available will not make up for a signal that never had a chance to arrive.

Think About It

Think about this: How would you explain weight initialization: starting right to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind weight initialization: starting right, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.