Backpropagation from Scratch: Chain Rule Magic

Open a food delivery app, add a ₹200 item to your cart, and watch the bill build up in stages: a packaging charge gets added, then GST is charged on that, then a flat delivery fee is tacked on at the end. Now suppose the restaurant raises the base price of that item by exactly ₹1. Does your final bill go up by ₹1? By more? By less? You could answer this by recomputing the entire bill from scratch with the new price — or you could notice that each stage in the chain only reacts to the stage right before it, work out how sensitive each stage is to its own input, and multiply those sensitivities together to get the answer in seconds.

That second approach is not just a trick for food bills. It is the exact mathematical engine that lets a neural network with millions of adjustable numbers figure out, after a single wrong prediction, precisely how much to blame each one of those numbers — and precisely how to fix them. That engine is called backpropagation, and underneath the intimidating name is an idea you already used above: the chain rule.

A Chain You Already Know: Your Food Bill

Let's actually trace it. Say the base item price is P = ₹200, there is a flat packaging fee of ₹20, then 5% GST is charged on (price + packaging), and finally a flat ₹30 delivery fee is added with no further tax on it:

• Subtotal = P + 20
• With GST = Subtotal × 1.05
• Final bill = With GST + 30

With P = ₹200: Subtotal = ₹220, With GST = ₹231.00, Final bill = ₹261.00. Now bump the price to P = ₹201: Subtotal = ₹221, With GST = ₹232.05, Final bill = ₹262.05. The final bill rose by exactly ₹1.05 for a ₹1 rise in the base price.

Notice you did not need to recompute the whole bill to predict that ₹1.05. Each stage has its own simple "sensitivity" — how much its output moves for a one-unit move in its own input. The subtotal moves 1-for-1 with the price (sensitivity 1). The GST stage multiplies its input by 1.05 (sensitivity 1.05). The final stage just adds a flat fee, so it also moves 1-for-1 with its input (sensitivity 1). Multiply the three sensitivities along the chain — 1 × 1.05 × 1 — and you get 1.05, matching what brute-force recalculation gave you, to the rupee.

Naming What Just Happened: Derivatives and the Chain Rule

That "sensitivity" has a formal name: it is the derivative of a function's output with respect to its input, a number that tells you approximately how much the output moves for a tiny move in the input. When a function is a straight line, like Subtotal = P + 20 or With GST = Subtotal × 1.05, the derivative is just the slope of that line and is the same everywhere: adding a constant has derivative 1, and multiplying by a constant k has derivative k.

The rule you used to combine three stages into one answer is one of the oldest and most useful results in calculus. If a quantity y depends on u, and u in turn depends on x, then dy/dx = (dy/du) × (du/dx). In words: the total sensitivity of y to x equals the product of the local sensitivities along the path connecting them. This holds for a chain of any length, two links or three hundred — you multiply every local derivative along the path from the far end back to the variable you care about. A neural network, as you are about to see, is nothing more than a long chain of very simple functions, which is why the chain rule turns out to be the single most important tool in deep learning.

One more building block will matter soon: squaring. If y = u², a small increase in u produces an increase in y of about 2u times as large — the derivative of u² is 2u. Check it with numbers: at u = 5.1, y = 5.1² = 26.01; nudge u up to 5.2, and y = 5.2² = 27.04, a jump of 1.03, very close to the 2 × 5.1 × 0.1 = 1.02 the derivative predicts for a nudge of 0.1 (the tiny remaining gap is a real, calculable correction term, not an error — derivatives describe the rate of change exactly only for an infinitely small nudge). Remember "the derivative of a square is twice the original": it is about to reappear the moment we compute a loss.

A Neuron Is Just a Chain of Simple Steps

Consider the smallest network still worth studying: one input, one hidden neuron, one output neuron. Its forward pass, the process of turning an input into a prediction, looks like this: x → [×w1, +b1] → z → [ReLU] → a → [×w2, +b2] → y_pred → [compare with y_true] → L.

Each arrow is one small, individually simple step in a computational graph: a row of labelled boxes for values (x, z, a, y_pred, L) joined by simple operations (multiply, add, apply ReLU, compare-and-square), with the forward pass filling in every box from left to right. The input x is multiplied by weight w1 and shifted by bias b1 to give z. That z is passed through the ReLU activation function, ReLU(z) = max(0, z), which zeroes out negative values and leaves positive values untouched, to give the activated value a. That a is multiplied by a second weight w2 and shifted by a second bias b2 to give the prediction y_pred. Finally, a loss function compares y_pred against the true value y_true and produces one number, L, measuring how wrong the prediction was.

Training the network means adjusting w1, b1, w2, and b2 so that L gets smaller. Gradient descent tells us how: nudge every weight a small step against its gradient, the derivative of L with respect to that weight. The trouble is that w1 never touches L directly. It affects L only by first affecting z, which affects a, which affects y_pred, which finally affects L. To find dL/dw1 we have no choice but to walk that entire chain and multiply the local derivatives along the way, just as with the food bill.

Doing this walk efficiently is what backpropagation actually is: start at L, where the sensitivity of L to itself is trivially 1, and move backward through the graph one step at a time, multiplying by the local derivative of each step, until every weight has received its gradient. This reverse walk is called the backward pass, and it is why the algorithm is named backpropagation — the error signal, born at the loss, propagates backward through the very graph the input walked forward through.

Setting Up a Tiny Network to Backpropagate by Hand

Let's make this concrete with numbers small enough to trace by hand and check with code afterward. Take an input x = 3 and a true target y_true = 10, with these starting weights:

• w1 = 0.8, b1 = 0.2 (the hidden neuron)
• w2 = 1.5, b2 = 1.0 (the output neuron)

Forward pass:

• z = w1 × x + b1 = 0.8 × 3 + 0.2 = 2.6
• a = ReLU(z) = ReLU(2.6) = 2.6 (positive, so ReLU leaves it untouched)
• y_pred = w2 × a + b2 = 1.5 × 2.6 + 1.0 = 4.9
• L = (y_true − y_pred)² = (10 − 4.9)² = 5.1² = 26.01

The network predicted 4.9 when the answer should have been 10, a large error, with a loss of 26.01 to show for it. Now we backpropagate to find exactly how much each of the four numbers, w1, b1, w2, and b2, is to blame.

The Backward Pass: Multiplying Local Derivatives, Link by Link

We walk the chain from L back to each weight, computing one local derivative at a time and multiplying as we go.

Step 1 — loss with respect to the prediction. The squared error is itself a two-link chain in disguise: let u = y_true − y_pred, so L = u². Using the squaring rule from earlier, dL/du = 2u, and du/dy_pred = −1 (increasing y_pred decreases u, since u is defined as y_true minus y_pred). Chain rule: dL/dy_pred = dL/du × du/dy_pred = 2u × (−1) = −2u = −2 × (y_true − y_pred) = −2 × 5.1 = −10.2. The negative sign makes sense: y_pred is too small, so increasing y_pred would decrease the loss.

Step 2 — prediction with respect to w2 and b2. Since y_pred = w2 × a + b2, the local gradient of y_pred with respect to w2 is simply a = 2.6, and with respect to b2 is 1. Applying the chain rule:

• dL/dw2 = dL/dy_pred × a = −10.2 × 2.6 = −26.52
• dL/db2 = dL/dy_pred × 1 = −10.2

Step 3 — prediction with respect to the activation. dy_pred/da = w2 = 1.5, so dL/da = dL/dy_pred × w2 = −10.2 × 1.5 = −15.3. The error signal keeps flowing backward, now expressed as "how much the loss would change if the hidden neuron's activated output changed."

Step 4 — activation with respect to z, through ReLU. The derivative of ReLU is 1 wherever z is positive and 0 wherever z is negative: flat below zero, slope 1 above it. Since z = 2.6 is positive here, da/dz = 1, so dL/dz = dL/da × 1 = −15.3. Had z been negative instead, dL/dz would have been exactly 0, no matter how large the error — a "dead" ReLU blocks the gradient from flowing back any further. That single fact is a big part of why activation choice matters so much in deep networks.

Step 5 — z with respect to w1 and b1. Since z = w1 × x + b1, dz/dw1 = x = 3 and dz/db1 = 1:

• dL/dw1 = dL/dz × x = −15.3 × 3 = −45.9
• dL/db1 = dL/dz × 1 = −15.3

Four weights, four gradients, one repeated idea. Every step used a derivative simple enough to write in one line — the hard part was never the calculus, it was keeping track of which local derivative to multiply next, and in what order. That bookkeeping is what backpropagation automates.

This particular network is an unbranched chain, so every weight influences the loss through exactly one path. In a wider network, a single weight often feeds into several neurons at once, and so influences the loss through several different paths simultaneously. The chain rule still applies to every path just as it did here; you simply add up the contributions from each path a weight participates in, instead of following only one. The multiplying never changes — only the bookkeeping grows, and bookkeeping is precisely what a computer is far better at than a human.

Checking the Hand Math with Code

The Python below performs the identical forward and backward pass, in the identical order, and prints every number computed above:

x = 3.0
y_true = 10.0
w1, b1 = 0.8, 0.2
w2, b2 = 1.5, 1.0

# ---- forward pass ----
z = w1 * x + b1
a = max(0, z)                     # ReLU
y_pred = w2 * a + b2
loss = (y_true - y_pred) ** 2
print(f"z={z:.4f}, a={a:.4f}, y_pred={y_pred:.4f}, loss={loss:.4f}")
# z=2.6000, a=2.6000, y_pred=4.9000, loss=26.0100

# ---- backward pass: chain rule, link by link ----
dL_dy_pred = -2 * (y_true - y_pred)      # Step 1
dL_dw2 = dL_dy_pred * a                  # Step 2
dL_db2 = dL_dy_pred * 1
dL_da  = dL_dy_pred * w2                 # Step 3
da_dz  = 1.0 if z > 0 else 0.0           # Step 4 (ReLU derivative)
dL_dz  = dL_da * da_dz
dL_dw1 = dL_dz * x                       # Step 5
dL_db1 = dL_dz * 1

print(f"dL_dw1={dL_dw1:.4f}, dL_db1={dL_db1:.4f}, "
      f"dL_dw2={dL_dw2:.4f}, dL_db2={dL_db2:.4f}")
# dL_dw1=-45.9000, dL_db1=-15.3000, dL_dw2=-26.5200, dL_db2=-10.2000

Run it, and the printed numbers match the hand computation, number for number. That is the whole of backpropagation for this tiny network: a handful of lines that each apply one local derivative, chained together by ordinary multiplication.

Does the Nudge Actually Help? Completing One Training Step

Gradients only tell you a direction and a size; gradient descent turns them into an update. Each weight moves a small step against its gradient, scaled by a learning rate η that controls how big the step is: w ← w − η × (dL/dw). With η = 0.01:

• w1: 0.8 − 0.01 × (−45.9) = 1.2590
• b1: 0.2 − 0.01 × (−15.3) = 0.3530
• w2: 1.5 − 0.01 × (−26.52) = 1.7652
• b2: 1.0 − 0.01 × (−10.2) = 1.1020

Run the forward pass again with these updated weights: z = 1.2590 × 3 + 0.3530 = 4.13, a = 4.13, y_pred = 1.7652 × 4.13 + 1.1020 ≈ 8.39, and the new loss is (10 − 8.39)² ≈ 2.58. One backpropagation step took the loss from 26.01 down to about 2.58 — the prediction moved from 4.90 to 8.39, much closer to the target of 10.

That drop looks dramatic mainly because this toy example has just one training point and started with a large error, which produced large gradients and therefore a large update. Real training loops rarely take such a big single step — they average gradients over batches of many examples and use learning rates tuned to move gently — but the mechanism inside every one of those small, steady steps is identical to what you just traced by hand: forward pass, chain rule backward, subtract a small multiple of the gradient, repeat.

Why Backprop Beats Guessing: The Efficiency Argument

You might wonder why we bother with derivatives at all. Why not just nudge each weight slightly, measure how the loss changes, and use that to estimate the gradient directly? This approach, called numerical differentiation, works in principle, but it needs one full forward pass per weight just to estimate one gradient. Our toy network has 4 weights, so that means 4 extra forward passes. GPT-3, a widely documented large language model, has 175 billion parameters; numerical differentiation would need 175 billion forward passes to compute one full set of gradients for a single training example. Suppose, purely for arithmetic's sake, that a single forward pass through such a model took just 1 millisecond (in reality it would take far longer) — numerical differentiation would still need roughly 175 billion milliseconds, more than five years of continuous computation, to produce gradients for one example.

Backpropagation computes every one of those gradients in a single backward pass, at roughly the computational cost of one extra forward pass, no matter how many weights the network has. That difference, one backward pass versus billions of forward passes, is not a minor optimisation. It is the reason training networks with billions of parameters is feasible at all.

Autograd: When the Framework Does the Chain Rule for You

In practice, nobody hand-derives backpropagation for real networks with dozens of layers. Frameworks like PyTorch and TensorFlow use automatic differentiation, or autograd: as you compute the forward pass, the framework quietly records every operation along with the local derivative rule attached to it, since it already knows the local derivative of multiplication, addition, ReLU, and every other basic operation. Call one method, and it walks that recorded graph backward, applying the chain rule automatically, just as you did above:

import torch

x = torch.tensor(3.0)
y_true = torch.tensor(10.0)

w1 = torch.tensor(0.8, requires_grad=True)
b1 = torch.tensor(0.2, requires_grad=True)
w2 = torch.tensor(1.5, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)

z = w1 * x + b1
a = torch.relu(z)
y_pred = w2 * a + b2
loss = (y_true - y_pred) ** 2

loss.backward()   # autograd runs the chain rule backward through the graph

print(f"dL/dw1 = {w1.grad.item():.4f}")
print(f"dL/db1 = {b1.grad.item():.4f}")
print(f"dL/dw2 = {w2.grad.item():.4f}")
print(f"dL/db2 = {b2.grad.item():.4f}")

requires_grad=True tells PyTorch to track a tensor through the graph, loss.backward() triggers the backward pass, and .grad holds the resulting gradient once it finishes. Running this prints dL/dw1 = -45.9000, dL/db1 = -15.3000, dL/dw2 = -26.5200, and dL/db2 = -10.2000 — matching the hand computation exactly, because autograd is not doing anything conceptually different. It is running the identical link-by-link multiplication, just automated and applied to graphs with millions of nodes instead of five.

Back to Your Bill: The Big Picture

The ₹1.05 you traced through a food delivery bill and the −45.9 you traced through a two-neuron network came from the same idea. Break a complicated dependency into a chain of simple stages, work out how sensitive each stage is to the one before it, and multiply those sensitivities together to learn how sensitive the far end is to the near end. Nothing about that idea changes when the chain has four links or four million, when the stages are GST calculations or matrix multiplications, or when the "far end" is a delivery bill or a loss function.

That is why backpropagation, despite the intimidating name, is not a separate topic from the chain rule you now understand — it is the chain rule, applied systematically and automated at scale. Every time a voice assistant gets a little better at recognising an accent, a fraud-detection model learns to flag a suspicious UPI transaction, or a recommendation feed improves its guess at what you will watch next, some version of the five steps you just traced by hand ran, probably billions of times, across billions of weights, in a fraction of a second. The magic was never magic. It was the chain rule, one local derivative at a time.

None of this required inventing new mathematics. Everything used today was addition, multiplication, and one derivative rule each for a line, a square, and ReLU. What backpropagation contributes is not cleverer calculus; it is discipline about the order in which those same simple rules get applied, so that a machine can repeat the process billions of times a second without ever losing track of which local derivative comes next.

• Backpropagation is the chain rule applied backward through a computational graph, from the loss to every weight.
• The forward pass computes the loss; the backward pass computes how responsible each weight is for it.
• Every individual local derivative is simple, a multiplication, an addition, or a one-line rule like ReLU's; the chain rule only asks you to multiply them in the right order.
• Gradient descent then uses those gradients to nudge every weight toward a smaller loss.
• Autograd frameworks automate this same bookkeeping, which is why understanding the hand computation is what lets you debug a network when something goes wrong.

Think About It

Think about this: How would you explain backpropagation from scratch: chain rule magic to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind backpropagation from scratch: chain rule magic, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.