Optimizers: SGD, Adam, and Friends

It is 9 PM on a Saturday, and you have just placed an order for biryani on Swiggy or Zomato. Before you have even chosen a tip amount, the app already shows a number: "Arriving in 34 minutes." A few weeks later you order from the very same restaurant, and this time it says 22 minutes. The app is not guessing randomly. Somewhere behind the scenes, a model has looked at your distance from the restaurant, the time of day, current traffic, and dozens of other signals, combined them using a set of internal numbers, and produced a prediction.

Those internal numbers are called weights, and getting them right is the entire game of machine learning. When a model is first built, its weights are little more than random guesses, and its predictions are nonsense. Somehow, after being shown thousands of real orders and their real delivery times, the weights settle into values that make genuinely useful predictions. The algorithm responsible for nudging those weights, order after order, from "random guess" toward "usefully accurate" is called an optimizer. This chapter is about how that nudging actually works — from the simplest version, Stochastic Gradient Descent, to the version almost every modern deep learning system reaches for by default, Adam.

Measuring Wrongness: The Loss Function

Before any weight can be corrected, the model needs a way to measure exactly how wrong it currently is. This measure is called a loss function. For a numeric prediction problem like delivery time, a common choice is squared error: if the model predicts y_pred minutes and the order actually took y_actual minutes, the loss is (y_pred - y_actual)². Squaring does two useful things — it makes every error positive, so a 10-minute overestimate and a 10-minute underestimate do not cancel each other out, and it punishes large errors far more harshly than small ones. Being off by 20 minutes contributes 400 "loss points," not merely twice the 100 points from being off by 10.

Now picture this loss as a landscape: the model's weights form the horizontal axes, and how large the loss is for those weight values forms the height. Good weights sit in low valleys; bad weights sit on high peaks. Training a model is nothing more than searching this landscape for the lowest point — the combination of weights that keeps the loss as small as possible across all the training orders.

The Gradient: Which Way Is Downhill?

A real neural network's loss landscape has millions of dimensions, one for every weight — far too many to search by trial and error. The tool that makes it searchable is the gradient: at any point in the landscape, the gradient is a vector that points in the direction the loss increases fastest. You may already know a simplified cousin of this idea from coordinate geometry — the slope m in y = mx + c tells you how steeply a line rises. The gradient is that same idea, extended to a landscape shaped by many weights instead of just one line.

Since the gradient points toward the steepest increase, moving in exactly the opposite direction takes you downhill as fast as possible from where you are standing. This is the whole idea behind gradient descent: repeatedly compute the gradient of the loss with respect to every weight, then take a small step in the opposite direction.

For our delivery-time model, suppose the prediction is a simple straight line: y_pred = w*x + b, where x is the distance in kilometres, w is minutes-per-kilometre, and b is a fixed base time covering kitchen prep and handoff to the rider. Using the power rule from calculus — the same rule that says the derivative of x² is 2x, which you will formalise fully in Grade 11 if you have not already — the gradient of the squared-error loss works out to two clean expressions:

gradient_w = 2 * (y_pred - y_actual) * x
gradient_b = 2 * (y_pred - y_actual)

Both formulas share the term (y_pred - y_actual): how wrong the prediction is, and in which direction. The w gradient additionally scales by x, because a change in w has a bigger effect on the prediction when the distance is large. Once we have these two numbers, gradient descent updates each weight by subtracting a small fraction of its own gradient:

w = w - learning_rate * gradient_w
b = b - learning_rate * gradient_b

The learning rate controls how big a step to take. Picture walking downhill in thick fog: too small a step and you barely move, wasting time; too large a step and you might vault clean over the valley floor and land higher up the opposite slope than where you started. Choosing a sensible learning rate is one of the most consequential decisions in training any deep learning model.

A Worked Example: Teaching a Line to Predict Delivery Time

Let's train this tiny model by hand. Start with a deliberately poor guess, w = 2 (minutes per kilometre) and b = 10 (base minutes), and a learning rate of 0.001 — small enough to keep our updates controlled. Now feed it one real order: a 5 km delivery that actually took 30 minutes.

Step 1 — Predict. y_pred = w*x + b = 2*5 + 10 = 20 minutes. The model thinks it will take 20 minutes.

Step 2 — Measure the error. error = y_pred - y_actual = 20 - 30 = -10. The model under-predicted by 10 minutes.

Step 3 — Compute the gradients.

gradient_w = 2 * (-10) * 5 = -100
gradient_b = 2 * (-10)     = -20

Step 4 — Update the weights.

w = 2  - 0.001 * (-100) = 2  + 0.1  = 2.1
b = 10 - 0.001 * (-20)  = 10 + 0.02 = 10.02

Notice the sign logic: both gradients were negative, so subtracting them actually increased the weights — exactly what should happen, since a model that is under-predicting needs to learn to output larger numbers. Also notice how a learning rate of 0.001 kept this update modest. Had we instead used a learning rate of 0.1, the update to w alone would have been 0.1 * 100 = 10, jumping straight from w = 2 to w = 12 in one step — a wild overshoot that would make the very next prediction worse, not better. That is exactly the danger a poorly chosen learning rate creates.

Now feed the model a second order — 2 km, which actually took 15 minutes — using the freshly updated weights.

Step 1 — Predict. y_pred = 2.1*2 + 10.02 = 14.22 minutes.

Step 2 — Measure the error. error = 14.22 - 15 = -0.78.

Step 3 — Compute the gradients.

gradient_w = 2 * (-0.78) * 2 = -3.12
gradient_b = 2 * (-0.78)     = -1.56

Step 4 — Update.

w = 2.1   - 0.001 * (-3.12) = 2.10312
b = 10.02 - 0.001 * (-1.56) = 10.02156

Notice how much smaller this update was — from a 0.1 jump in w down to a 0.00312 jump — simply because the second order's error (-0.78) was much smaller than the first (-10). This is a genuinely useful property of gradient descent: it naturally takes large corrective steps when it is very wrong, and small, careful steps as it gets closer to the right answer, without anyone having to program that behaviour explicitly.

Try it yourself: using w = 2.10312 and b = 10.02156, work through a third order — 8 km, actual time 42 minutes — and compute the next prediction, error, gradients, and updated weights. If your arithmetic is right, the prediction before this update should come out to roughly 26.85 minutes.

The Same Example, in Code

Everything above is exactly what the following Python function does. There is no hidden magic inside a deep learning framework's optimizer — underneath, it is this loop, applied millions of times, to millions of weights.

def predict(w, b, x):
    return w * x + b

data = [(5, 30), (2, 15), (8, 42)]  # (distance_km, actual_minutes)

w, b = 2.0, 10.0
learning_rate = 0.001

for distance, actual_time in data:
    prediction = predict(w, b, distance)
    error = prediction - actual_time

    grad_w = 2 * error * distance
    grad_b = 2 * error

    w = w - learning_rate * grad_w
    b = b - learning_rate * grad_b

    print(f"distance={distance:>2} km | predicted={prediction:6.2f} | "
          f"actual={actual_time:>2} | new_w={w:.5f} | new_b={b:.5f}")

Running this prints:

distance= 5 km | predicted= 20.00 | actual=30 | new_w=2.10000 | new_b=10.02000
distance= 2 km | predicted= 14.22 | actual=15 | new_w=2.10312 | new_b=10.02156
distance= 8 km | predicted= 26.85 | actual=42 | new_w=2.34558 | new_b=10.05187

Every number in that output matches the hand calculation above, right down to the third row you worked out yourself. That is the entire point: an optimizer is not a mysterious black box, it is this exact arithmetic, repeated relentlessly.

Batch, Stochastic, and Mini-Batch: How Much Data Before Each Update?

The loop above updated the weights after every single order — this is the "stochastic" in Stochastic Gradient Descent (SGD): each step uses one randomly picked example, so the gradient is a noisy estimate of the true downhill direction, but you get to take many steps very quickly. At the opposite extreme sits batch gradient descent, which computes the gradient using every single order in the entire training set, averages them, and only then takes one step. This produces a far more accurate, less noisy direction, but if your delivery app has ten million past orders, you would have to process all ten million before making even one tiny weight adjustment — painfully slow, and often too large to even fit in memory at once.

In practice, almost every deep learning system uses the middle ground: mini-batch gradient descent, which averages the gradient over a small batch — commonly 32, 64, or 128 examples — before each update. This is fast enough to take frequent steps, stable enough that those steps are not wildly noisy, and it fits comfortably on a GPU. When engineers or frameworks like PyTorch say "SGD," they almost always mean this mini-batch version, not the strict one-example-at-a-time version. For our delivery app, this would mean: collect 64 completed orders, average their gradients, update the weights once, then repeat with the next 64.

Plain Gradient Descent's Weak Spot: Zig-Zagging

Plain gradient descent has a real weakness. Picture a loss landscape shaped like a narrow, steep-sided valley — steep across the valley but only gently sloped along the valley floor toward the true minimum. Moving always exactly opposite to the current gradient, plain gradient descent bounces back and forth between the steep walls, wasting many small zig-zag steps instead of making swift progress along the gentle direction where the real improvement lies. This shape is common in real neural networks, which often have exactly this kind of lopsided landscape across their many weights. Two ideas fix it: remembering the direction you have been consistently moving in, called momentum, and giving each weight its own personalised step size, called an adaptive learning rate. Adam, as we will see, uses both at once.

Momentum: Remembering the Trend

Momentum borrows an idea directly from physics — formalised for optimization by the mathematician Boris Polyak in 1964 — and it fixes the zig-zag problem elegantly. Instead of moving purely according to the current gradient, momentum keeps a running "velocity": an exponentially weighted average of recent gradients.

velocity = beta * velocity + (1 - beta) * gradient
weight   = weight - learning_rate * velocity

Here beta (often 0.9) controls how much of the past is remembered — with beta = 0.9, roughly the last ten gradients meaningfully influence the current velocity. Picture a heavy ball rolling down that narrow valley: the sideways, zig-zagging components of the gradient keep flipping sign from step to step, so they partly cancel out in the running average, while the component pointing consistently along the valley floor keeps reinforcing itself and builds up speed. The practical effect for our delivery-time model: if distance has been under-predicted for several orders in a row, momentum notices that consistent trend and accelerates the correction, instead of repeating the same tiny step every single time.

Adaptive Learning Rates: AdaGrad and RMSProp

Momentum smooths the direction of travel, but it still applies the same learning rate to every weight. That is often wasteful: some weights, such as one attached to a rarely-true feature like "is it currently raining," receive gradient signal only occasionally, while others, like the weight on distance, receive it on every single order. AdaGrad, proposed by Duchi, Hazan, and Singer in 2011, addressed this by tracking the cumulative sum of squared gradients for each weight individually and dividing its learning rate by the square root of that sum — weights that have historically received large or frequent gradients get their effective learning rate shrunk, while rarely-updated weights keep taking full-sized steps. The catch is that because the sum only ever grows, the effective learning rate keeps shrinking throughout training and can grind to a near-halt long before the model has finished learning.

RMSProp, introduced by Geoffrey Hinton around 2012, fixed exactly this problem with one small change: instead of an ever-growing sum of squared gradients, it keeps an exponentially weighted moving average of them. Old squared-gradient values gradually fade out rather than accumulating forever, so a weight's effective learning rate can recover if its gradients shrink for a while.

Adam: Momentum and Adaptive Learning Rates Together

Adam — short for Adaptive Moment Estimation — was introduced by Diederik Kingma and Jimmy Ba in a 2014 paper presented at ICLR 2015, and it has since become the default optimizer for the large majority of deep learning projects, from image classifiers to large language models. Adam's insight is simple: why choose between momentum and an adaptive learning rate when you can keep both? It maintains two running averages for every single weight — a first moment, m, which is the momentum-style exponential average of the gradient itself, controlled by beta1 (default 0.9), and a second moment, v, which is an exponential average of the squared gradient, exactly like RMSProp, controlled by beta2 (default 0.999).

m = beta1 * m + (1 - beta1) * gradient
v = beta2 * v + (1 - beta2) * gradient ** 2

m_corrected = m / (1 - beta1 ** t)
v_corrected = v / (1 - beta2 ** t)

weight = weight - learning_rate * m_corrected / (sqrt(v_corrected) + epsilon)

The division by (1 - beta1 ** t) and (1 - beta2 ** t), where t is the current step number, is a bias correction: since m and v both start at zero, early updates would otherwise be biased toward zero, so this correction scales them back up for the first several steps and fades to almost nothing later on. The tiny epsilon (typically 1e-8) in the denominator exists purely to avoid dividing by zero when v is very small. The paper's suggested defaults — learning rate 0.001, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8 — work well enough across such a wide range of problems that most engineers never change them.

The net effect: Adam moves like a ball with momentum, thanks to beta1, while also personalising the step size for every single weight based on how large and how consistent that weight's recent gradients have been, thanks to beta2 — exactly the two fixes the zig-zagging valley needed, applied simultaneously and automatically, weight by weight.

Choosing an Optimizer in Practice

Real deep learning code rarely writes these update rules by hand — frameworks provide them as a single line. In PyTorch, switching an entire training run from SGD with momentum to Adam is barely more than a one-word change:

import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# or, swapping in Adam instead:
optimizer = optim.Adam(model.parameters(), lr=0.001)

So which should you reach for? Adam is the sensible default when starting a new project: it converges quickly, is forgiving of an imperfectly chosen learning rate, and rarely needs much tuning. That robustness is why it dominates in natural language processing and in most research code you will encounter. For some computer vision tasks, however, researchers often still find that a carefully tuned SGD with momentum, paired with a learning rate that gradually decays over training, generalises slightly better on unseen data once you are willing to invest the tuning effort. A practical rule of thumb: start with Adam to get something working quickly, and treat SGD with momentum as a tool for later, once you have time to tune and want to squeeze out the last bit of accuracy.

Key Ideas at a Glance

• Gradient descent moves a weight opposite to its gradient, scaled by the learning rate.
• SGD updates using one example, or a mini-batch, at a time — fast but noisy.
• Momentum smooths that noise by averaging recent gradients, accelerating consistent progress.
• AdaGrad and RMSProp give each weight its own adaptive learning rate.
• Adam combines momentum and adaptive learning rates, which is why it is the default choice for most deep learning today.

Back to the Delivery App

Every algorithm in this chapter — SGD, momentum, AdaGrad, RMSProp, Adam — answers exactly one question: given a gradient, how should a weight change? None of them change what a network is capable of learning in the first place; a real delivery-time predictor, built from a deep neural network with millions of weights, still relies on a separate algorithm called backpropagation just to compute all those gradients before the optimizer ever touches them. What the optimizer decides is how efficiently and reliably the network turns those gradients into genuine improvement, order after order, until "Arriving in 34 minutes" stops being a shot in the dark and becomes a prediction you can actually trust. The two-weight, three-order example you traced by hand in this chapter runs on exactly the same arithmetic as the optimizer training a production model on ten million real orders — just repeated far more times, across far more weights, entirely by a computer.

Think About It

Think about this: How would you explain optimizers: sgd, adam, and friends to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind optimizers: sgd, adam, and friends, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.