Three months before her Class 10 board exams, Ananya finds a trick that feels a little like cheating. She asks ten seniors how many hours a day they studied in the weeks before their own boards, and what percentage they eventually scored. She plots the two numbers against each other and draws the straight line that fits best: for roughly every extra hour of daily study, add about four percentage points to the score.
The line looks reasonable in the middle of the data, at five, six, seven hours a day. Push it to the edges and it falls apart. Extend it out to fourteen hours of daily study and it predicts a score past 100%, which no marksheet can show. Extend it the other way, toward zero hours, and depending on exactly where it crosses the axis, it can predict a negative score, equally impossible. A straight line has no idea that percentage marks are trapped between 0 and 100. It keeps climbing, or falling, forever in both directions, because that is the one thing a straight line is mathematically required to do.
What actually happens to most students' scores as study hours increase looks closer to a curve than a line: barely any gain for the first couple of hours, since that is little more than what an educated guess would already get you, a steep climb through the middle range as concepts start clicking into place, and then a flattening near the top, because a student already scoring 95% has very little room left to gain from one more hour of revision. That rise-then-flatten shape, held between a floor and a ceiling, is exactly what a family of functions called activation functions exists to produce. Before using one properly, it helps to see precisely why the plain arithmetic inside a neural network cannot produce that shape on its own, no matter how many layers get stacked on top of each other.
What a Single Neuron Actually Computes
Strip away the biology and an artificial neuron is arithmetic. Multiply each input by a weight, add the results together, add one more number called the bias, and the result is a value usually written z, the neuron's pre-activation.
z = w1*x1 + w2*x2 + ... + wn*xn + b
If a neuron stopped right there, it would be doing exactly what linear regression already does: fitting a straight line, or with more inputs a flat plane, through the data. That stays true even if many such neurons are chained into layers, so long as every one of them stops at z. However many layers, however many neurons per layer, a network built entirely from weighted sums is still just linear regression underneath. The next few sections prove that claim directly, using a single running example.
The Step Function: Where the Idea Began
In 1943, Warren McCulloch and Walter Pitts proposed the first mathematical model of an artificial neuron. It compared a weighted sum against a threshold and switched on or off, a hard step function:
step(z) = 1 if z > 0, otherwise 0
In 1958, Frank Rosenblatt proposed the Perceptron, one of the first trainable artificial neurons, built on this same step function. It worked for simple decisions, but it hid a fatal flaw for learning: its slope is zero everywhere except at one point where it is not even defined. A training algorithm that nudges weights based on how much a small change would improve the output has nothing to push against, since the function is flat almost everywhere it gets measured. In 1969, Marvin Minsky and Seymour Papert went further and proved that a single layer of step-function neurons cannot learn the XOR pattern, output 1 when exactly one of two binary inputs is 1, output 0 otherwise, because no single straight line can separate XOR's outputs correctly. That proof is often credited as one of the reasons funding for neural network research dried up for the following decade. The eventual fix needed two things together: stacking layers, and replacing the step with something smoother. In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams showed how a multi-layer network of smooth, differentiable neurons could be trained efficiently with an algorithm called backpropagation, which needs exactly the kind of gradually-changing slope a step function refuses to provide. Sigmoid, covered next, was one of the first activation functions to make that training method work in practice.
Sigmoid: Back to That Exam Curve
The sigmoid function is the mathematical shape behind Ananya's rise-then-flatten curve:
sigmoid(z) = 1 / (1 + e^-z)
Here e is Euler's number, approximately 2.718, the same constant used in compound interest formulas. Whatever value z takes, positive or negative, huge or tiny, sigmoid squeezes it into the open interval between 0 and 1. Large positive z pushes the output close to 1, large negative z pushes it close to 0, and z = 0 lands exactly at 0.5. That is why it suits a neuron whose output should read as a probability: how likely a transaction is to be fraud, how likely a student is to cross 90%.
Sigmoid has two real weaknesses. First, its slope never exceeds 0.25, reached only at z = 0, and shrinks toward zero as z moves away from the origin in either direction. During training, the signal that tells early layers how to adjust gets multiplied by one such slope at every layer it passes through on its way back through the network. Stack several sigmoid layers, multiply several numbers under 0.25 together, and the signal reaching the earliest layers shrinks toward nothing: the vanishing gradient problem. Second, sigmoid's output is never negative, which slows learning in the layers that feed it. For these reasons, sigmoid today mostly appears in output layers of binary classifiers rather than in hidden layers.
Tanh: A Zero-Centred Alternative
The hyperbolic tangent has the same S-shape as sigmoid, stretched to range between -1 and 1 instead of 0 and 1:
tanh(z) = (e^z - e^-z) / (e^z + e^-z)
tanh(0) = 0, and the function is zero-centred: roughly half its output values are negative and half positive across typical inputs, which lets gradients flow more evenly through a network than sigmoid's always-positive output does. Its maximum slope is also higher, 1.0 at z = 0, four times sigmoid's 0.25. Tanh is really a rescaled sigmoid rather than an unrelated invention: tanh(z) = 2*sigmoid(2z) - 1, which is worth checking by substitution if the algebra is ever in doubt. Tanh still flattens out for large positive or negative z, so very deep networks built purely from tanh layers still run into vanishing gradients, just less severely than sigmoid does.
ReLU: The Modern Default
The Rectified Linear Unit, or ReLU, is almost absurdly simple:
ReLU(z) = max(0, z)
Positive z passes through unchanged. Negative z becomes exactly 0. Graphed, it looks like a bent wire: flat along zero for every negative input, then a straight diagonal line for positive ones. That is the whole function, and two properties made it the default for hidden layers in most modern networks. Its slope is exactly 1 for any positive z, so it never shrinks the signal the way sigmoid or tanh do, as long as z stays positive, gradients pass through many layers undiminished. It is also cheap: no exponentials to compute, only a comparison. When AlexNet, a deep convolutional network built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, won the 2012 ImageNet competition using ReLU throughout its hidden layers, researchers noticed it trained several times faster than equivalent tanh-based networks of the same size, and ReLU quickly became the standard starting point for hidden layers across the field.
ReLU is not perfect. Because both its output and its slope are exactly 0 for any negative z, a neuron whose weights drift so that it only ever receives negative input effectively shuts off for good: it always outputs 0, its slope is always 0, and it never updates again. This is called the dying ReLU problem. The worked example later in this chapter shows the same clipping mechanism acting on a single transaction, one neuron silenced by one negative input, the small-scale version of what dying ReLU looks like when it happens for every input a neuron ever sees.
Leaky ReLU and the Smoother Variants
Leaky ReLU patches the dying ReLU problem with one small change: instead of flattening negative inputs to exactly 0, it lets a small fraction through.
LeakyReLU(z) = z if z > 0, otherwise 0.01*z
That small slope, commonly 0.01, keeps a non-zero signal alive for negative inputs, so a neuron producing negative pre-activations can recover instead of dying for good. It costs almost nothing extra to compute. Researchers have since proposed further refinements, GELU and Swish among the more widely used in large modern networks, all built on the same idea: round off ReLU's sharp corner at zero while keeping its cheap, gradient-friendly behaviour for positive inputs.
A Toy Network: Should This UPI Transaction Be Flagged?
To see the difference activation functions actually make, build a miniature fraud-screening network with two inputs describing a single transaction on a payments app:
x1: the transaction amount, scaled to a 0-to-1 range (0 = tiny, 1 = very large for this user)x2: how unusual the time of the transaction is for this user, also scaled 0 to 1 (0 = a completely normal time, 1 = a time this user has never transacted at before)
One transaction arrives with x1 = 0.8 (a large payment) and x2 = 0.2 (a fairly ordinary time). This toy network has two hidden neurons, h1 and h2, feeding one output neuron that produces a risk score between 0 and 1. A real fraud model at a bank or payments company has far more inputs, layers, and neurons, and is trained on millions of past transactions, but the arithmetic inside each neuron here is identical to what such a system runs at every step. Suppose training has already set these weights and biases (this chapter skips the training process itself, to focus on what a completed network does with them):
h1: weight 0.6 onx1, weight -0.9 onx2, bias 0.1h2: weight -0.7 onx1, weight 0.5 onx2, bias -0.3- output neuron: weight 0.9 on
h1, weight 0.8 onh2, bias -0.1
First, Without Any Activation Function
Leave every neuron as a plain weighted sum, with nothing squashing its output, and compute the two hidden pre-activations:
z_h1 = 0.6(0.8) + (-0.9)(0.2) + 0.1 = 0.48 - 0.18 + 0.1 = 0.40
z_h2 = -0.7(0.8) + 0.5(0.2) + (-0.3) = -0.56 + 0.10 - 0.30 = -0.76
With no activation function, h1 = 0.40 and h2 = -0.76 exactly, unchanged. Feed these into the output neuron:
z_o = 0.9(0.40) + 0.8(-0.76) + (-0.1) = 0.36 - 0.608 - 0.1 = -0.348
That produces a single number, -0.348, for this one transaction. But is this two-layer arrangement doing anything a single neuron with two inputs could not do alone? Substitute the formulas for z_h1 and z_h2 directly into the output equation and expand it:
z_o = 0.9(0.6x1 - 0.9x2 + 0.1) + 0.8(-0.7x1 + 0.5x2 - 0.3) - 0.1
= (0.54x1 - 0.81x2 + 0.09) + (-0.56x1 + 0.40x2 - 0.24) - 0.1
= (0.54 - 0.56)x1 + (-0.81 + 0.40)x2 + (0.09 - 0.24 - 0.1)
= -0.02x1 - 0.41x2 - 0.25
Check it against the numbers above: -0.02(0.8) - 0.41(0.2) - 0.25 = -0.016 - 0.082 - 0.25 = -0.348, exactly matching the full two-layer computation. Two hidden neurons, four weights, and two biases collapsed into one straight-line equation in x1 and x2. Every one of those extra parameters was doing nothing that a single neuron could not have done alone. This is not a quirk of these particular numbers: any stack of layers built entirely from weighted sums, regardless of how many layers or neurons, always collapses algebraically into one equivalent layer. A hundred-layer network built this way has exactly the representational power of one neuron.
Now, With ReLU and Sigmoid in Place
Restore the activation functions: ReLU after the hidden layer, sigmoid after the output layer. Same inputs, same weights, same biases. The pre-activations do not change, since activation functions apply after the weighted sum, not before:
z_h1 = 0.40, z_h2 = -0.76
Apply ReLU to each:
h1 = ReLU(0.40) = 0.40
h2 = ReLU(-0.76) = 0
Neuron h2 is silenced for this transaction, clipped to zero exactly the way a permanently dying neuron would be clipped for every transaction it ever saw. For this one input, it simply means h2 contributes nothing further downstream; the network's other path, through h1, still carries a signal forward. Feed the two hidden activations into the output neuron:
z_o = 0.9(0.40) + 0.8(0) + (-0.1) = 0.36 + 0 - 0.1 = 0.26
Finally, squash z_o with sigmoid to get a probability:
sigmoid(0.26) = 1 / (1 + e^-0.26) ≈ 1 / 1.7711 ≈ 0.565
The network's risk score for this transaction comes out to about 0.565, or 56.5%. If the app flags anything above 0.5 for manual review, this transaction gets flagged.
Compare this to the no-activation version, which produced z_o = -0.348 for the exact same input, weights, and biases. That number was not a probability and had no natural bound, and it was already proven to reduce to one straight-line equation across every possible transaction. The ReLU-and-sigmoid version cannot be reduced that way. The moment h2 gets clipped to zero, the equation connecting the output to x1 and x2 changes shape: for transactions where h2 stays positive, the network behaves like one linear equation; for transactions where h2 goes negative, like a different one. Gluing together many small straight-line pieces at the bends is how a network built from simple linear arithmetic ends up approximating a curve, and that is only possible because of the non-linear ReLU sitting between the layers.
The Same Computation in Code
The same forward pass, written in Python with NumPy, should reproduce every number calculated by hand above:
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def relu(z):
return np.maximum(0, z)
# one transaction: [scaled_amount, scaled_time_unusualness]
x = np.array([0.8, 0.2])
# hidden layer: 2 neurons, one weight row per neuron
W1 = np.array([[ 0.6, -0.9], # neuron h1
[-0.7, 0.5]]) # neuron h2
b1 = np.array([0.1, -0.3])
z_hidden = W1 @ x + b1
h = relu(z_hidden)
# output layer: 1 neuron
W2 = np.array([0.9, 0.8])
b2 = -0.1
z_out = W2 @ h + b2
risk_score = sigmoid(z_out)
print("hidden pre-activation:", z_hidden) # [ 0.4 -0.76]
print("hidden after ReLU: ", h) # [0.4 0. ]
print("output pre-activation:", z_out) # 0.26
print("risk score: ", risk_score) # 0.5646...
Running this prints hidden pre-activation: [ 0.4 -0.76], hidden after ReLU: [0.4 0. ], output pre-activation: 0.26, and finally risk score: 0.5646..., matching the hand calculation digit for digit. Whenever a neural network calculation seems abstract, this is the habit worth building: trace it by hand on a tiny example first, then confirm the code agrees, rather than trusting either one blindly.
Choosing the Right Activation Function in Practice
- Hidden layers, default choice: ReLU. It is cheap, it does not saturate for positive inputs, and it is the starting point in nearly every modern deep learning library.
- Hidden layers, if ReLU keeps dying: Leaky ReLU or one of its smoother relatives, all of which keep a small slope alive for negative pre-activations.
- Output layer, a yes-or-no decision: Sigmoid, exactly as in the risk-score example, since its output reads directly as a probability between 0 and 1.
- Output layer, choosing one of several categories: Softmax, which generalises sigmoid to more than two classes.
- Output layer, predicting an unbounded number: no activation at all, since squashing a prediction that has no natural upper or lower limit would only distort it.
Softmax takes a whole layer of raw scores at once and turns them into a probability distribution that sums to exactly 1:
softmax(z_i) = e^z_i / (sum of e^z_j over every class j)
Suppose a three-way version of the risk score produces raw scores of 1.2 for Genuine, 0.4 for Suspicious, and -0.8 for Blocked, instead of a single flag-or-not number. Softmax turns these into probabilities of about 63%, 28%, and 9%, in that order, numbers that are easy to compare, easy to threshold, and guaranteed to add up to 100%. A plain weighted sum could never guarantee that last property by itself; passing the scores through an exponential and normalising the result is what makes it work.
Back to the Exam Curve
Ananya's straight line failed for the same root reason the no-activation version of the transaction network failed: a weighted sum has no way to bend, flatten, or cap itself. It will predict a score of 140%, or a fraud probability of -0.35, whenever the arithmetic says so, because nothing in its definition tells it to stop. Every function in this chapter, step, sigmoid, tanh, ReLU, Leaky ReLU, softmax, solves a version of the same problem: take an unbounded weighted sum and turn it into something shaped like the real quantity it is meant to represent, a probability between 0 and 1, a boundary that can bend around a corner no straight line could reach, an output that respects the limits of the question being asked.
Depth without non-linearity is an illusion: a hundred linear layers computing plain weighted sums are, underneath, one linear layer. The moment a non-linear activation function sits between those layers, each one starts earning its keep, and the network gains the ability to bend its decision boundary through the data instead of drawing a single flat line across it. That, precisely, is what "non-linearity is key" means, for a fraud-detection network deciding on Ananya's transaction, and for whatever curve actually connects her study hours to her result in March.
Think About It
Think about this: How would you explain activation functions: non-linearity is key to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind activation functions: non-linearity is key, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.