Activation Functions: From ReLU to Swish

Picture a bank's fraud-detection system scanning UPI transactions the instant you tap "Pay" at a Bengaluru kirana store. Behind that half-second decision sits a neural network with, say, six hidden layers, each layer computing a weighted sum of the previous layer's outputs. Here is the uncomfortable fact every architect of that system has to confront on day one: if every one of those six layers only computes a weighted sum and passes it straight to the next layer, the entire six-layer network is mathematically identical to a single layer. Not "similar to." Identical. Depth buys you nothing. The fraud model would be exactly as powerful as a plain logistic regression on the raw transaction amount, merchant code, and time-of-day features — something you could write in one line of scikit-learn. Activation functions are the one ingredient that stops this collapse, and the specific choice of activation function — sigmoid, ReLU, Leaky ReLU, or Swish — decides whether a deep network trains at all, trains slowly, or trains fast and well. This chapter builds that story from the algebra up, with every claim checked by hand.

Why Nonlinearity Is Non-Negotiable

Let layer 1 compute h1 = W1x + b1 and layer 2 compute h2 = W2h1 + b2, with no activation function in between. Substitute the first equation into the second:

h2 = W2(W1x + b1) + b2
   = (W2W1)x + (W2b1 + b2)
   = W'x + b'

where W' = W2W1 and b' = W2b1 + b2 are just another weight matrix and bias vector. Two stacked linear layers reduce algebraically to one linear layer with different numbers inside it. Stack sixty layers this way and you still get one linear layer — the composition of any number of linear (affine) maps is itself a single linear map. A network built entirely of such layers can only ever draw a straight line (or a flat hyperplane, in higher dimensions) through its feature space. It cannot separate "genuine transaction" from "fraud" when the boundary between them curves — and in the real world, that boundary almost always curves, because fraud patterns depend on interactions between features (an unusually large amount combined with an unfamiliar merchant combined with an odd hour), not on any single feature crossing a fixed threshold.

An activation function g is a fixed, elementwise, nonlinear function inserted after each linear layer: h1 = g(W1x + b1). Because g is nonlinear, the substitution trick above no longer collapses the two layers into one. Each additional layer can now bend the decision boundary in a new way, and this is precisely what the Universal Approximation Theorem formalizes: a network with even one hidden layer, given a nonlinear activation and enough hidden units, can approximate any continuous function on a bounded domain to arbitrary accuracy. The theorem says nothing about which nonlinear function to use, or how easy the network will be to train with that choice — and that gap is exactly what the sixty years of activation-function history covered below is about.

The First Choice: Sigmoid, and the Vanishing Gradient It Quietly Causes

The historically dominant activation was the logistic sigmoid, σ(z) = 1 / (1 + e^(-z)), chosen because it squashes any real number into (0, 1), which reads naturally as a probability, and because it is smooth and differentiable everywhere — a requirement for gradient-based training. Its derivative has a clean closed form: σ'(z) = σ(z)(1 - σ(z)).

Plot that derivative and you will find it peaks at exactly z = 0, where σ(0) = 0.5, giving σ'(0) = 0.5 × 0.5 = 0.25. That number — a maximum of 0.25 — is the seed of the vanishing gradient problem, and it matters even when neurons are nowhere near saturation. Here is a fully traced example. Build a toy network with three sigmoid layers, each a single scalar neuron, weights w1 = w2 = w3 = 0.5, zero biases, input x = 1, and target output 1:

z1 = w1·x        = 0.5000   a1 = σ(z1) = 0.6225
z2 = w2·a1        = 0.3112   a2 = σ(z2) = 0.5772
z3 = w3·a2        = 0.2886   a3 = σ(z3) = 0.5717
Loss L = 0.5(a3 - target)^2 = 0.0917

Backpropagating the gradient of L with respect to w1 requires the chain rule through all three layers:

dL/dw1 = (a3-target) · σ'(z3) · w3 · σ'(z2) · w2 · σ'(z1) · x
       = (-0.4283) · (0.2449) · (0.5) · (0.2440) · (0.5) · (0.2350) · (1)
       = -0.001504

Notice that none of the three sigmoid derivatives — 0.2449, 0.2440, 0.2350 — is near zero; none of these neurons is deeply saturated. Yet each factor is comfortably below 1, and three such factors multiply together, along with two more factors of 0.5 from the weights. Run the identical network with ReLU activations instead (defined in the next section) and the same weights, same input, and the gradient works out to -0.21875 — about 145 times larger. That gap, widened further at every additional layer, is why deep sigmoid networks in the 1990s and 2000s effectively stopped learning in their early layers: the gradient signal shrank by a large factor at every layer it crossed, so by the time it reached layer 1 it carried almost no usable information, however many training epochs you allowed. tanh(z) = 2σ(2z) - 1 rescales the output to (-1, 1) and centers it at zero, which helps optimization somewhat, but its derivative still peaks at 1.0 only at z = 0 and falls off on both sides, so it suffers the same multiplicative shrinkage in deep networks, just with a slightly larger ceiling.

ReLU: A Derivative of Exactly 1, and the Trap That Comes With It

The Rectified Linear Unit, ReLU(z) = max(0, z), popularized in deep networks around 2011, replaces the smooth S-curve with two straight rays: flat at zero for z ≤ 0, and the identity line for z > 0. Its derivative is a step function — exactly 1 for z > 0, exactly 0 for z < 0 ( at the kink, conventionally set to 0 or 1 in implementations). Wherever a ReLU neuron is active, it passes the incoming gradient through completely unscaled — no 0.25 ceiling, no shrinkage — which is exactly what produced the 145× larger gradient in the comparison above and is the main reason ReLU made training genuinely deep networks (dozens to hundreds of layers) practical.

The cost of that flat left half is the "dying ReLU" problem. Take a neuron with weight w = 0.5 and input x = -1: the pre-activation is z = 0.5 × (-1) = -0.5, so ReLU(z) = 0 and, critically, ReLU'(z) = 0. During backpropagation, the local gradient at this neuron is multiplied by that 0, so the gradient flowing back to update w through this path is exactly zero — not small, zero. If a neuron's weights land in a region where this happens for most or all of the training examples it sees, it stops updating entirely: its weight gradient is 0 on every batch, so gradient descent never moves it, and it never turns back on. It has "died," and no amount of further training will revive it through this connection, because a zero gradient contains no information about which direction would help.

Patching the Flat Side: Leaky ReLU, PReLU, ELU

Leaky ReLU repairs exactly this failure mode with a one-line change: instead of a flat zero for negative inputs, use a shallow slope, LeakyReLU(z) = z if z > 0, else αz for a small constant like α = 0.1 (or 0.01, the more common default). Rerun the dying-neuron example: z = -0.5 now gives LeakyReLU(z) = 0.1 × (-0.5) = -0.05, and the derivative there is 0.1 — small, but never zero. The gradient shrinks by a factor of 10 through this neuron rather than vanishing outright, which is usually enough for gradient descent to eventually steer the neuron's weights back toward a region where it fires positively again. Parametric ReLU (PReLU) makes α a learned parameter rather than a fixed constant, letting each neuron (or each channel) discover its own best negative slope from the training data. Exponential Linear Unit (ELU) takes a different route on the negative side, using α(e^z - 1) instead of a straight line, which curves smoothly toward a floor of -α rather than continuing linearly downward; the smooth curve gives ELU a defined, non-zero second derivative everywhere, which some optimizers exploit for faster convergence, at the cost of an exponential (more expensive than a comparison) in every forward pass.

Swish: A Smooth, Self-Gated Function That Is Not Monotonic

Swish, proposed by a Google Brain search over activation-function space in 2017, is defined as Swish(z) = z · σ(z) — the input multiplied by its own sigmoid. Read it as a "self-gating" mechanism: σ(z) acts as a soft gate between 0 and 1 that decides how much of z to let through, and unlike ReLU's hard 0-or-1 gate, this one is smooth and depends continuously on z itself. Three properties distinguish Swish from the entire ReLU family:

It is smooth everywhere — no kink at the origin, so its derivative doesn't jump discontinuously the way ReLU's does. It is unbounded above, like ReLU, avoiding the saturation-at-large-|z| that kills sigmoid's gradient. It is non-monotonic — for moderately negative z, Swish actually dips below zero before rising back toward zero as z → -∞. Trace the numbers: Swish(-1) = -1 × σ(-1) = -1 × 0.2689 = -0.2689, and Swish(-3) = -3 × σ(-3) = -3 × 0.0474 = -0.1423. The function reaches its most negative point around z ≈ -1.28, where it dips to roughly -0.278, before both smoothly returning toward 0. A ReLU neuron with negative pre-activation outputs exactly 0 and forwards exactly 0 gradient; a Swish neuron in that same region outputs a small negative value and still forwards a small, non-zero gradient, so dying neurons in the strict ReLU sense cannot happen. The derivative of Swish has a clean closed form built from Swish itself: Swish'(z) = σ(z) + z·σ(z)·(1-σ(z)) = Swish(z) + σ(z)(1 - Swish(z)); at z = 1 this evaluates to 0.9277, and at z = -1 to 0.0723 — small but, again, never exactly zero. A close cousin, GELU (Gaussian Error Linear Unit, z · Φ(z) where Φ is the standard normal CDF), has an almost identical S-shaped self-gating curve and is the activation used inside the feed-forward blocks of BERT and GPT-family transformers; Swish and GELU are close enough in shape and behavior that results transfer between them, which is why "smooth, self-gated activations" now dominate large-scale architectures where ReLU and its direct variants dominated a decade earlier.

The Four Curves, Side by Side

The diagram below plots all four functions over the same input range z ∈ [-4, 4], so their differences are visible directly rather than described in words. Watch three things: how flat each curve is for negative z (this controls dying-neuron risk), how each curve behaves near z = 0 (this controls gradient strength for typical, unsaturated activity), and whether the curve ever dips below the horizontal axis (only Swish does).

Two things the plot makes obvious that formulas alone tend to hide. First, for z > 0, ReLU, Leaky ReLU, and Swish are nearly indistinguishable — they all rise close to the diagonal y = z line — which is why ReLU's simplicity is hard to beat when your inputs are mostly positive to begin with (after batch normalization, for instance). Second, all the improvement Swish offers over ReLU lives entirely in the small window around z = 0 and slightly negative z — a region a plain ReLU handles by simply cutting the signal off.

Verifying With Code

Trace this snippet by hand before running it. For each of five representative inputs, it computes all four activations using NumPy's vectorized operations, so a single call processes the whole array at once:

import numpy as np

z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def relu(z):
    return np.maximum(0, z)

def leaky_relu(z, alpha=0.1):
    return np.where(z > 0, z, alpha * z)

def swish(z):
    return z * sigmoid(z)

print(np.round(sigmoid(z), 4))
print(np.round(relu(z), 4))
print(np.round(leaky_relu(z), 4))
print(np.round(swish(z), 4))

[0.1192 0.3775 0.5    0.6225 0.8808]
[0.     0.     0.     0.5    2.    ]
[-0.2   -0.05   0.     0.5    2.   ]
[-0.2384 -0.1888  0.      0.3112  1.7616]

Check the third entry (z = 0) for each row: sigmoid gives exactly 0.5 (its defined midpoint), ReLU and Leaky ReLU both give exactly 0 (their shared kink), and Swish also gives exactly 0, since 0 × σ(0) = 0 regardless of σ(0)'s value. Check the first entry (z = -2): ReLU clips it entirely to 0, Leaky ReLU lets through 0.1 × (-2) = -0.2 exactly as its formula demands, and Swish computes -2 × σ(-2) = -2 × 0.1192 = -0.2384, matching the printed value.

Common Misconception, Corrected

Most students, on first meeting the vanishing gradient problem, assume it only strikes when a sigmoid neuron is "saturated" — pushed out to the flat extremes of the curve, where |z| is large and σ(z) is very close to 0 or 1. That is only half the story, and the worked example above proves the other half by direct calculation. In the three-layer sigmoid network, every pre-activation — 0.5, 0.3112, 0.2886 — sat well within the "healthy," unsaturated middle of the sigmoid curve, nowhere near the flat tails. None of these neurons was saturated in the sense students usually picture. Yet the resulting gradient at layer 1, -0.001504, was still roughly 145 times smaller than the gradient the identical ReLU network produced from the same weights and input. The real cause is structural, not a special "unlucky" case: sigmoid's derivative is capped at 0.25 even at its very best point (z = 0), so every layer a gradient crosses multiplies it by a factor no larger than 0.25, whether or not that layer's neurons are saturated. Ten layers of even best-case sigmoid derivatives multiply the gradient by at most 0.25^10 ≈ 0.00000095 — under one part in a million — purely from the shape of the function, before saturation even enters the picture. Saturation makes vanishing gradients catastrophically worse, but it is not the precondition for the problem to exist; the 0.25 ceiling alone is enough to strangle a genuinely deep sigmoid network, which is exactly why ReLU's unscaled derivative of 1 was such a consequential fix rather than a marginal one.

Active Recall

Attempt each question before reading its answer.

1. Why does a neural network with five stacked linear layers and no activation functions behave identically to a network with just one linear layer?
2. A sigmoid neuron has pre-activation z = 2. Compute σ(2) and σ(-2), and state which one corresponds to a larger local gradient σ'(z) — justify without recomputing the derivative formula from scratch.
3. A ReLU neuron receives pre-activation z = -3 during a forward pass. What does it output, and what gradient does it pass backward to the layer before it? Would a Leaky ReLU neuron with alpha = 0.01 behave differently at the same z, and by how much?
4. Explain, using the 145× gradient comparison from this chapter, why a practitioner in 2010 who switched a ten-layer sigmoid network to ReLU would likely see a training-speed improvement far larger than 145× just from that swap — think about what happens when the per-layer multiplier compounds over many layers.
5. Compute Swish(3) by hand, given σ(3) ≈ 0.95257.
6. A student claims "Swish can never output a negative number, since it's built from sigmoid, which is always positive." Identify the flaw in this reasoning.

Answers

1. Composing affine maps h2 = W2(W1x+b1)+b2 algebraically simplifies to h2 = (W2W1)x + (W2b1+b2) = W'x + b', which is itself a single affine map. This holds no matter how many linear layers are chained, so five linear layers reduce to one, with no gain in representational power.
2. σ(2) = 0.8808 and σ(-2) = 0.1192 — these two are the same distance from the extremes but on opposite sides. σ'(z) = σ(z)(1-σ(z)) is symmetric around z=0, and its value depends only on how close σ(z) is to 0.5, not on the sign of z. Since |0.8808-0.5| = |0.1192-0.5| = 0.3808, the two gradients are exactly equal (0.8808 x 0.1192 = 0.1192 x 0.8808 = 0.1050); neither is "larger" — the question's premise that one must be larger is the trap.
3. ReLU(-3) = 0, and the backward gradient through this neuron is exactly 0 (dead for this input). Leaky ReLU with alpha=0.01 outputs 0.01 x (-3) = -0.03 and passes back a local gradient of 0.01 — still small, but not zero, so the neuron remains updatable.
4. The 145× figure was measured across only three layers. Because each additional sigmoid layer multiplies the surviving gradient by roughly another factor at or below 0.25 while each ReLU layer (when active) multiplies by 1, the gap compounds geometrically with depth — at ten layers the sigmoid network's gradient could be smaller by a factor of many thousands, not 145, which is why the ReLU switch mattered so much more for genuinely deep networks than a shallow one.
5. Swish(3) = 3 x σ(3) = 3 x 0.95257 = 2.8577.
6. The flaw is treating z . σ(z) as if only σ(z)'s sign matters. σ(z) is indeed always positive (it maps into (0,1)), but Swish multiplies it by z itself, and z can be negative. For any z < 0, Swish(z) = z . σ(z) is a negative number times a positive number, which is negative — e.g. Swish(-1) = -1 x 0.2689 = -0.2689, confirming Swish does output negative values.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where activation functions: from relu to swish is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting activation functions: from relu to swish to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind activation functions: from relu to swish, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.