Modern Activation Functions: ReLU, GELU, and Beyond

A network that stops learning at 62%

A team building a paddy-leaf disease classifier for a Krishi Vigyan Kendra field trial trains a 50-layer convolutional network on drone imagery: healthy leaf, bacterial blight, brown spot, tungro. Training accuracy climbs for the first six epochs, then flatlines at 62% while the loss curve goes almost perfectly flat. The obvious suspects get checked first: learning rate, data augmentation, class imbalance. None of them move the needle. Someone finally instruments the network layer by layer and finds the actual fault: in three of the deeper convolutional blocks, more than 40% of the channels output exactly zero for every single image in the validation set, not just some images — every image. Those channels are not "weak." They are dead. Their weights have not moved in four epochs, because the gradient reaching them is exactly zero, every step, regardless of what image is shown.

This is the dying ReLU problem, and it is the starting point for understanding why deep learning moved from a single dominant activation function in the mid-2010s to a small zoo of alternatives — Leaky ReLU, ELU, GELU, Swish, Mish, and the gated variants (SwiGLU, GEGLU) that power LLaMA, PaLM, and Mistral today. Each one is a specific, motivated fix to a specific failure mode of the one before it. This chapter builds that lineage from the mechanism outward: what an activation function's derivative actually does inside backpropagation, why ReLU's specific derivative shape creates the exact failure above, and what each successor changes about that shape and why.

What an activation function actually has to do

Inside any layer, a neuron computes a pre-activation z = w·x + b, a linear function of its inputs, and then applies a nonlinearity a = f(z). Stack linear layers without a nonlinearity in between and the whole network collapses algebraically into one linear function no matter how many layers you stack — W2(W1x) = (W2W1)x is still just a matrix multiply. The nonlinearity is what lets depth buy anything.

But the choice of which nonlinearity matters for a second, less obvious reason that has nothing to do with what the function computes going forward and everything to do with what its derivative does going backward. Backpropagation multiplies gradients along the chain rule, and at every activation the running gradient gets multiplied by f′(z) at that neuron's pre-activation value. If f′(z) is close to zero, the gradient arriving at every parameter upstream of that neuron gets crushed toward zero too — no matter how large the error signal was downstream. The entire history of activation function design is a history of controlling the shape of f′(z): where it saturates, where it vanishes, whether it is ever exactly zero, and how expensive it is to compute on real hardware.

The legacy problem: sigmoid and tanh saturate everywhere

Before ReLU, the default nonlinearities were the logistic sigmoid σ(z) = 1/(1+e^{-z}) and tanh(z). Both are smooth and bounded, which sounds like a virtue, but boundedness is exactly the problem: σ′(z) = σ(z)(1-σ(z)) peaks at 0.25 when z=0 and decays toward zero as |z| grows in either direction. Stack ten sigmoid layers and a typical pre-activation lands somewhere away from zero often enough that the product of ten derivatives, each well under 1, shrinks toward numerical noise. This is the classical vanishing-gradient problem, and it is why pre-2011 deep networks beyond a handful of layers were notoriously hard to train. Sigmoid and tanh saturate — squash toward a flat asymptote — on both sides.

ReLU: sparsity, cheap compute, and a new failure mode

Nair and Hinton's 2010 paper "Rectified Linear Units Improve Restricted Boltzmann Machines" and Glorot, Bordes, and Bengio's 2011 "Deep Sparse Rectifier Neural Networks" popularized the Rectified Linear Unit:

ReLU(z) = max(0, z)
ReLU'(z) = 1  if z > 0
         = 0  if z < 0   (undefined at z = 0, conventionally set to 0 or 1)

ReLU fixes half of the sigmoid problem outright: for any z > 0, the derivative is exactly 1, not a fraction less than 1. Gradients pass through the active half of the network completely undiminished — no saturation, no shrinkage, arbitrarily deep. It is also essentially free to compute (one comparison, no exponentials), which matters enormously at the scale of billions of activations per forward pass, and it induces genuine sparsity: roughly half the units in a randomly initialized network are off for a given input, which has a regularizing, disentangling effect that the RBM and sparse-rectifier papers both measured empirically.

But look at the other half of that derivative: ReLU′(z) = 0 for every z < 0, not a small number — exactly zero, no matter how negative z gets. A sigmoid unit deep in its saturated tail still has a derivative like 10⁻⁶: minuscule, but not literally zero, so in principle gradient still flows, just slowly. A ReLU unit with a negative pre-activation contributes nothing to any upstream gradient, ever, for that input. If a large weight update pushes a neuron's pre-activation negative for its entire input distribution — not just one example, but every example the network will ever see — the local gradient is zero on every subsequent step. The neuron cannot recover through gradient descent, because gradient descent has literally no signal to tell it which direction would help. This is exactly what starved 40% of the channels in the paddy-leaf network: an aggressive learning rate early in training pushed those channels' biases sharply negative, and once negative, no data point could ever pull them back.

Worked example: watching a neuron die

Take a single neuron with one input, weight w, bias b, and squared-error loss L = ½(a - y)². Initialize w₀ = 1.5, b₀ = 0.5. Feed it a training example where the target output is zero — for instance, a feature that this particular channel is not supposed to fire on: x₁ = 2, y₁ = 0.

z0 = w0*x1 + b0 = 1.5*2 + 0.5 = 3.5
a0 = ReLU(z0) = 3.5                    # z0 > 0, so ReLU is just identity here
L0 = 0.5*(a0 - y1)^2 = 0.5*(3.5)^2 = 6.125

dL/da = a0 - y1 = 3.5
da/dz = 1                              # z0 > 0, ReLU'=1
dL/dz = 3.5 * 1 = 3.5
dL/dw = dL/dz * x1 = 3.5 * 2 = 7.0
dL/db = dL/dz * 1  = 3.5

Now apply a gradient step with a learning rate of 2 — high, but exactly the kind of value that shows up early in training before a schedule has warmed down, or on a channel whose gradient happens to be larger than its neighbors':

w1 = w0 - lr*dL/dw = 1.5 - 2*7.0  = -12.5
b1 = b0 - lr*dL/db = 0.5 - 2*3.5  = -6.5

Recompute the forward pass on the same input:

z1 = w1*x1 + b1 = -12.5*2 + (-6.5) = -31.5
a1 = ReLU(z1) = 0
L1 = 0.5*(0 - 0)^2 = 0

The loss on this example has dropped to exactly zero — the update looks, by the loss curve alone, like a clean success. But the neuron has been pushed so far negative that it will output zero for essentially any realistic input, not just x=2. Confirm this on a second, unrelated training example where this channel is supposed to fire strongly: x₂ = 5, y₂ = 10.

z2 = w1*x2 + b1 = -12.5*5 + (-6.5) = -69.0
a2 = ReLU(z2) = 0                       # wildly wrong, should be near 10

dL/da = a2 - y2 = 0 - 10 = -10
da/dz = 0                               # z2 < 0, ReLU'=0
dL/dz = -10 * 0 = 0
dL/dw = 0 * 5 = 0
dL/db = 0

The error on this example is enormous — the network is off by 10 units — yet every gradient touching w and b is exactly zero, because ReLU′ is exactly zero at z₂ = -69. No optimizer, no learning-rate schedule, no amount of additional training data changes this: the chain rule multiplies the huge error by a hard zero and the product is zero. This neuron is now permanently dead, and it will stay dead until the network is reinitialized. That is the dying ReLU problem, traced through actual numbers rather than described qualitatively — and it is a strictly worse failure than sigmoid's vanishing gradient, because vanishing gradients merely shrink learning; a dead ReLU stops it exactly and permanently.

Patching the leak: Leaky ReLU, PReLU, ELU

The direct fix is to stop the derivative from ever touching exactly zero. Maas, Hannun, and Ng's 2013 paper "Rectifier Nonlinearities Improve Neural Network Acoustic Models" introduced Leaky ReLU:

LeakyReLU(z) = z        if z > 0
             = α·z      if z ≤ 0        (α ≈ 0.01, a small fixed constant)
LeakyReLU'(z) = 1        if z > 0
              = α        if z ≤ 0

The negative-side slope is small, but it is not zero. A dead-looking neuron still receives a gradient scaled by α, small but real, so gradient descent can in principle walk it back out of the dead region. He, Zhang, Ren, and Sun's 2015 paper "Delving Deep into Rectifiers" made α a learned parameter per channel instead of a fixed constant (Parametric ReLU, PReLU), letting the network decide per-neuron how much negative-side leak it wants — the same paper that first reported neural networks surpassing human-level accuracy on ImageNet classification. Clevert, Unterthiner, and Hochreiter's Exponential Linear Unit (ELU, 2015/ICLR 2016) took a different route on the negative side: ELU(z) = z for z>0, and ELU(z) = α(eᶻ-1) for z≤0, which saturates smoothly toward -α rather than growing linearly, giving the negative branch a soft floor and pushing mean activations closer to zero, which the paper showed speeds convergence.

GELU: a probabilistic reframing

Hendrycks and Gimpel's 2016 paper "Gaussian Error Linear Units (GELUs)" approached the problem from a different angle entirely: instead of hand-designing a piecewise negative-side slope, treat the gate itself as stochastic. Multiply the input by a Bernoulli mask that fires with probability Φ(z), the standard normal CDF — larger z is more likely to pass through unchanged, smaller (more negative) z is more likely to be zeroed, exactly like ReLU's hard gate but graded by probability instead of a hard threshold at zero. Taking the expectation over that stochastic gate gives a deterministic closed form:

GELU(z) = z · Φ(z) = z · ½[1 + erf(z/√2)]

Because Φ is smooth everywhere, so is GELU — there is no kink at z=0 the way there is for ReLU, and its derivative, GELU′(z) = Φ(z) + z·φ(z) (where φ is the standard normal density), is never exactly zero for any finite z. Verify this at the exact coordinates from the dying-neuron example above: GELU′(-31.5) ≈ -4.31 × 10⁻²¹⁵ — astronomically small, but not the hard zero that killed the ReLU channel. A GELU unit pushed into the same deeply negative region as the dead ReLU neuron above is not technically un-killable, but the failure mode is qualitatively different: infinitesimal gradient rather than none.

The erf function has no elementary closed form, so production code almost never calls it directly. Hendrycks and Gimpel's paper gives a tanh-based approximation that BERT, GPT-2, and most transformer implementations actually use:

GELU_approx(z) ≈ 0.5·z·(1 + tanh( √(2/π) · (z + 0.044715·z³) ))

Checking both forms at z = 1.0: Φ(1.0) = 0.8413447, so GELU_exact(1.0) = 1.0 × 0.8413447 = 0.8413447. The tanh approximation gives an inner term √(2/π)·(1 + 0.044715) = 0.833562, and tanh(0.833562) = 0.682384, so GELU_approx(1.0) = 0.5×(1+0.682384) = 0.841192. The two differ by roughly 1.5×10⁻⁴ — close enough that no published architecture reports a measurable quality difference between the exact and approximate forms, which is precisely why the cheaper tanh version won in practice. Note also that Φ(1.0) = 0.8413447 and erf(1/√2) = 0.6826895 are the same numbers, dressed differently, that give the familiar "68% of a normal distribution lies within one standard deviation" — GELU literally is that statistic, applied as a gate.

Unlike ReLU, GELU is non-monotonic: it dips slightly below zero for moderately negative inputs before flattening toward zero. The minimum sits at z ≈ -0.75, where GELU(-0.75) ≈ -0.170, and the derivative there is actually negative (GELU′(-1) ≈ -0.083) — increasing z slightly in that region decreases the output. This small negative lobe is deliberate: it lets a unit represent "I want a small negative signal here," something a strictly non-negative function like ReLU can never do, and empirically it helps gradient-based search escape flat regions near the origin.

Swish, SiLU, and Mish: smoothness by search

Ramachandran, Zoph, and Le's 2017 paper "Searching for Activation Functions" ran a neural architecture search over a space of candidate activation functions and found Swish(z) = z·σ(z) — the input times its own sigmoid — as the best performer across several vision benchmarks. The identical function had actually appeared a year earlier under the name SiLU (Sigmoid-weighted Linear Unit) in Elfwing, Uchibe, and Doya's reinforcement-learning paper, an example of independent rediscovery that is common enough in this literature to be worth knowing rather than assuming one paper's naming is canonical. Swish/SiLU shares GELU's core shape — smooth, non-monotonic, a small negative dip (minimum at z ≈ -1.278, value ≈ -0.278) — but is built from a plain sigmoid rather than the Gaussian CDF, which is slightly cheaper and, critically, exactly recoverable rather than approximated: Swish(2) = 2×σ(2) = 1.761594, computed with no approximation error at all, versus GELU's inherent erf-approximation trade-off. Misra's 2019 paper "Mish: A Self Regularized Non-Monotonic Activation Function," Mish(z) = z·tanh(softplus(z)), pushes the same idea further with an even smoother, unbounded-above, softly-floored curve, reporting small but consistent gains over both ReLU and Swish on image classification benchmarks, though it has seen far less production adoption than GELU or SiLU because of its higher compute cost per activation.

Gating instead of squashing: GLU, SwiGLU, and the modern transformer FFN

Every function discussed so far answers the same question — "how much of this single number should pass through?" A Gated Linear Unit answers a related but structurally different question: given two linear projections of the same input, how much of one should be let through, controlled by the other? Shazeer's 2020 paper "GLU Variants Improve Transformer" tested this inside the transformer's feed-forward block, replacing the standard two-matrix FFN with a three-matrix gated version:

# standard transformer FFN (two weight matrices, ReLU or GELU)
FFN(x) = W2 · f(W1 · x)

# GLU-variant FFN (three weight matrices)
FFN_GLU(x) = W2 · ( f(W1·x) ⊙ (V·x) )     # ⊙ = elementwise multiply

W1·x is squashed by an activation function exactly as before, but the result is then multiplied elementwise by a second, entirely separate linear projection V·x that acts as a per-dimension gate — the network learns which components of the transformed signal to admit and which to suppress, conditioned on the same input that produced them. SwiGLU pairs this gating structure with the Swish/SiLU activation for f; GEGLU pairs it with GELU. Both are now the default feed-forward block in LLaMA, PaLM, and Mistral, replacing the plain-GELU FFN that BERT and GPT-2 used. A third weight matrix is not free — to hold the FFN's total parameter count and FLOPs roughly constant against a two-matrix baseline, Shazeer's paper scales the hidden dimension of the gated version down to about two-thirds of the ungated hidden width (three matrices at 2/3 the width ≈ two matrices at full width, in total multiply-adds), which is why LLaMA-family configs report FFN hidden dimensions that look like odd fractions of 4×d_model rather than the round number BERT used. The payoff Shazeer measured was a small but consistent perplexity improvement across several language-modeling benchmarks, for the same compute budget — a case where the "activation function" question and the "architecture" question stop being separable.

Reading the curves

Two structural facts matter more than the exact curve shapes. First, ReLU has a genuine kink at z=0 — its derivative jumps discontinuously from 0 to 1 — while GELU and Swish are smooth there, with no discontinuity in either the function or its derivative; this smoothness is why they tend to produce better-conditioned loss surfaces at the second-derivative level, which matters for optimizers that implicitly or explicitly use curvature information. Second, GELU and Swish both dip slightly below the x-axis for moderately negative inputs before flattening out, while Leaky ReLU keeps descending linearly forever (with slope α) and plain ReLU is pinned at exactly zero — the flat red dot on the left edge of the plot marks the failure mode from the worked example above.

Production tradeoffs: the "best" function isn't always the one used

If GELU and Swish are strictly more expressive and less failure-prone than ReLU, why does ReLU remain the default in a large fraction of production vision models and in every efficiency-sensitive edge deployment? Three engineering reasons, layered on top of the numerical ones above. First, raw compute: ReLU is a single comparison per element; GELU (exact) requires an error function, and even the tanh approximation needs a cubic, a multiply by an irrational constant, and a hyperbolic tangent — on a GPU or TPU processing billions of activations per forward pass, this difference shows up directly in wall-clock training and inference time, which is precisely why the tanh approximation exists at all rather than computing erf exactly. Second, quantization: ReLU's piecewise-linear shape survives aggressive int8 quantization for edge inference cleanly, because a linear function on each side of a hard threshold quantizes with predictable, bounded error; GELU and Swish's curved, non-monotonic shape is harder to represent faithfully with a handful of quantization levels, which matters when a model has to run inside a phone's NPU power budget rather than a data-center GPU. Third, memory bandwidth in serving: inference at scale is frequently bandwidth-bound rather than compute-bound, and an activation function itself is rarely the bottleneck once the model is deployed — the FFN block's total FLOPs and parameter count (the GLU-variant tradeoff discussed above) dominate cost far more than which pointwise nonlinearity sits inside it. This is why the choice in practice tracks the deployment target: ReLU or Leaky ReLU for latency- and power-constrained vision inference, GELU inside transformer encoders where compute is already dominated by attention and matrix multiplies, and SwiGLU inside the largest decoder-only language models where the architecture search has already been done at enormous scale by the labs that trained them.

The misconception to unlearn

Students who have just learned that "ReLU solves the vanishing gradient problem" tend to generalize this into "ReLU cannot suffer from a vanishing-gradient-style failure, because it's linear and doesn't saturate." That is only half true, and the missed half is the more dangerous one. ReLU removes saturation on the positive side of the input, where its derivative is a constant 1 forever, no matter how large z gets. But on the negative side, its derivative is not small — it is exactly, permanently zero, which is a strictly harder failure than the sigmoid's slow decay toward zero. A saturated sigmoid neuron can, in principle, recover given enough training steps and a strong enough error signal, because its gradient is merely tiny, never exactly nothing. A dead ReLU neuron, as the worked trace above shows precisely, cannot recover under gradient descent at all, regardless of the size of the error, because zero times any finite number is exactly zero. "ReLU doesn't saturate" is true for one half of its domain and false for the other; the dying-ReLU literature exists specifically because engineers first believed the half-truth and then had to explain 40%-dead networks in production.

Active recall

Attempt each question before reading its answer.

1. A neuron has pre-activation z = 2.0. Compute GELU(2.0) using the exact erf form and Swish(2.0), and state which is larger.
2. Explain, in terms of the chain rule and f′(z), why a sigmoid network's gradient problem is called "vanishing" but a ReLU network's equivalent problem is called "dying" rather than "vanishing." Are these the same failure mode?
3. Take the worked dying-ReLU example (w0=1.5, b0=0.5, example 1 x1=2, y1=0). Redo the single gradient step using Leaky ReLU with α=0.01 instead of ReLU, and lower the learning rate from 2 to 0.5. Report w1, b1, and the new a1 for example 1, then check whether example 2 (x2=5, y2=10) still produces a zero gradient.
4. A transformer's standard FFN block uses two weight matrices; a SwiGLU FFN uses three. Why does the SwiGLU version typically use a smaller hidden dimension than the standard version, rather than the same one?
5. True or false, with justification: "GELU is a strictly increasing function of z, just like ReLU, only smoother."

Worked answers

1. Φ(2.0) ≈ 0.977250, so GELU(2.0) = 2.0 × 0.977250 = 1.954500. σ(2.0) ≈ 0.880797, so Swish(2.0) = 2.0 × 0.880797 = 1.761594. GELU is larger — at z=2, Φ(z) has already climbed closer to 1 than σ(z) has, so GELU passes more of the input through than Swish does at this particular point (the two curves cross and re-cross; neither dominates everywhere).

2. Both failures come from the same chain-rule mechanism — a small or zero local derivative multiplying and shrinking the upstream gradient — but they differ in degree, not in the mechanism. "Vanishing" describes a gradient that shrinks toward zero but is never exactly zero at any single unit; it decays multiplicatively across many layers, and in principle a large enough error signal or long enough training can still move the parameter, just slowly. "Dying" describes a gradient that is exactly zero at a single unit, in a single layer, for every input in the current data distribution, permanently — no amount of additional training moves the parameter, because zero times anything is zero. Dying ReLU is a harder, more absolute version of the same underlying idea, not a different phenomenon.

3. Because z0 = 3.5 > 0, Leaky ReLU behaves identically to ReLU at the initial forward pass (slope 1 in the positive region for both), so the gradients dL/dw = 7.0 and dL/db = 3.5 are unchanged from the original example. With lr=0.5: w1 = 1.5 - 0.5×7.0 = -2.0, b1 = 0.5 - 0.5×3.5 = -1.25. Forward pass on example 1: z1 = -2.0×2 + (-1.25) = -5.25, and since Leaky ReLU never hard-zeros, a1 = 0.01×(-5.25) = -0.0525 — close to the target of 0 but not clipped to exactly 0. On example 2: z2 = -2.0×5 + (-1.25) = -11.25, a2 = 0.01×(-11.25) = -0.1125, giving dL/da = -0.1125-10 = -10.1125, and because the Leaky ReLU slope on the negative side is α=0.01 rather than 0, dL/dz = -10.1125×0.01 = -0.101125, which propagates to dL/dw2 = -0.505625 and dL/db2 = -0.101125 — both nonzero. Two separate changes combined here: switching to Leaky ReLU is what keeps the gradient on example 2 from being exactly zero, while lowering the learning rate is what keeps the step from overshooting nearly as far into negative territory in the first place (compare b1=-1.25 here against b1=-6.5 in the original). Either change alone would have helped; together they leave the neuron both alive and much closer to its starting region.

4. The extra gate matrix V adds a third full d_model × d_ff weight matrix to the block. If the hidden width d_ff were left unchanged, the gated FFN would cost 1.5× the parameters and FLOPs of the standard two-matrix version for the same width, which would make any quality comparison unfair — you'd be comparing a bigger model to a smaller one, not one activation choice to another. Shazeer's paper instead shrinks d_ff for the gated version, roughly to two-thirds of the standard width, so that three matrices at the smaller width cost approximately the same total compute as two matrices at the original width — an apples-to-apples comparison at matched parameter count and FLOPs, which is also why LLaMA-family models report FFN hidden dimensions that don't look like round multiples of d_model.

5. False. ReLU is monotonic non-decreasing everywhere, but GELU is not: it dips slightly negative for moderately negative z, reaching a minimum around z ≈ -0.75 where GELU(-0.75) ≈ -0.170, before flattening back toward zero as z → -∞. In that dip, GELU′(z) is actually negative — for instance GELU′(-1) ≈ -0.083 — meaning a small increase in z around that point decreases the output. GELU is smoother than ReLU, but "smoother" and "monotonic" are independent properties, and GELU trades away the second to gain a small negative-output region that ReLU structurally cannot represent.

Think About It

Think about this: How would you explain modern activation functions: relu, gelu, and beyond to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind modern activation functions: relu, gelu, and beyond, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.