UIDAI's Face Authentication system lets a ration-shop dealer or a bank correspondent verify an Aadhaar holder's identity by matching a live camera photo against the face stored in the Aadhaar database, without a fingerprint scanner. The matcher is a convolutional neural network trained on millions of faces, and on clean photographs it is extremely accurate — accuracy figures in the high nineties are routinely reported for such systems. Now suppose someone prints a photograph of person A's face with a barely visible grid of coloured dots overlaid on it — a pattern too faint for a human doorstep-verification agent to notice, but computed precisely from the network's own mathematics. Held up to the camera, that photograph is confidently matched by the network to person B. Nothing about the image looks tampered with to a human eye. The network is not confused because it is a bad network; it is confused because someone did calculus on it. That is the subject of this chapter: how a model that is mathematically well-behaved — smooth, differentiable, trained to convergence — is exactly the kind of model that is easiest to steer to a wrong answer with a change too small to see.
What an adversarial example actually is
Let fθ be a trained classifier with parameters θ, and let x be a legitimate input that the network classifies correctly: fθ(x) = y. An adversarial example is a perturbed input x' = x + δ such that two conditions hold simultaneously:
1. Perceptual constraint: δ is small under some distance measure — usually an L∞ bound ‖δ‖∞ ≤ ε, meaning no single coordinate of the input changes by more than ε. Under this bound a human (or a downstream sensor) is meant to judge x' as essentially the same input as x.
2. Behavioural constraint: the network's answer changes. In an untargeted attack, any wrong answer counts: fθ(x') ≠ y. In a targeted attack, the attacker fixes a specific wrong label in advance and forces fθ(x') = ytarget — the harder and more dangerous case, since it lets an attacker choose exactly which mistake the model makes.
Attacks are further classified by how much the attacker is allowed to see. A white-box attacker has the full parameter set θ and can compute exact gradients — this is the setting a security auditor uses to measure worst-case robustness. A black-box attacker only gets to submit inputs and observe outputs (as an attacker probing a live UPI fraud-detection API would). Black-box attacks are still highly effective, for reasons explained later in this chapter, which is why "the attacker doesn't know our model's weights" is not a valid line of defence on its own.
Why a well-trained network is vulnerable: the linearity argument
The instinctive explanation — "the model hasn't been trained enough" or "it's overfitting" — is wrong, and correcting it is the single most important conceptual move in this chapter. Goodfellow, Shlens and Szegedy showed in 2014 that adversarial vulnerability follows from the opposite property: modern networks are, layer by layer, close to linear functions of their input (ReLU, matrix multiplication, and even the "kink" in a max-pool are all piecewise-linear or nearly so). A linear function of a high-dimensional input is exactly what makes tiny, coordinated perturbations dangerous.
Consider the simplest possible classifier, logistic regression, which is just the last layer of every deep network with the earlier layers stripped away: z = w·x + b, ŷ = σ(z) = 1/(1+e−z), decision boundary at z = 0. If we perturb every coordinate of x by a tiny amount ε, chosen with its sign matched to the sign of the corresponding weight wi, the logit moves by
Δz = Σᵢ ε · sign(wᵢ) · wᵢ = ε · Σᵢ |wᵢ| = ε · ‖w‖₁Each individual perturbation is invisible — bounded by ε — but the total shift in the logit grows with the L1 norm of the weight vector, which itself grows with the number of input dimensions. A colour photograph fed to a CNN has on the order of 150,000 input dimensions (224×224 pixels × 3 colour channels). Even if every one of those 150,000 pixel-weight products is individually microscopic, their sum is not: ε · ‖w‖₁ can comfortably exceed the distance to the decision boundary, flipping the prediction, while every single pixel changed by an amount far below what a human eye can register. High dimensionality is not a defence here — it is the attacker's leverage. This is exactly why the attack described in the opening paragraph works on a photograph but would not work nearly as well on, say, a 3-input medical triage score: fewer dimensions means less L1 norm to exploit for the same per-coordinate budget.
The Fast Gradient Sign Method (FGSM)
The linearity argument turns directly into an attack algorithm. If the network is locally close to linear, the direction that increases the loss fastest is the gradient of the loss with respect to the input, and since we only care about the sign of each coordinate (not its magnitude — the perturbation budget is fixed at ε per coordinate), we take the sign of that gradient:
x' = x + ε · sign( ∇ₓ L(θ, x, y) )This is the Fast Gradient Sign Method. It costs exactly one forward pass and one backward pass — the same backpropagation machinery used to train the network, run once, with the roles reversed: instead of nudging the weights to reduce the loss for a fixed input, FGSM nudges the input to increase the loss for fixed weights. Everything a network learns about computing gradients efficiently for training is immediately reusable for attacking it.
Worked example: flipping a fraud classifier's decision
Take a toy 2-feature logistic-regression fraud detector — small enough to trace by hand, but the arithmetic is identical to what a real gradient step performs. Let the two features be normalized transaction signals (say, deviation from the user's average transaction amount, and deviation from their usual transaction hour), with learned weights w = [3, −1], bias b = 0, and label convention y = 1 meaning "genuine". Feed in a real, genuine transaction x = [1, 1]:
z = w·x + b = 3(1) + (−1)(1) + 0 = 2
ŷ = σ(2) = 1 / (1 + e⁻²) = 0.8808The model is confident and correct: 88.08% probability of "genuine", well above the 0.5 threshold. The cross-entropy loss for the true label y = 1 is L = −ln(ŷ) = −ln(0.8808) = 0.1269. To attack the input, differentiate the loss with respect to x. Using the standard logistic-regression identity ∂L/∂z = ŷ − y and the chain rule ∂z/∂x = w:
∇ₓL = (ŷ − y) · w = (0.8808 − 1) · [3, −1] = [−0.3576, 0.1192]
sign(∇ₓL) = [−1, +1]Choose a perturbation budget ε = 0.6 per feature (deliberately generous here, for arithmetic clarity — in a real pixel-space attack ε is tiny, e.g. 8/255 ≈ 0.031 per channel, and the effect still accumulates as shown above). The FGSM step is:
δ = 0.6 · [−1, +1] = [−0.6, +0.6]
x' = x + δ = [1 − 0.6, 1 + 0.6] = [0.4, 1.6]Re-run the classifier on x':
z' = 3(0.4) + (−1)(1.6) = 1.2 − 1.6 = −0.4
ŷ' = σ(−0.4) = 1 / (1 + e⁰·⁴) = 0.4013The prediction has flipped from 0.8808 ("genuine", correct) to 0.4013 ("fraud", wrong) — a full misclassification produced by moving each of the two features by at most 0.6 in a coordinated direction computed from a single gradient. Notice the shortcut formula from the previous section checks out exactly: predicted shift = ε · sign(ŷ−y) · ‖w‖₁ = 0.6 · (−1) · (3+1) = −2.4, and indeed z' = z + (−2.4) = 2 − 2.4 = −0.4. That identity is not a coincidence — it is the whole mechanism of the attack laid bare in one line: the sign function aligns every coordinate of the perturbation with the corresponding weight, so the shift in the logit is always the full L1 norm of the weights times ε, the largest possible shift achievable with an L∞ budget of ε. FGSM is, in this precise sense, optimal against a linear model. The diagram below plots this exact example.
Look at where x' lands relative to the dashed square (the set of every input FGSM is allowed to reach): it sits exactly on a corner. That is not an artifact of these particular numbers — it is a structural property of the sign(·) function. Every coordinate of the gradient gets mapped to exactly +ε or −ε, never anything in between, so FGSM always jumps to a vertex of the L∞ box around x, never to an interior point. That vertex is provably the point inside the box that maximizes the first-order (linear) approximation of the loss increase — which is precisely why the argument in the previous section calls FGSM optimal against a linear model, and only approximately optimal against a real, mildly nonlinear network.
PGD: what happens when one gradient step isn't enough
Real networks are not perfectly linear, so a single FGSM step, computed from the gradient at the original point x, is only a good direction near x. Projected Gradient Descent (PGD) repeats the FGSM idea in small increments, re-evaluating the gradient at each new point and clipping back into the allowed ε-ball after every step:
x₀ = x
x_{t+1} = Π_{x+S} ( x_t + α · sign(∇ₓ L(θ, x_t, y)) )where α is a small step size (much smaller than ε), S is the allowed perturbation set (the L∞ ball), and Π projects any point that steps outside the box back onto its boundary. Because it follows the true (locally curved) loss surface rather than trusting one linear guess, PGD with enough steps reliably finds a stronger adversarial example than FGSM for the same total budget ε — at the cost of many forward/backward passes instead of one. PGD is the standard benchmark attack used to measure a model's worst-case robustness, precisely because it is close to the strongest first-order attack achievable within the budget.
You don't need the weights: transferability and physical attacks
The worked example above assumed white-box access — the attacker could compute the exact gradient because they had w and b. Real deployed systems don't hand out their weights. Two facts make attacks feasible anyway. First, transferability: adversarial examples crafted against one model frequently fool a completely different model trained on similar data for the same task, even with a different architecture. An attacker can train their own substitute classifier, generate adversarial examples against it using FGSM or PGD, and fire those same inputs at the real target with a meaningful success rate — because both models, trained on overlapping data distributions, tend to carve similar decision boundaries and share similar vulnerable directions. Second, purely query-based black-box attacks estimate the gradient numerically by observing how the output score changes across many submitted inputs, without ever needing the parameters directly.
These attacks are not confined to digital pixels handed directly to a model. Eykholt et al. (2018) demonstrated that a handful of small black-and-white stickers placed on a real stop sign, computed via an optimization very similar to FGSM/PGD but constrained to be robust across viewing angles, lighting, and distance, reliably caused a road-sign classifier to read the sign as a speed-limit sign — a "physical-world" adversarial example, surviving a camera sensor, printer, and outdoor conditions between the computed perturbation and the network's input. The mechanism is the same linearity-driven gradient step this chapter has derived by hand; only the delivery medium changes, from a change to a pixel array to a change to a physical surface.
The misconception to unlearn
The single most common wrong intuition here is: "a highly accurate model must be adversarially robust — if it were easy to fool, it wouldn't score so well." This conflates two entirely different axes. Test accuracy measures performance on the natural data distribution — inputs that look like ordinary photographs, ordinary transactions. Adversarial robustness measures worst-case performance over an entire ε-ball around each natural input, most of which the model never saw during training and was never optimized to handle. A network can carve an extremely accurate decision boundary through the cloud of natural data points and still have that boundary pass astonishingly close to almost every one of those points in some direction the training data never probed — which is exactly what the linearity argument predicts, and exactly what the worked example demonstrated: a model that was 88% confident and correct was flipped to a wrong answer by a single, cheaply computed gradient step. State-of-the-art ImageNet classifiers with over 95% top-5 accuracy are routinely reduced to near-zero accuracy under an L∞ budget of ε = 8/255, a perturbation invisible in a side-by-side image comparison. Accuracy and robustness are correlated with nothing about each other by default; robustness has to be trained for explicitly.
A brief word on defence
The most reliable known defence is adversarial training: instead of training only on clean data, replace each training batch (or augment it) with adversarial examples generated on the fly, typically via PGD, and train the network to get those right too. This is a min-max optimization — the attacker (inner loop) tries to maximize the loss within the ε-ball around each training point, and training (outer loop) minimizes that worst case. It measurably improves robustness but is expensive (each training step now costs several forward/backward passes) and typically trades away a few points of clean accuracy. A trap worth naming explicitly: many published defences that looked effective turned out to work only by gradient masking — making the loss surface jagged or non-differentiable enough that gradient-based attacks like FGSM/PGD simply fail to find a good direction, without the model actually being robust. Athalye et al. (2018) broke the large majority of such defences from a single ICLR year using adaptive attacks that worked around the masking, which is why any robustness claim today is expected to be tested against an attack specifically adapted to that defence, not just the default FGSM/PGD recipe.
Active recall
Attempt each question before reading its answer.
1. A binary classifier has w = [2, −4], b = 1. For genuine input x = [1, 0.5], y = 1. Using FGSM with ε = 0.3, compute the adversarial x' and its new prediction. Does the attack succeed?
2. Explain, in one or two sentences, why the same per-pixel perturbation budget ε is far more dangerous against a 150,000-dimensional image input than against a 3-feature tabular input.
3. What is the difference between a targeted and an untargeted adversarial attack, and which requires solving a harder optimization problem?
4. True or false, with justification: "A 99% accurate fraud model is inherently adversarially robust."
5. Why does PGD generally find a stronger adversarial example than single-step FGSM for the same total ε, and what does it cost to get that improvement?
6. In the worked diagram, why does the adversarial point x' always land exactly on a corner of the ε-box rather than somewhere inside it?
Worked answers
1. z = 2(1) + (−4)(0.5) + 1 = 2 − 2 + 1 = 1, so ŷ = σ(1) = 0.7311 — confidently genuine. Gradient: ∇ₓL = (ŷ−y)·w = (0.7311−1)·[2,−4] = [−0.538, 1.076], so sign(∇ₓL) = [−1, +1]. Perturbation: δ = 0.3·[−1,+1] = [−0.3, +0.3], giving x' = [0.7, 0.8]. New logit: z' = 2(0.7) − 4(0.8) + 1 = 1.4 − 3.2 + 1 = −0.8, so ŷ' = σ(−0.8) = 0.3100. Since 0.31 < 0.5, the prediction flips to "fraud" — the attack succeeds. (Check via the shortcut: shift = ε·sign(ŷ−y)·‖w‖₁ = 0.3·(−1)·6 = −1.8, and z' = 1 − 1.8 = −0.8 ✓.)
2. The shift in the logit produced by an aligned-sign perturbation equals ε · ‖w‖₁, the sum of the absolute weights across every input dimension. With 150,000 dimensions that sum is large even if each individual weight and each individual pixel change is tiny; with 3 dimensions the sum is bounded by three small terms and rarely clears the margin to the decision boundary. Dimensionality multiplies the attacker's leverage even though the per-coordinate visibility budget stays fixed.
3. Untargeted attacks only need f(x') ≠ y — any wrong class satisfies the attacker. Targeted attacks require f(x') = y_target, a single specific wrong class chosen in advance, which constrains the optimization far more (the perturbation must not just leave the correct class's region, it must land inside one particular other class's region), making it the harder, more expensive attack to mount.
4. False. Accuracy is measured on the natural data distribution; robustness is measured on the worst case across an entire neighbourhood around every natural point, a region training never directly optimizes over. The linearity argument and the worked example both show a confidently correct, accurate prediction can be flipped by a small, computed perturbation — accuracy and robustness are independent properties unless the model was explicitly trained (e.g., adversarially) to close that gap.
5. A single FGSM step commits to the gradient direction measured only at the starting point x, which is a good direction solely under the assumption that the loss surface is linear over the whole ε-ball — an assumption that gets worse as ε grows or the network's true nonlinearity increases. PGD re-measures the gradient after each small step and re-projects into the ε-ball, following the curved loss surface instead of one straight-line guess, so with enough iterations it converges much closer to the true worst-case point inside the ball. The cost is computational: PGD needs many forward/backward passes (one per step) instead of FGSM's one.
6. FGSM sets each coordinate of the perturbation to ε · sign(gradient coordinate), and sign(·) only ever outputs +1 or −1 — never a fractional value. So every coordinate is always pushed to its maximum allowed magnitude in the ε-box, which by definition means the resulting point sits on a vertex of the box, never in its interior.
Think About It
Think about this: How would you explain adversarial attacks: breaking neural networks to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where adversarial attacks: breaking neural networks is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting adversarial attacks: breaking neural networks to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind adversarial attacks: breaking neural networks, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.