In 2018, researchers at the University of Michigan and Washington took an ordinary stop sign and stuck a handful of black-and-white rectangles onto it — the kind of vinyl patches you'd assume were graffiti or wear-and-tear. To a human driver, it was still unmistakably a stop sign. To the camera-based traffic-sign classifier they were testing, it read as a Speed Limit sign, reliably, from multiple angles and distances, in a real parked car with a real camera (Eykholt et al., "Robust Physical-World Attacks on Deep Learning Visual Classification," CVPR 2018). Nothing about the network was broken. It had not been hacked in the sense of stolen credentials or injected code. Its weights, its training data, its architecture were all exactly as its designers intended. The attack worked entirely by exploiting the geometry of what the network had learned.
This matters for India specifically because camera-based Advanced Driver Assistance Systems — traffic-sign recognition, lane-keeping, automatic emergency braking — are no longer a research demo here. They ship in production SUVs from Tata, Mahindra and others sold on Indian roads today. And the same underlying vulnerability shows up far from cameras and stop signs: a bank's fraud-detection model scoring a UPI transaction, a resume-screening model, a churn predictor — any system where a classifier draws a boundary through a high-dimensional space is, in principle, exposed to inputs deliberately engineered to sit just across that boundary. This chapter builds the mathematics of exactly how that engineering works, why the standard first line of defense against it is weaker than it looks, and what actually helps.
What "adversarial" means here
A trained classifier partitions its input space into regions, one per class, separated by a decision boundary. For a model with parameters θ, an input x is correctly classified if it lands on the correct side of that boundary. An adversarial example is a point x′ = x + δ, where δ is a perturbation deliberately chosen to push x across the boundary into the wrong region, subject to a constraint that δ stays "small" by some measure — small enough that a human reviewer, or a downstream sanity check, wouldn't flag the input as tampered with.
"Small" is formalized with an Lp norm budget. For a perturbation vector δ = (δ1, …, δn):
- L∞ norm: maxi |δi| — the largest single-feature change. An L∞ budget of ε means every feature can move by at most ε, independently.
- L2 norm: √(Σ δi2) — total Euclidean displacement, allowing many small changes or a few large ones.
- L0 "norm": count of nonzero δi — how many features are touched at all, regardless of magnitude.
The choice of norm encodes what the attacker is actually constrained by. Image attacks typically use L∞ because it models a camera's or a display's per-pixel quantization limit. We'll use L∞ throughout this chapter because it's also the constraint that makes the algebra transparent.
Threat models further split along two independent axes. White-box attacks assume the attacker can read the model's weights and compute exact gradients; black-box attacks assume only query access (send an input, observe the output). Untargeted attacks just want any wrong answer; targeted attacks want a specific wrong answer (make the stop sign read as this exact speed limit, not just "not a stop sign"). A separate, orthogonal distinction is evasion (fool an already-trained model at inference time, which is our main focus) versus poisoning (corrupt the training data so the model itself is learned wrong — covered later in this chapter). Black-box attacks are more practically important than they sound, because adversarial examples exhibit transferability: a perturbation crafted against one model frequently fools a different model trained on similar data, even one with a different architecture (Papernot et al., 2017). An attacker without API access to the target bank's exact fraud model can train their own substitute model on similar transaction patterns, craft a white-box attack against their substitute, and have a good chance it works against the real target too.
FGSM: turning "small perturbation" into an actual number
The Fast Gradient Sign Method (Goodfellow, Shlens & Szegedy, 2014) is the simplest way to compute an L∞-constrained adversarial perturbation, and it's the right place to build intuition because for a linear model it is provably exact, not approximate. Given a loss function J(θ, x, y) measured against the true label y, FGSM sets:
x′ = x + ε · sign(∇x J(θ, x, y))
Read this as: at every feature, move in whichever direction (plus or minus) increases the loss for the correct label, by exactly ε. Using only the sign of the gradient rather than its full magnitude looks like it's throwing away information, but it's what makes the attack respect an L∞ budget exactly — every coordinate uses the full allowed ε, in whichever direction helps.
Worked example: evading a fraud classifier
Take a (deliberately small, for tractability) logistic-regression fraud detector used to score a transaction from two normalized features: x1 = transaction amount relative to the account's historical average, and x2 = account age. It outputs a fraud probability p = σ(z), z = w1x1 + w2x2 + b, with trained weights w1 = 1.2, w2 = −0.8, b = 0.1 (larger, more unusual amounts push the score up; older accounts pull it down). A transaction flagged p > 0.5 is held for review.
A genuinely fraudulent transaction arrives with x1 = 0.6, x2 = 0.3, true label y = 1 (fraud):
z = 1.2(0.6) − 0.8(0.3) + 0.1 = 0.72 − 0.24 + 0.1 = 0.58
p = σ(0.58) = 1 / (1 + e−0.58) ≈ 1 / 1.5599 ≈ 0.6411
Correctly flagged, 64.11% confidence. Now the attacker — who can nudge the reported amount and knows roughly how the account-age feature is computed — wants to push p below 0.5 without changing which underlying transaction it is. For binary cross-entropy loss, the gradient with respect to the logit z is exactly (p − y), a standard and easily re-derivable result, so by the chain rule ∇xJ = (p − y)·w:
p − y = 0.6411 − 1 = −0.3589
∂J/∂x1 = −0.3589 × 1.2 = −0.4307 → sign = −1
∂J/∂x2 = −0.3589 × (−0.8) = 0.2871 → sign = +1
With a perturbation budget ε = 0.35 per feature (a large but not absurd fudge on a self-reported/estimable field):
x1′ = 0.6 + 0.35(−1) = 0.25 x2′ = 0.3 + 0.35(+1) = 0.65
z′ = 1.2(0.25) − 0.8(0.65) + 0.1 = 0.30 − 0.52 + 0.1 = −0.12
p′ = σ(−0.12) ≈ 1 / 2.1275 ≈ 0.4699
The transaction now scores 46.99% — below the review threshold — while the underlying facts (it is still fraud) haven't changed at all. Here's a sanity check worth doing on every FGSM example: Δz = z′ − z = −0.12 − 0.58 = −0.70. Compare that to ε·‖w‖1 = 0.35 × (1.2 + 0.8) = 0.70. They match in magnitude (opposite sign, since Δz is negative here because the step minimizes the logit while ε·‖w‖1 is reported as a positive magnitude). That's not a coincidence: for a linear model, z is a linear functional of x, and a linear functional maximized (or here, minimized) over a hypercube of side 2ε is always extremized at a vertex of that cube — never in the interior, never on a face. The sign-of-gradient step is exactly the vertex selection rule. So for a linear model, FGSM isn't an approximation to the worst point inside the ε-ball; it is the worst point, found in one step.
That linear-model exactness also explains something Goodfellow's original paper emphasized: attack strength scales with the number of input dimensions n, even at fixed per-feature ε, because Δz = ε·‖w‖1, and ‖w‖1 tends to grow with n. A high-resolution image classifier has on the order of tens of thousands of pixel-features; nudging each one by an amount too small for a human to notice can still accumulate into a large, decision-flipping change in the logit — which is exactly why adversarial images look untouched to the eye while completely fooling the network, whereas our two-feature toy example needed a comparatively large ε = 0.35 per feature to cross the boundary at all.
Reading the geometry
Look at where the four gray "random noise" arrows land: every one of them stays inside the pink fraud region, because the not-fraud sliver is a thin wedge tucked in the box's top-left corner — a small fraction of the box's area, and a random direction has to get lucky to land in it. The purple arrow doesn't get lucky. It's computed, not sampled, and it goes precisely to the one corner of the box that the boundary actually cuts through. That contrast is the entire content of adversarial robustness as a security property: the attack surface isn't "the model is fooled by noise," it's "the model is fooled by search."
The misconception to unlearn: augmentation is not adversarial training
Because random-crop, random-noise, and color-jitter data augmentation is taught early in computer vision as a way to make models "more robust," it's a very natural but wrong inference that a model trained with heavy noise augmentation has also become adversarially robust. It has not, and the diagram above is exactly why. Data augmentation samples perturbations from a broad, unstructured distribution and asks the model to be correct on average across that distribution. FGSM and its stronger relatives don't sample — they solve an optimization problem to find the single worst point inside the allowed budget, using the model's own gradients as a map. A model can have excellent average-case accuracy under Gaussian pixel noise (because almost all noise directions land safely inside the correct region, exactly like the gray arrows) and still be trivially broken by an attacker who computes rather than guesses (exactly like the purple arrow), because the adversarial direction is a specific, thin, gradient-identifiable slice of the perturbation space rather than a randomly-reachable one.
The formally correct notion of adversarial training, due to Madry et al. (2017), makes this precise as a min-max optimization over the model parameters θ:
minθ E(x,y)~D [ max‖δ‖≤ε J(θ, x+δ, y) ]
The inner maximization is an attack — find the worst perturbation within budget, for the model as it currently stands. The outer minimization is ordinary training — update θ to do as well as possible against that worst case. Because deep networks aren't globally linear the way our two-feature model is, one FGSM step doesn't reliably solve the inner maximization; Projected Gradient Descent (PGD) is used instead, taking k small steps of size α and, after each step, projecting back onto the ε-ball (clipping any coordinate that has drifted outside [x−ε, x+ε]) so the budget is never exceeded even though the gradient direction keeps changing as the point moves:
x0 = x
x_t = x0
for t in range(k):
grad = gradient_of_loss_wrt_x(model, x_t, y_true)
x_t1 = x_t + alpha * sign(grad)
x_t1 = clip(x_t1, x - epsilon, x + epsilon) # project back into the L∞ ball
x_t1 = clip(x_t1, valid_input_min, valid_input_max)
x_t = x_t1
Training against PGD-found examples is far more expensive than clean training — every mini-batch now requires an inner attack loop — but it directly targets the actual worst case instead of an average case that random augmentation only ever samples from.
A defense that looks like it works and doesn't: gradient masking
A tempting shortcut, once you know attacks like FGSM need ∇xJ, is to make that gradient hard to compute: round inputs to a small set of values, run them through a non-differentiable JPEG-style compression step, or otherwise break the smooth computational graph an attacker would need to backpropagate through. This class of defense reliably makes gradient-based white-box attacks fail to find anything — and reliably fails to make the model actually safer. Athalye, Carlini and Wagner (2018) went through the majority of adversarial defenses accepted at ICLR that year and showed most of them relied on exactly this trick, which they named obfuscated gradients, and broke nearly all of them using techniques like BPDA (approximating the missing gradient with a differentiable stand-in) or simply attacking a substitute model and relying on transferability. The lesson generalizes past this one paper: hiding the gradient doesn't move the decision boundary. If the boundary still sits close to real data points, an attacker with query access, a substitute model, or a cleverer gradient estimator will still find the nearby crossing — the defense only raised the cost of one specific attack technique, not the cost of attacking the model.
Beyond evasion: the wider model-security surface
Evasion attacks assume the model is already trained and fixed; two other attack families target the pipeline around it. A poisoning attack corrupts training data so the model itself learns the wrong thing. If a fraud model is periodically retrained on transactions that human reviewers have cleared as legitimate, an attacker who can get some of their own crafted transactions labeled "not fraud" — by staying just under a review threshold repeatedly, say — is injecting poisoned examples that shift the decision boundary in their favor for future, larger transactions, without ever touching the model's weights directly. Model extraction targets a model exposed only as an API: a company offering, say, a paid document-verification or OCR service can have a competitor query it heavily enough to train a surrogate model that approximately reproduces its behavior, stealing the functional value of the model without stealing a single weight. Membership inference targets a different asset — not the model's function but its training data's privacy — by asking whether a specific record was used in training, based on the model behaving subtly more confidently on data it has seen. For a model trained on customer transaction histories, a successful membership-inference attack can leak that a specific person was a customer, which is itself a privacy breach under India's Digital Personal Data Protection Act, 2023, independent of whether any single prediction was ever wrong.
These three attack families and evasion share a common shape: all of them exploit the gap between what a model was designed to do and what its actual, literal mathematical behavior permits an adversary to extract or manipulate. Robustness to one doesn't imply robustness to another — a model hardened against PGD evasion attacks says nothing about whether its training pipeline accepts poisoned labels, and vice versa.
The robustness-accuracy tension
Adversarially trained models routinely show slightly lower accuracy on clean, unperturbed test data than the same architecture trained normally. Tsipras et al. (2018) argued this isn't a training artifact to be optimized away but close to unavoidable: some features that are genuinely predictive on clean data (fine, brittle statistical patterns a network can pick up) are exactly the features an Lp-bounded adversary can most easily erase or flip, so a model forced to be robust under the min-max objective is implicitly penalized for relying on them, and has to fall back on coarser, more perceptually-aligned features that yield a wider but less finely-tuned decision margin. Robustness and accuracy aren't the same optimization target pointed in the same direction; a security requirement, once you write it as an explicit constraint in the loss, has a real accuracy price attached, which is worth knowing before promising a fraud team "zero accuracy cost" for a robustness upgrade.
Active recall
Attempt each of these before reading the answer beneath it.
Q1. A linear fraud model has w = (2, −1), b = 0.2. A transaction x = (0.4, 0.5) has true label y = 1 (fraud). Compute p. Then compute the FGSM-perturbed point for ε = 0.2, the new score p′, and verify Δz against ε‖w‖1.
A1. z = 2(0.4) − 1(0.5) + 0.2 = 0.5, so p = σ(0.5) ≈ 0.6225 (flagged). Gradient: (p−y) = −0.3775; ∂J/∂x1 = −0.3775(2) = −0.755 (sign −1); ∂J/∂x2 = −0.3775(−1) = 0.3775 (sign +1). x1′ = 0.4−0.2 = 0.2, x2′ = 0.5+0.2 = 0.7. z′ = 2(0.2)−1(0.7)+0.2 = −0.1, p′ = σ(−0.1) ≈ 0.4750 (evades). Δz = −0.1−0.5 = −0.6; ε‖w‖1 = 0.2(2+1) = 0.6. Matching magnitudes (opposite sign) confirms the vertex-of-hypercube argument.
Q2. A model scores 99% accuracy on clean images and 99% accuracy on the same images corrupted with Gaussian pixel noise. A colleague concludes it's adversarially robust. What's wrong with that conclusion?
A2. Gaussian noise samples roughly uniformly across many directions in a high-dimensional space; only a thin, gradient-identifiable slice of that space actually crosses the decision boundary within the allowed budget (the top-left wedge in the diagram), so random sampling almost never lands there and average-case accuracy stays high. A gradient-based attack doesn't sample — it solves for that exact slice. High accuracy under random corruption says nothing about accuracy under an adversary who computes rather than guesses the perturbation direction.
Q3. Classify each scenario by threat model (white-box/black-box) and attack type (evasion/poisoning): (a) an attacker with no access to a bank's model repeatedly queries its public fraud-check API to train a substitute model, then attacks the substitute; (b) an insider with model-repo access computes the exact gradient of the deployed weights to craft one bypass transaction; (c) an attacker gets several of their own transactions mislabeled "legitimate" during a human review process that feeds the next training run.
A3. (a) black-box evasion, relying on transferability. (b) white-box evasion. (c) poisoning (label-flipping), independent of white-box/black-box access since it corrupts training data rather than attacking inference directly.
Q4. Why is PGD generally a stronger attack than single-step FGSM against a real (non-linear) deep network, and what step does PGD add after each gradient update that a single FGSM step doesn't need?
A4. FGSM is exact only when the loss surface is linear in x over the whole ε-ball, which holds for our toy logistic model but not for a deep network with curved decision boundaries. PGD recomputes the gradient at each new point and takes several small steps, better tracking the true worst-case direction as it moves. Because each step of size α could push a coordinate outside [x−ε, x+ε], PGD must project (clip) back into the ε-ball after every step, which single-step FGSM never needs since it only ever takes one full-ε step.
Q5. A team defends their vision model by adding a non-differentiable JPEG-recompression step before the network, arguing gradient-based attacks can't backpropagate through it. What's the flaw?
A5. This is gradient masking / obfuscated gradients: it blocks one specific class of white-box attack that needs an exact analytic gradient through the whole pipeline, but the underlying decision boundary is unchanged and still close to real data. Attackers can use a differentiable approximation of the blocking step (BPDA), attack a substitute model and rely on transferability, or use a black-box, query-only attack — all of which sidestep the missing gradient entirely. Athalye et al. (2018) broke most ICLR 2018 defenses this way.
Q6. Why might a model trained with the Madry et al. min-max adversarial objective show lower clean-test accuracy than the same architecture trained normally, even though both are optimizing the same training data?
A6. Some features that are genuinely predictive on unperturbed data are also the features easiest for an Lp-bounded adversary to flip; the min-max objective explicitly penalizes reliance on exactly those brittle features, pushing the model toward coarser features with a wider margin but a worse fit to the fine structure of clean data. Robustness and clean accuracy are not the same optimization target, so a real robustness gain typically comes with a measurable, non-zero accuracy cost (Tsipras et al., 2018).
Think About It
Think about this: How would you explain adversarial robustness and model security to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where adversarial robustness and model security is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting adversarial robustness and model security to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind adversarial robustness and model security, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.