AI Security: Adversarial Attacks and Defenses

The stop sign that read "45"

In 2018, a team of researchers from the University of Michigan, University of Washington, and UC Berkeley published a result that unsettled a lot of people building self-driving systems. They took a real stop sign, printed a handful of black-and-white stickers designed by an optimization algorithm, not by hand, and stuck them onto the sign in an arrangement that a human driver would read as graffiti or grime. A standard convolutional road-sign classifier, the kind trained on the German Traffic Sign benchmark that many production sign-recognition pipelines are built on, read the same sign as a speed-limit-45 sign across the large majority of the distances and viewing angles they tested, including a drive-by test from a moving car. Eykholt et al., "Robust Physical-World Attacks on Deep Learning Visual Classification," CVPR 2018, called their method RP2 (Robust Physical Perturbations). Nothing was wrong with the camera, the sign, or the road. Something was wrong with the geometry of what the network had learned to call "stop."

This is not a curiosity confined to one 2018 paper. The same class of convolutional classifier sits behind the ADAS sign-and-obstacle recognition now shipping in Indian passenger EVs, the last-mile sidewalk delivery robots piloted in Bengaluru and Gurugram, and every image classifier deployed at scale, from crop-disease identification apps used by Indian farmers to document-fraud detectors used by banks. The attack that fooled a stop sign in Michigan generalizes to any system built the same way. Understanding exactly how it works, and why the obvious-sounding fixes keep failing, is the point of this chapter.

What an adversarial example actually is

Formally: given a classifier f, an input x that f correctly labels as y, an adversarial example is a perturbed input x' = x + δ such that the perturbation is small under some norm, ||δ||_p ≤ ε, but f(x') ≠ y (untargeted) or f(x') = y_target for some attacker-chosen wrong label (targeted). For images the norm is almost always L∞, which caps the maximum change to any single pixel rather than the total change across all pixels, because a human eye is far more sensitive to one pixel changing by a lot than to every pixel changing by a little.

Why do these perturbations exist at all, and why can they be so small? Goodfellow, Shlens, and Szegedy ("Explaining and Harnessing Adversarial Examples," ICLR 2015) gave the explanation that still holds up: deep networks are globally nonlinear but locally, in the neighborhood of a given input, they behave close to linearly. Consider the dot product between a weight vector w and a perturbation δ bounded per-coordinate by ε. Each individual term w_i · δ_i is tiny, but the sum w · δ grows with the number of input dimensions n, because you are summing n small terms that all point the same way. A 224×224 RGB image has over 150,000 dimensions. An imperceptible per-pixel nudge, aligned correctly across all of them, adds up to a large shift in the model's output. This is not a training bug that better data fixes; it is close to a structural property of any high-dimensional, locally-linear decision function. Their famous demonstration: a GoogLeNet classifier that labeled an image "panda" with 57.7% confidence relabeled the same image, after a perturbation invisible to a human eye, "gibbon" with 99.3% confidence.

Attacks are usually grouped by how much the attacker can see. A white-box attacker has the model's weights and can compute exact gradients (FGSM, PGD, C&W, all covered below). A black-box attacker only gets to query the model's inputs and outputs, the situation any attacker faces against a deployed API, and must either estimate gradients from queries or exploit transferability (covered later in this chapter). A physical-world attacker, like Eykholt's team, needs a perturbation that survives being printed, lit differently, photographed from an angle, and re-digitized by a camera pipeline before it ever reaches the model, which is a much harder constraint than perturbing a digital image file directly.

FGSM: the fastest way to break a classifier

The Fast Gradient Sign Method, introduced in the same Goodfellow et al. 2015 paper, is the simplest attack and the one every later attack is compared against. The idea: instead of searching for the worst point inside the ε-ball, take one step in the direction that increases the loss fastest, linearized at x.

Given loss J(θ, x, y), the gradient ∇_x J tells you, to first order, how a small change in each input coordinate changes the loss. If your budget is an L∞ ball of radius ε, the perturbation that maximizes a linear function c · δ subject to ||δ||∞ ≤ ε is not a small step in the gradient's exact direction, it is a step to a corner of the hypercube: δ_i = ε · sign(c_i). This is a basic fact from linear programming (the dual pairing of L1 and L∞ norms), and it is why FGSM uses the sign of the gradient rather than the gradient itself:

x_adv = x + ε · sign(∇_x J(θ, x, y))

clipped back to the valid input range afterward. It costs exactly one forward pass and one backward pass, which is why it was originally proposed as a fast way to generate adversarial training data, not as the strongest possible attack.

Worked example: flipping a stop-sign detector

A full CNN's gradient can't be hand-traced, so this example collapses the classifier's final decision layer, the part that actually decides "stop sign" vs "not stop sign" from learned high-level features, into a single logistic-regression neuron over four abstracted features. The mechanism is identical to what happens inside a real network's last layer; only the earlier convolutional layers are being skipped for tractability.

Weights w = [1.5, -0.5, 2.0, -1.0], bias b = 0.2, input x = [0.6, 0.2, 0.5, 0.1], true label y = 1 (this is a stop sign). The logit is z = w·x + b:

z = 1.5(0.6) + (-0.5)(0.2) + 2.0(0.5) + (-1.0)(0.1) + 0.2
  = 0.90 - 0.10 + 1.00 - 0.10 + 0.20 = 1.90

p = sigmoid(1.90) ≈ 0.8699   →  confidently correct: "stop sign," 87%

For binary cross-entropy loss with sigmoid output, the gradient of the loss with respect to the logit is exactly (p - y), and by the chain rule the gradient with respect to the input is ∇_x J = (p - y) · w:

p - y = 0.8699 - 1 = -0.1301
∇_x J = -0.1301 · [1.5, -0.5, 2.0, -1.0]
       = [-0.1952, 0.0651, -0.2602, 0.1301]
sign(∇_x J) = [-1, +1, -1, +1]

Notice the sign pattern is exactly the negative of sign(w). That is not a coincidence here: whenever p < y (the model is already less confident than the true label demands, which holds for any correctly-but-not-perfectly classified point), the scalar factor (p-y) is negative, so sign((p-y)·w) = -sign(w) for every coordinate. This gives a shortcut worth keeping: x_adv = x - ε·sign(w), and the resulting change in the logit is Δz = -ε · Σ|w_i|, independent of the exact value of p. Applying it with budget ε = 0.5:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w = np.array([1.5, -0.5, 2.0, -1.0])
b = 0.2
x = np.array([0.6, 0.2, 0.5, 0.1])
y = 1

z = np.dot(w, x) + b
p = sigmoid(z)
print(f"clean: z={z:.2f}, p(stop)={p:.2f}")

grad_x = (p - y) * w
epsilon = 0.5
x_adv = np.clip(x + epsilon * np.sign(grad_x), 0.0, 1.0)
print("x_adv:", [f"{v:.2f}" for v in x_adv])

z_adv = np.dot(w, x_adv) + b
p_adv = sigmoid(z_adv)
print(f"adversarial: z={z_adv:.2f}, p(stop)={p_adv:.2f}")

Tracing it: x_adv = [0.60-0.50, 0.20+0.50, 0.50-0.50, 0.10+0.50] = [0.10, 0.70, 0.00, 0.60], all valid after clipping. The new logit: z_adv = 1.5(0.10) - 0.5(0.70) + 2.0(0.00) - 1.0(0.60) + 0.2 = 0.15 - 0.35 + 0 - 0.60 + 0.20 = -0.60, matching 1.90 - 0.5×5.0 = -0.60 from the shortcut formula (Σ|w_i| = 1.5+0.5+2.0+1.0 = 5.0). The printed output is:

clean: z=1.90, p(stop)=0.87
x_adv: ['0.10', '0.70', '0.00', '0.60']
adversarial: z=-0.60, p(stop)=0.35

A perturbation of at most 0.5 in each of four abstracted features flips the prediction from "87% stop sign" to "35% stop sign," a majority vote for the wrong class, using nothing but the sign of one gradient computation. The general result worth keeping: for a model that is locally linear near x, the minimum ε needed to push the logit to the decision boundary (z=0) is ε_min = |z| / Σ|w_i|, here 1.90 / 5.0 = 0.38. Below that budget the attack weakens the prediction but doesn't flip it; above it, the model is fooled.

The diagram shows the actual mechanism, not a metaphor for it: x sits confidently inside the stop-sign region, near enough to the boundary that its entire ε-box (the set of every image FGSM is allowed to consider) reaches across the line. FGSM doesn't search that box; it computes one gradient, takes its sign, and jumps straight to the one corner of the box that lands furthest across the boundary, which is exactly x_adv in the earlier trace.

The misconception: "robust to noise" means "robust to attack"

Students who have seen data augmentation assume that a model trained with random noise, blur, or JPEG artifacts is also resistant to adversarial examples, since both look like "the image got a bit messed up." They are not the same threat. Random noise picks a perturbation direction uniformly at random from an astronomically large space of directions; in a 150,000-dimensional pixel space, the overwhelming majority of random directions do not align with the gradient of the loss, so a model can be correct on x + random_noise with 99.9% accuracy while an attacker who computes sign(∇_x J) finds the one deliberate direction, out of that same enormous space, that crosses the decision boundary fastest. Robustness to random corruption and robustness to worst-case, gradient-aligned corruption are different properties, measured by different tests, and a model can score well on one while failing the other almost completely. This is precisely why the FGSM formula uses a gradient rather than noise: it replaces a random search with an exact, first-order-optimal direction-finder.

Beyond one step: PGD, C&W, and the physical world

FGSM's linear approximation is only exact at x itself; a few pixels away, the true loss surface curves. Madry, Makelov, Schmidt, Tsipras, and Vladu ("Towards Deep Learning Models Resistant to Adversarial Attacks," ICLR 2018) framed the strongest possible attack as Projected Gradient Descent (PGD): repeat the FGSM step many times with a small step size α, projecting back onto the ε-ball after each step, so the search actually explores the interior of the ball instead of jumping to one corner:

import torch

def pgd_attack(model, loss_fn, x, y, epsilon, alpha, steps):
    # model, loss_fn: assumed helpers, not shown (a trained classifier
    # and its training loss, e.g. torch.nn.CrossEntropyLoss())
    x_adv = x.clone().detach()
    for _ in range(steps):
        x_adv.requires_grad_(True)
        loss = loss_fn(model(x_adv), y)
        grad = torch.autograd.grad(loss, x_adv)[0]
        x_adv = x_adv + alpha * grad.sign()
        x_adv = torch.max(torch.min(x_adv, x + epsilon), x - epsilon)
        x_adv = torch.clamp(x_adv, 0, 1).detach()
    return x_adv

Madry et al. also reframed defense itself as a min-max optimization problem: find parameters θ that minimize the loss even under the worst-case perturbation inside the ball, min_θ E[max_{δ∈S} J(θ, x+δ, y)]. PGD approximates the inner maximization; training on PGD-generated examples approximates a solution to the outer minimization. This is adversarial training, still the most reliable empirical defense over a decade of follow-up work.

Carlini and Wagner ("Towards Evaluating the Robustness of Neural Networks," IEEE S&P 2017) went further, formulating the attack as a direct optimization over the network's raw logits rather than the softmax-cross-entropy loss FGSM and PGD use, with a margin-based objective that keeps searching until it finds a genuinely minimal perturbation rather than stopping at the first successful one. The C&W attack routinely finds smaller, harder-to-detect perturbations than FGSM or PGD, and, critically, it exposed a defense that everyone had believed was working.

Outside the digital image file, Eykholt et al.'s RP2 algorithm had to solve a harder problem than any of these: a perturbation optimized for one exact pixel array will usually fail once it is printed, lit differently, photographed at an angle, and re-encoded by a camera's image pipeline. RP2 optimizes the perturbation against an expectation over a whole distribution of simulated distances, angles, and lighting conditions, so the stickers keep working across the range of views a real camera on a real car would actually see, not just the one photograph the researchers happened to optimize against.

Defenses, and why most of them are illusions

Papernot, McDaniel, Wu, Jha, and Swami ("Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks," IEEE S&P 2016) proposed defensive distillation: train a second network on temperature-softened soft labels produced by a first network, which flattens the softmax near training points and shrinks the gradients FGSM and PGD rely on. It looked like a real defense, because those specific attacks stopped working. Carlini and Wagner's 2017 paper broke it: because their attack optimizes on the logits directly rather than the flattened softmax, it found adversarial examples against the distilled network at almost the same success rate and perturbation size as against the undistilled one. Distillation had masked the gradient an attacker would naively use; it had not removed the underlying vulnerability at all.

Athalye, Carlini, and Wagner ("Obfuscated Gradients Give a False Sense of Security," ICML 2018) turned that single case into a general warning. They audited nine defenses accepted at ICLR 2018 that had reported strong robustness, and found seven of them relied on some form of gradient obfuscation, shattered gradients from non-differentiable preprocessing, randomized gradients from stochastic transformations, or vanishing gradients from very deep defensive pipelines, rather than any real change to the model's decision surface. Using targeted techniques to see through each obfuscation (approximating a non-differentiable step with a differentiable stand-in, averaging over randomness, rescaling vanishing gradients), they broke six of the seven completely, in most cases down to near-zero robust accuracy, and the seventh only partially. The methodological lesson generalizes past any one defense: a claimed defense is credible only if it is tested against an adaptive attacker who knows the defense mechanism and designs specifically around it, not merely against off-the-shelf FGSM or PGD run with default settings.

Cohen, Rosenfeld, and Kolter ("Certified Adversarial Robustness via Randomized Smoothing," ICML 2019) took a different route entirely: instead of testing empirically against known attacks, they built a classifier g(x) defined as the class a base classifier f most often predicts across many copies of x corrupted with Gaussian noise. Using the Neyman-Pearson lemma on the noised output distributions, they derive a certified radius around each input, proportional to the noise scale σ and to how large a probability margin the winning class holds under that noise, inside which g's prediction is provably constant. No attack of any kind, known or not yet invented, can change the smoothed prediction within that certified radius, which is a fundamentally stronger guarantee than "we tested it against the attacks we currently know." The tradeoff: the certified radius shrinks as input dimensionality grows and as σ is tuned down, and the guarantee itself is probabilistic, holding with high confidence from Monte Carlo sampling rather than exactly.

Beyond evasion: the rest of the threat surface

Everything above is an evasion attack: the model is already trained and frozen, and the attacker manipulates an input at inference time. Two other categories round out AI security and are worth naming even without a full treatment here. A poisoning attack corrupts the training data or process itself, for instance planting a small number of mislabeled or trigger-tagged examples into a dataset scraped from the open web, so the model behaves normally everywhere except when a specific trigger pattern appears, at which point it does exactly what the attacker wants; this matters for any pipeline that retrains on user-submitted or crawled data. A model extraction attack repeatedly queries a deployed model's API to train a local substitute that approximately replicates it, which both steals the intellectual property embedded in the weights and, as covered above, hands the attacker a white-box surrogate to craft transfer attacks against the real thing.

Active recall

Attempt each question before reading its answer.

1. Using the worked example's clean input, weights, and bias, if the attack budget is reduced from ε = 0.5 to ε = 0.2, does FGSM still flip the classification? Show the new x_adv, z_adv, and p_adv.
2. Suppose the model is retrained and only the fourth weight changes, from -1.0 to -3.0, giving w' = [1.5, -0.5, 2.0, -3.0]. With x, b, y, and ε = 0.5 unchanged: (a) recompute the clean logit and confidence, (b) state whether the FGSM perturbation direction changes, (c) recompute the adversarial confidence, and (d) recompute the minimum ε needed to flip the prediction. Is the retrained model more or less vulnerable?
3. Why does adversarial training using only FGSM-generated examples often fail to produce real robustness, even though it defeats FGSM itself?
4. An attacker has no access to a deployed fraud-detection model's weights, only its public prediction API. How can they still craft a working adversarial example against it?
5. Defensive distillation stopped FGSM and PGD from working. Why did Carlini and Wagner's attack still succeed against it, and what general lesson does that establish?
6. How does randomized smoothing's guarantee differ in kind, not just in degree, from adversarial training's guarantee?

Answers

1. x_adv = x + 0.2·sign(∇_x J) = [0.6-0.2, 0.2+0.2, 0.5-0.2, 0.1+0.2] = [0.40, 0.40, 0.30, 0.30]. z_adv = 1.5(0.4) - 0.5(0.4) + 2.0(0.3) - 1.0(0.3) + 0.2 = 0.60-0.20+0.60-0.30+0.20 = 0.90, matching the shortcut 1.90 - 5.0×0.2 = 0.90. p_adv = sigmoid(0.90) ≈ 0.71. The prediction is weakened (confidence drops from 0.87 to 0.71) but not flipped, since 0.2 < ε_min = 0.38.

2a. z' = 1.5(0.6) - 0.5(0.2) + 2.0(0.5) - 3.0(0.1) + 0.2 = 0.90-0.10+1.00-0.30+0.20 = 1.70, p' = sigmoid(1.70) ≈ 0.85. 2b. No: sign(w') is unchanged from sign(w) in every coordinate (the fourth weight is still negative, just larger in magnitude), and since p'-y is still negative, the FGSM direction stays sign(∇_x J) = [-1,+1,-1,+1], so x_adv is the identical [0.10, 0.70, 0.00, 0.60]. 2c. Σ|w'_i| = 1.5+0.5+2.0+3.0 = 7.0, so z'_adv = 1.70 - 0.5×7.0 = -1.80, p'_adv = sigmoid(-1.80) ≈ 0.14, a stronger flip than the original model's 0.35. 2d. ε'_min = 1.70/7.0 ≈ 0.243, down from 0.38. Counterintuitively, the retrained model is more vulnerable, not less: even though the clean confidence barely changed (0.87 to 0.85), increasing one weight's magnitude increased Σ|w_i|, which is the denominator in ε_min = |z|/Σ|w_i|. This is a concrete instance of why a large weight norm and adversarial fragility are linked, and why Lipschitz-constrained or weight-regularized architectures are studied as robustness interventions.

3. FGSM is a single-step linear approximation. A network trained against only that one direction can learn to distort its loss surface specifically around the FGSM step (sometimes overfitting so sharply that the true label leaks back through the perturbation, a failure mode called label leaking), which defeats FGSM without removing the actual worst-case point inside the ε-ball. PGD, by taking many small steps and re-projecting, explores the ball far more thoroughly and finds that worst-case point, which is why Madry et al. (2018) require PGD-based adversarial training, not FGSM-based, for robustness that survives a real attacker.

4. Query the API with chosen inputs, record its outputs, and use those input-output pairs to train a local substitute model that approximates the target's decision boundary (Papernot et al., 2017). Adversarial examples exploit properties of decision boundaries shared across models trained on similar tasks, not idiosyncrasies of one specific set of weights, so examples crafted with full white-box access to the substitute (via FGSM or PGD) transfer to the real target at a high rate, without the attacker ever seeing its gradients.

5. Distillation flattens the softmax output near training points, shrinking the gradients that FGSM and PGD compute from the softmax-cross-entropy loss, so those two specific attacks fail. Carlini and Wagner's attack instead optimizes an objective defined directly on the logits, bypassing the flattened softmax entirely, and finds adversarial examples at nearly the same success rate as against an undistilled network. The general lesson: a defense that only makes gradients smaller or less useful (gradient masking) has not made the model more robust, it has only broken the specific attacks that happened to rely on that gradient, which is why any claimed defense needs testing against an adaptive attacker before it can be trusted.

6. Adversarial training is an empirical guarantee: the model is optimized against a specific attack (PGD) and tends to hold up against attacks of similar strength, but nothing rules out some stronger future attack breaking it, and history (defensive distillation, the seven obfuscated-gradient defenses) is full of empirical defenses that later fell. Randomized smoothing's certified radius is a mathematical guarantee derived from the Neyman-Pearson lemma: within that radius, no perturbation of any kind, by any algorithm known or not yet invented, can change the smoothed prediction. It trades a smaller guaranteed region and a clean-accuracy cost for a claim that does not depend on which attacks happen to have been tried.

Think About It

Think about this: How would you explain ai security: adversarial attacks and defenses to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where ai security: adversarial attacks and defenses is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting ai security: adversarial attacks and defenses to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind ai security: adversarial attacks and defenses, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.