Adversarial Robustness: Defending Against Attacks

In 2019, a team led by Samuel Finlayson published a paper in Science titled "Adversarial attacks on medical machine learning." The core demonstration was unsettling in its simplicity: take a deployed clinical classifier — a model reading a dermoscopy image for melanoma, a chest X-ray for pneumothorax, a retinal scan for diabetic retinopathy — and add a perturbation so small a radiologist cannot see it. The model's diagnosis flips. Finlayson and coauthors argued this is not a lab curiosity. U.S. healthcare billing runs on the diagnosis a model outputs, insurers and providers have direct financial incentives to nudge that output, and once a classifier decides whether a claim gets reimbursed, it becomes a target the same way a tax form or a customs declaration is a target. The uncomfortable question their paper raises for anyone building such a system is not "can I find a defense that survives the attacks I've tried?" It is "can I prove the model's decision cannot change inside some region around this scan, against attacks I haven't thought of yet?" Those are different questions, and the gap between them is the subject of this chapter. A companion chapter in this course covers the standard attack-and-defend toolkit — FGSM, PGD, adversarial training. Here we go one level deeper, into the machinery that turns "we tested it and it held" into a number you can write down and defend in an audit.

Empirical robustness versus certified robustness

Fix a classifier f, an input x, and a threat model: an attacker may add any perturbation δ with ‖δ‖₂ ≤ ε (an L2 ball of radius ε — one common choice among several norm balls used in practice). Two very different claims can be made about how f behaves on that ball.

Empirical robustness says: we ran attack algorithm A (FGSM, PGD, Carlini–Wagner, whatever) against f, and it failed to find a δ inside the ball that flips the prediction. This is a statement about one algorithm's failure, not about the ball. A stronger, adaptive, or simply different attacker might succeed where A did not — the ball has not been shown to be safe, only that one search over it came up empty.

Certified robustness says: for every δ with ‖δ‖₂ < R, provably, f(x+δ) = f(x). No attack algorithm, adaptive or not, gradient-based or not, can flip the prediction inside that radius, because the guarantee was proven mathematically, not tested empirically. R is a number a hospital's risk officer can put in a compliance document.

Most defenses proposed in the adversarial ML literature are empirical. Certified defenses are rarer because they are harder to build — but the rest of this chapter builds one from scratch: randomized smoothing, introduced by Jeremy Cohen, Elan Rosenfeld, and Zico Kolter at ICML 2019 ("Certified Adversarial Robustness via Randomized Smoothing"). It is the mechanism that currently gives the tightest, most practical certified L2 radii for large deep networks, and it will let us compute an actual robustness guarantee, not just claim one.

The misconception: "the defense stopped my attack, so the model is robust"

Before building the certified defense, it is worth naming precisely why empirical defenses keep failing in a specific, technical way — because the failure mode recurs constantly and a student who hasn't seen it will walk straight into it. In 2018, Anish Athalye, Nicholas Carlini, and David Wagner examined nine defenses accepted to ICLR 2018, each of which had reported near-zero attack success rate against standard gradient-based attacks. Their paper, "Obfuscated Gradients Give a False Sense of Security," showed that seven of the nine were not actually robust — they were exploiting a specific weakness of the attacker, not fixing a weakness of the model.

The trick each defense used, in one of a few variants, was to make the loss surface unfriendly to gradient descent without making the underlying decision boundary any safer. A defense might quantize inputs so the gradient is zero almost everywhere ("shattered gradients"), or inject randomness so a single gradient computation is nearly uncorrelated with the true expected gradient ("stochastic gradients"), or chain many layers so gradients vanish or explode numerically. PGD, which needs a usable gradient to climb the loss, simply stalls — not because no adversarial example exists nearby, but because the attacker's compass is broken. Athalye et al. fixed the compass. For shattered gradients they substituted a differentiable approximation of the offending operation on the backward pass while keeping the true operation on the forward pass (BPDA — backward pass differentiable approximation). For stochastic defenses they averaged the gradient over many noise draws before stepping (EOT — expectation over transformation). With a working compass restored, six of the seven defenses were broken completely, and the seventh only forced a modest reduction in perturbation budget.

The corrected version of the claim: a defense surviving a fixed attack algorithm is evidence about that algorithm, not a proof about the model. "Robust under PGD" and "robust" are not synonyms. This is precisely the gap randomized smoothing is built to close — it does not try to make gradients harder to compute, it proves a radius that holds against any attacker, because the proof never assumes anything about how the attacker searches.

Randomized smoothing: averaging away the boundary's fragility

Take any base classifier f — it can be an arbitrary neural network, non-smooth, non-robust, trained however you like. Define a new classifier g, the smoothed classifier, by corrupting the input with isotropic Gaussian noise and taking a majority vote:

g(x) = argmax_c  P[ f(x + ε) = c ],   ε ~ N(0, σ²I)

In words: to classify x, don't ask f once. Ask it what it would say on infinitely many noisy copies of x, and report whichever class wins the vote. g inherits none of f's jaggedness directly — its decision boundary is a smoothed-out (literally, Gaussian-convolved) version of f's boundary, which is why a small shift in x tends to shift the vote fractions only a little, even when f itself is violently non-smooth right at the boundary.

Cohen, Rosenfeld, and Kolter's central theorem makes "tends to shift only a little" exact. Let c_A be the class with the highest vote probability at x, with p_A = P[f(x+ε) = c_A], and let p_B be the probability of the runner-up class (or, conservatively, an upper bound on every other class's probability). Then:

g(x + δ) = c_A   for every δ with  ‖δ‖₂ < R,
where   R = (σ / 2) · ( Φ⁻¹(p_A) − Φ⁻¹(p_B) )

Φ⁻¹ is the inverse CDF (quantile function) of the standard normal distribution. The intuition behind why this particular expression appears — rather than some other function of p_A and p_B — comes from the Neyman–Pearson lemma. Shifting the center of the noise distribution from x to x+δ turns N(x, σ²I) into N(x+δ, σ²I); among all possible "decision regions" a classifier could carve out of the input space, the one that most damages the vote share of c_A under this shift — the worst case an attacker could ever engineer, even hypothetically, with an f redesigned solely to hurt this bound — is a linear half-space perpendicular to δ. The likelihood ratio between two shifted Gaussians is a function only of the perpendicular distance to that half-space, and integrating a Gaussian tail against that half-space is exactly where the Φ⁻¹ terms come from. The result is not a heuristic bound; it is provably tight against the worst base classifier consistent with the observed p_A, p_B — no adversary, however cleverly it exploits f's internals, can beat it.

Worked example 1: the radius from a closed-form probability

Suppose noise level σ = 0.5, and at some clean input the smoothed classifier's vote is overwhelmingly one-sided: p_A = 0.99, p_B = 0.01 (the two-class worst case, so they sum to 1). Look up the standard normal quantiles:

Φ⁻¹(0.99) =  2.3263
Φ⁻¹(0.01) = −2.3263          (by symmetry, Φ⁻¹(1−p) = −Φ⁻¹(p))

R = (0.5 / 2) · ( 2.3263 − (−2.3263) )
  = 0.25 · 4.6527
  = 1.1632

Whenever p_B = 1 − p_A the general formula collapses to the cleaner R = σ · Φ⁻¹(p_A), and indeed 0.5 × 2.3263 = 1.1632 — same answer, useful as a sanity check whenever the runner-up class absorbs all remaining probability mass. The certified claim: for this input, no perturbation smaller than 1.1632 (in L2 norm, in the same units as the input) can change the smoothed classifier's output, full stop, against any conceivable attack.

Worked example 2: estimating the radius from samples (the part a real system actually runs)

The formula above assumed you already know p_A exactly. In practice p_A is an integral over the base classifier's response to infinitely many noise draws — you cannot compute it in closed form, only estimate it by Monte Carlo sampling and account for the estimation error statistically. Cohen et al.'s practical procedure (they call it CERTIFY) draws n noisy copies of x, counts how many the base classifier assigns to the candidate top class, and uses a Clopper–Pearson exact confidence interval to get a statistically valid lower bound p_A_lower on the true p_A — then plugs that lower bound into R = σ · Φ⁻¹(p_A_lower), using the worst case p_B = 1 − p_A since the true runner-up probability is unknown. This guarantees the certified radius is valid with probability at least 1 − α over the randomness of sampling — the one deliberate way this proof is allowed to fail, at a rate you choose.

Below, base_classifier stands in for a trained model's hard decision on a single scalar "malignancy score" — a drastic simplification of a real CNN acting on an image tensor, kept one-dimensional purely so every step of the Monte Carlo estimate can be traced by hand. A production system would replace it with a real network's forward pass on a noised image and would draw two independent sample batches — a small one to pick the top class, a separate one to estimate its probability (why that separation matters is Active Recall Q4). This demo reuses one batch for both, which is a valid simplification only because we already know deterministically what the true top class is (x0 = 0.30 > 0, so class 1 is correct by construction) — never do this when the top class itself is uncertain.

import numpy as np
from scipy.stats import norm, binomtest

def base_classifier(perturbed_score):
    return 1 if perturbed_score > 0 else 0

def sample_predictions(x0, sigma, n, rng):
    noise = rng.normal(0.0, sigma, size=n)
    return np.array([base_classifier(x0 + e) for e in noise])

def certify(x0, sigma, n, alpha, rng):
    preds = sample_predictions(x0, sigma, n, rng)
    n_a = int(preds.sum())
    ci = binomtest(n_a, n).proportion_ci(
        confidence_level=1 - 2 * alpha, method='exact')
    p_a_lower = float(ci.low)
    if p_a_lower <= 0.5:
        return n_a, p_a_lower, None       # abstain: not confident enough
    radius = sigma * norm.ppf(p_a_lower)
    return n_a, p_a_lower, float(radius)

rng = np.random.default_rng(42)
x0, sigma, n, alpha = 0.30, 0.50, 1000, 0.001

n_a, p_a_lower, radius = certify(x0, sigma, n, alpha, rng)
print(f"n_A = {n_a} / {n}")
print(f"p_A_lower = {p_a_lower:.4f}")
print(f"certified radius R = {radius:.4f}")

Run exactly as written (NumPy 2.4, SciPy 1.18, seed 42), this prints:

n_A = 726 / 1000
p_A_lower = 0.6805
certified radius R = 0.2345

Trace the arithmetic: x0 = 0.30, σ = 0.5, so the true probability that a noisy sample lands above 0 is P(0.30 + N(0, 0.25) > 0) = P(Z > −0.6) = Φ(0.6) ≈ 0.7257. The sample count 726/1000 = 0.726 lands close to that, as expected. The Clopper–Pearson lower bound at confidence level 1 − 2(0.001) = 0.998 pulls that down further, to 0.6805, to account honestly for sampling noise — the price of a guarantee that must hold with probability ≥ 1 − α = 0.999, not just on average. Plugging in: R = 0.5 × Φ⁻¹(0.6805) = 0.5 × 0.4691 ≈ 0.2345, matching the printed value. Compare this to Worked Example 1's R = 1.1632: that example assumed a near-unanimous vote (p_A = 0.99); this one has a real, noisier vote (p_A ≈ 0.68), and the certified radius shrinks by roughly 5×. The formula rewards confidence, and confidence is expensive to earn honestly from finite samples.

What the certificate does not cover

Three limits matter enough that treating randomized smoothing as a universal fix would be its own new misconception. First, the guarantee is scoped exactly to the threat model in the theorem — an L2 ball. It says nothing about an L∞ perturbation of the same norm-radius (an L∞ ball of radius r in d dimensions has L2 radius up to r√d, so an L∞ attacker can be far outside the certified L2 ball while staying inside its own, smaller-looking budget), and it says nothing at all about a physically realized attack — an adversarial patch stuck on an X-ray light box, or a corrupted CPT billing code fed through a completely different, non-image pipeline. Finlayson et al.'s deeper point was exactly this: a hospital's attack surface is not one pixel-perturbation channel, and a certificate that closes one channel does not close the others.

Second, there is a real accuracy–robustness knob, and it is σ. A larger σ mechanically inflates R for any fixed p_A (the formula is linear in σ), but heavier noise also degrades the base classifier's raw accuracy, since it is now trying to classify a much-corrupted input — which pulls p_A down for many inputs, some of which cross below 0.5 and are forced to abstain entirely. Chasing a bigger certified radius by cranking σ is not free; it is traded directly against how often the model is willing to answer at all, and at what baseline accuracy.

Third, the certificate is expensive to obtain: it requires thousands to hundreds of thousands of noisy forward passes per input, not one, which is why it is typically run at evaluation or audit time on select inputs, not as the live inference path for every request.

Active recall

Attempt each question before reading the worked answer beneath it.

1. An engineer reports: "our defense reduced PGD's attack success rate from 87% to 2%, so the model is now robust." What is wrong with this claim, and what specific mechanism (name it) might be responsible for the apparent success?
2. Using the general formula, compute the certified radius for σ = 0.25, p_A = 0.9, p_B = 0.1.
3. Starting from Worked Example 1 (σ=0.5, p_A=0.99, R=1.1632): suppose you double σ to 1.0, and as a direct consequence the extra noise degrades the base classifier enough that p_A falls to 0.95 (worst case p_B=0.05). Trace every consequence of this change — not just the new radius.
4. In the Python certify function above, the same batch of n samples is used both to determine the sample count n_a and (implicitly) to have already fixed which class is "class A." Why is reusing one batch for both jobs statistically unsound in general, and what does a correct implementation do differently?
5. A hospital certifies its retinal-scan classifier with R = 0.24 in pixel-intensity L2 space. Does this certificate protect the hospital against (a) a sticker physically placed over part of a printed scan, and (b) a fraudulent edit to the DICOM metadata field carrying the billing code? Justify each answer from the scope of the theorem.
6. A ResNet-50 forward pass costs roughly 4 GFLOPs. Cohen et al.'s procedure draws n0 = 100 samples to select the top class and a further n = 100,000 samples to estimate its probability. How many total forward passes, and how many GFLOPs, does certifying one input cost, compared to one ordinary inference?

Worked answers

1. A drop in PGD's success rate is evidence about PGD, not about the existence of nearby adversarial examples. The likely mechanism is obfuscated gradients (Athalye, Carlini & Wagner, 2018): the defense may have shattered, randomized, or numerically destabilized the gradient PGD relies on, without actually removing adversarial examples from the input space. Seven of nine ICLR 2018 defenses reporting exactly this kind of success were broken once the attacker's gradient was repaired with techniques like BPDA and EOT. The correct test is an adaptive attack built with knowledge of the defense's mechanism, or — as this chapter argues — a certified bound that doesn't depend on any attacker's algorithm at all.

2. Φ⁻¹(0.9) = 1.2816, and since p_B = 1 − p_A, use the collapsed form R = σ·Φ⁻¹(p_A) = 0.25 × 1.2816 = 0.3204. (Check via the general form: Φ⁻¹(0.1) = −1.2816, so R = (0.25/2)(1.2816 − (−1.2816)) = 0.125 × 2.5631 = 0.3204 — matches.)

3. The direct number: Φ⁻¹(0.95) = 1.6449, and with p_B = 1 − p_A, R_new = σ·Φ⁻¹(p_A) = 1.0 × 1.6449 = 1.6449. That is larger than the original 1.1632, a 41% increase, even though the vote confidence dropped from 0.99 to 0.95 — because R scales linearly in σ while Φ⁻¹ is a slowly varying function near its argument's typical range, so the doubled σ dominates the calculation. But the full ripple goes further than this one point's radius: (a) the same noise increase that dropped this point's p_A from 0.99 to 0.95 will drop other points' p_A too, and some of those may cross below 0.5 and be forced to abstain — so the certified accuracy averaged over a whole test set can fall even while this individual point's radius grows; (b) the Monte Carlo sample count n needed for the confidence interval is a value you choose in advance as part of the protocol — it does not shrink automatically just because the true p_A moved further from 0.5, so the computational cost of certifying is unchanged; (c) before adopting σ=1.0 in production, the clean (radius-zero) accuracy of the smoothed classifier at that noise level must be re-checked — a bigger certified radius on a model that is now less accurate to begin with is not obviously a better model to deploy in a hospital.

4. If the same batch is used to both pick the top class and estimate its probability, the reported count n_a is not an unbiased draw from Binomial(n, p_A) for a class fixed in advance — it is (at least) the maximum count among several correlated candidate classes, because the class was chosen because it had the most votes in that very batch. The Clopper–Pearson interval's coverage guarantee assumes the class being measured was fixed before the data was seen; violating that (a form of selection bias / multiple comparisons) inflates the apparent p_a_lower and invalidates the stated confidence level, silently producing certified radii that are not actually valid at the claimed 1−α. The correct procedure draws n0 samples purely to pick ĉ_A (the SELECT step), discards that batch, then draws a fresh, independent batch of n samples to compute the confidence bound on P[f(x+ε)=ĉ_A] for that now-fixed class (the ESTIMATE step).

5. Neither. (a) A sticker is a large, spatially concentrated perturbation, not a small one spread across the whole image — its L2 norm can vastly exceed 0.24, and it may not even resemble Gaussian-smoothed noise in structure, so it sits far outside the region the theorem covers. (b) The theorem is a statement purely about perturbations to the pixel tensor fed to the model; it says nothing whatsoever about a separate metadata field consumed by a different part of the billing pipeline. This is exactly Finlayson et al.'s broader warning: certifying one channel does not certify the system.

6. Total forward passes: n0 + n = 100 + 100,000 = 100,100, versus 1 for an ordinary prediction — a 100,100× multiplier. In compute: 100,100 × 4 GFLOPs = 400,400 GFLOPs ≈ 400.4 TFLOPs for one certified prediction, against 4 GFLOPs for one plain inference. This is why certification is typically run offline on audited or high-stakes inputs rather than on every live request.

Think About It

Think about this: How would you explain adversarial robustness: defending against attacks to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind adversarial robustness: defending against attacks, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.