Federated Learning: Privacy-Preserving AI

A model that never sees a scan, and a gradient that gives one away

Picture five hospitals under an ABDM-style federated diagnostic program, training a shared model to flag anomalies in chest CT scans. No hospital uploads a single image. Each round, the coordinating server broadcasts the current model weights; each hospital computes a gradient from its own patients and sends that gradient back; the server averages the five gradients and updates the shared model. This is the textbook pitch for federated learning: the data never leaves the hospital, so patient privacy is preserved by construction.

Two properties of this exact setup should worry you. First, five hospitals is a small cohort. Cross-silo federations like this one commonly run five to fifty participants, not the hundreds of thousands of phones in a cross-device deployment like a keyboard predictor. Second, because each hospital's local dataset is small and the hospitals want the shared model to stay tightly synchronized across very different scanner hardware and patient populations, it is common in cross-silo FL to configure exactly one local gradient step per round (FedSGD) rather than several epochs of local training (FedAvg) before sending an update. That combination, few participants and a single clean gradient step per client per round, turns out to be exactly the condition under which "the data never left the hospital" stops being reassuring. A gradient computed from one training example carries a deterministic, invertible fingerprint of that example. This chapter derives that fingerprint by hand, shows the general attack it is a special case of, and builds the specific client-side defense that closes it.

Why a gradient is not "just a summary"

Take the simplest possible model a hospital could be training a piece of: a single linear neuron, y_hat = w·x + b, trained with squared-error loss L = 0.5·(y_hat − y)². For one training example (x*, y*), backpropagation gives two numbers:

error = y_hat - y*
dL/dw = error * x*
dL/db = error

Look at what these two numbers jointly encode. dL/db is the raw error term. dL/dw is that same error term scaled by x*. If you know w and b (and in federated learning the server always knows w and b, because the server is the one who broadcast them this round), then you have two equations in two unknowns, x* and y*, and you can solve them exactly:

x* = (dL/dw) / (dL/db)
y* = w * x* + b - (dL/db)

This is not a hypothetical. It is the algebraic skeleton of an attack called Deep Leakage from Gradients (Zhu, Liu, and Han, NeurIPS 2019), which showed that an attacker holding only a client's gradient and the shared model weights can reconstruct that client's training input, and for image classifiers, reconstruct it pixel-for-pixel with high fidelity. The single-neuron case above is the exact solvable core of that idea, stripped down so you can trace it with a pencil.

Worked example: reconstructing a hospital's data point from two numbers

Suppose the server has just broadcast w = 0.4, b = 0.1. One hospital computes its local gradient on a single private example, x* = 3, y* = 2 (think of x* as a normalized feature extracted from a scan, y* as the true diagnostic score). Trace the forward and backward pass:

w, b = 0.4, 0.1              # global model broadcast by the server
x_star, y_star = 3.0, 2.0    # PRIVATE: one hospital's local example

y_hat = w * x_star + b
error = y_hat - y_star       # dL/d(y_hat) for L = 0.5*(y_hat - y)^2
g_w = error * x_star         # dL/dw
g_b = error                  # dL/db

print(round(y_hat, 4), round(g_w, 4), round(g_b, 4))
# 1.3 -2.1 -0.7

The hospital sends the pair (g_w, g_b) = (−2.1, −0.7) to the server, believing it has shared only "a direction to nudge the model," not a patient record. Now play the attacker. The attacker knows w and b (it broadcast them) and has just received (g_w, g_b). Nothing else is needed:

# --- attacker, seeing only (g_w, g_b) and the public (w, b) ---
x_recovered = g_w / g_b
y_recovered = w * x_recovered + b - g_b
print(round(x_recovered, 4), round(y_recovered, 4))
# 3.0 2.0

x_recovered = −2.1 / −0.7 = 3.0. y_recovered = 0.4·3.0 + 0.1 − (−0.7) = 1.2 + 0.1 + 0.7 = 2.0. Both exact. From two floating-point numbers that look like nothing more than a training signal, the attacker has recovered the hospital's private data point exactly, with no guessing, no optimization, no iteration. That is the whole point of this example: for the simplest possible model, gradient leakage is not probabilistic or approximate. It is algebra.

From one neuron to a real network: why DLG needs optimization instead of algebra

A deep convolutional network has millions of parameters and no closed-form inverse like the one above. Zhu, Liu, and Han's actual algorithm replaces algebra with gradient descent on the input itself. The attacker initializes a dummy input x′ and dummy label y′ from random noise, runs them through the (known) shared model to get a dummy gradient ∇W′, and minimizes the distance between the dummy gradient and the real gradient it received:

D = || grad_W'  -  grad_W_real ||^2

Because the model itself is differentiable with respect to its input, D is differentiable with respect to x′ and y′, so the attacker can take gradient steps on x′ and y′ (the original paper uses L-BFGS) exactly the way training normally takes gradient steps on weights. Over a few hundred iterations, x′ converges toward the real x*, well enough that on CIFAR-10 and ImageNet-scale classifiers the paper reconstructs recognizable training images from nothing but the leaked gradient. The single-neuron closed form you just solved by hand is the batch-size-one, single-parameter special case of this same distance-minimization idea, where the "optimizer" happens to have an exact algebraic solution instead of needing iterations.

A follow-up paper, Geiping, Bauermeister, Dröge, and Moeller's Inverting Gradients: How Easy Is It to Break Privacy in Federated Learning? (NeurIPS 2020), sharpened this further: instead of matching the gradient in Euclidean distance, it matches gradient direction using cosine similarity, and shows the attack still reconstructs high-resolution images even when the client trains on a full batch of many examples for several local steps, precisely the "just do more local computation before sharing" defense you might guess would help. It does help, it raises the attacker's compute cost substantially, but it is not a hard guarantee the way the numbers in the next section are.

Why secure aggregation does not close this on its own

The most direct-sounding fix is secure aggregation: cryptographically ensure the server only ever sees the sum of all clients' gradients in a round, never any individual client's contribution (Bonawitz et al., Practical Secure Aggregation for Privacy-Preserving Machine Learning, CCS 2017). If the server literally cannot isolate one hospital's gradient, the algebra above has nothing to operate on.

The catch is in the protocol's own fine print. Bonawitz et al.'s scheme is built around a dropout-tolerance threshold t out of n clients: it only decrypts and returns the aggregate if at least t clients' shares survive the round. This threshold exists because real clients drop out mid-round (a phone loses signal, a hospital's connection times out), and the protocol needs to recover cleanly rather than stall. But run the arithmetic for a five-hospital cross-silo federation with t = 4: if only that hospital and one other complete the round, the "aggregate" is the sum of two updates, not thousands, and a curious server that also controls (or colludes with) the second hospital can subtract out the known contribution and read the first hospital's raw gradient exactly. In the cross-device setting the scheme was designed for, n is large enough that this is astronomically unlikely; in cross-silo FL with a handful of participants, it is a realistic operating condition, not an edge case. Secure aggregation is a genuinely strong defense against a passive, honest-but-curious server watching many clients at once; it is not, on its own, a defense against small cohorts, colluding participants, or a server that can force isolation by design.

The misconception this attack corrects

The belief worth naming directly: "we're only sharing gradients, not the data, so it's private." That sentence conflates two different things: the format of what is transmitted (a vector of numbers, not a JPEG or a database row) and the information content of what is transmitted. The worked example above shows the information content of a raw per-example gradient can be exactly equal to the information content of the data point that produced it, for the trivial case provably so, for deep networks empirically so at high fidelity. "Not the raw data" describes the wire format. It says nothing about what a receiver holding the model's own weights can algebraically or numerically invert out of that format. Privacy-preserving federated learning requires a specific, quantified defense on the gradient itself, not an assumption that gradients are safe because they don't look like the data.

The defense: bound what a single gradient can leak, before it is sent

The defense that closes the single-neuron attack above is the same mechanism behind Abadi et al.'s DP-SGD (Deep Learning with Differential Privacy, CCS 2016), applied at the point where the client computes its update rather than at the point where the server later releases an aggregate. Two steps, both performed locally, before the gradient ever leaves the hospital's machine:

Clip. Rescale the gradient vector so its L2 norm never exceeds a fixed bound C. This caps how much any single example can move the model, and it caps how much of that example's information the gradient's magnitude can encode.

Add noise. Add Gaussian noise calibrated to C and a target privacy budget, so the exact real-valued gradient the model saw is never transmitted, only a noised version of it.

Apply both to the same hospital's gradient from before:

import math

C = 1.0  # per-example clip norm (client-side, before transmission)
grad = (-2.1, -0.7)
norm = math.sqrt(grad[0]**2 + grad[1]**2)          # 2.2136
scale = min(1.0, C / norm)                          # 0.4518
clipped = (grad[0]*scale, grad[1]*scale)             # (-0.9487, -0.3162)

noise = (0.05, -0.03)        # illustrative fixed draw from N(0, sigma^2)
sent = (clipped[0]+noise[0], clipped[1]+noise[1])    # (-0.8987, -0.3462)

x_attack = sent[0] / sent[1]
y_attack = 0.4*x_attack + 0.1 - sent[1]
print(round(x_attack, 4), round(y_attack, 4))
# 2.5956 1.4845 -- true values were 3.0 and 2.0

The attacker runs the identical inversion algebra it ran before, on the same known w and b, and now recovers x′ = 2.60 against a true x* = 3.00 (a 13% error), and y′ = 1.48 against a true y* = 2.00 (a 26% error). The exact reconstruction is gone. The attacker gets a number in the right neighborhood, not the hospital's actual data point, and as the noise scale (σ) is increased the reconstruction degrades further, at the cost of a noisier, slower-converging training signal for the honest model too. That trade-off, between C and σ on one side and model accuracy on the other, is the same statistical tension a privacy budget always carries; the point of doing it here, at the client, on the raw per-example gradient, is that it protects the update at the one moment secure aggregation's threshold guarantee can fail: before it is ever pooled with anyone else's.

Where this generalizes, and where it gets harder

The closed-form inversion above works because a single linear neuron trained on one example gives exactly two equations and two unknowns. Real deep leakage attacks on convolutional networks do not have a closed form; they rely on the gradient-matching optimization described earlier, and their success degrades as batch size and model depth increase, because a larger batch's gradient is an average over many examples' individual gradients, which is a strictly harder system to invert (more unknowns than equations, as you'll see in the recall questions below) and a deeper network's gradient with respect to an early layer is many compositions removed from the raw input. This is why cross-device FedAvg, with many local epochs over large local batches, is empirically harder to attack this way than cross-silo FedSGD with single-example or small-batch updates. "Harder" is doing real work in that sentence, not "impossible": Geiping et al.'s cosine-similarity attack was built specifically to claw back reconstruction quality against exactly this large-batch, multi-step defense, and succeeded well enough to reconstruct recognizable 224×224 images. Batch size and local-step count are a genuine friction, not a proof.

Active recall

Attempt each question before reading its answer.

Q1. A hospital sets w = 0.4, b = 0.1 as before but its private example is now x* = 5, y* = 1. Compute the gradient it would send, and verify the attacker's closed-form inversion still recovers (5, 1) exactly.

Q2. Suppose the hospital's local model has already converged so well on this particular example that y_hat = y* exactly. What gradient does it send, and what happens when the attacker tries to run x* = g_w / g_b?

Q3. A colleague argues: "If we set w = 0, the model's prediction y_hat = b no longer depends on x at all, so the attacker can no longer recover x* from the gradient." Is this correct? Derive g_w when w = 0 and check.

Q4. The hospital switches from a batch size of 1 to a batch size of 2, averaging the gradient over two private examples (x1*, y1*) and (x2*, y2*) before sending it. Does the same two-line closed-form inversion from the worked example still recover both examples exactly? Why or why not?

Q5. In the clip-and-noise defense worked example, what happens to the attacker's reconstruction error if the clip norm C is loosened from 1.0 to 5.0, with the same fixed noise vector (0.05, −0.03)? Reason about it before recomputing.

Worked answers

A1. y_hat = 0.4·5 + 0.1 = 2.1. error = 2.1 − 1 = 1.1. g_w = 1.1·5 = 5.5, g_b = 1.1. Inversion: x_recovered = 5.5/1.1 = 5.0. y_recovered = 0.4·5.0 + 0.1 − 1.1 = 2.0 + 0.1 − 1.1 = 1.0. Exact, confirming the closed form is not an artifact of the first example's specific numbers, it holds for any (x*, y*) as long as g_b ≠ 0.

A2. If y_hat = y* exactly, error = 0, so g_b = 0 and g_w = 0·x* = 0. The gradient sent is (0, 0), the model has nothing left to learn from this example this round, and the attacker's formula x* = g_w/g_b becomes 0/0, . This is a genuine limit of the attack, not a defense anyone engineers on purpose: a perfectly fit example simply carries no gradient signal, private or otherwise, in that round.

A3. Incorrect, and this is the ripple-effect question. Setting w = 0 changes the forward pass (y_hat = 0·x* + b = b, independent of x*) but not the gradient with respect to w, because dL/dw = error · x* by the chain rule (dy_hat/dw = x* regardless of w's current value). So g_w = (b − y*)·x* still. Concretely, with w = 0, b = 0.1, x* = 3, y* = 2: y_hat = 0.1, error = 0.1 − 2 = −1.9, g_w = −1.9·3 = −5.7, g_b = −1.9. x_recovered = −5.7 / −1.9 = 3.0, exactly right. The gradient of a parameter can encode information about an input that the forward pass, at that instant, does not use, because backpropagation differentiates through how the output would have changed with respect to that parameter, not just what the output currently is. This is the misconception worth sitting with: zeroing a weight does not zero the information its gradient carries.

A4. No. With two examples averaged, g_w = (e1·x1* + e2·x2*)/2 and g_b = (e1 + e2)/2, where e1 = w·x1* + b − y1* and e2 = w·x2* + b − y2*. That is two known numbers (g_w, g_b) constraining four unknowns (x1*, y1*, x2*, y2*), an underdetermined system with infinitely many algebraic solutions. The closed form breaks down exactly at batch size 2. This is precisely why the real Deep Leakage from Gradients algorithm abandons algebra for gradient-descent optimization on the dummy inputs: optimization can still find a good, if not always unique or exact, solution to an underdetermined or nonlinear system by searching, where direct algebra cannot. It is also the formal reason larger batches are a real, if not absolute, friction against this attack.

A5. Loosening C from 1.0 to 5.0 means the clip step barely touches the gradient at all, since the original gradient's norm (2.2136) is already well under 5.0, so scale = min(1.0, 5.0/2.2136) = 1.0 and clipped = (−2.1, −0.7) unchanged. Only the fixed noise (0.05, −0.03) is added: sent = (−2.05, −0.73). Recomputing, x_attack = −2.05/−0.73 ≈ 2.808, y_attack = 0.4·2.808 + 0.1 − (−0.73) ≈ 1.953. Compare to the C = 1.0 case (x′ = 2.60, y′ = 1.48): loosening C moves the reconstruction much closer to the true (3, 2), because a looser clip leaves more of the original signal-bearing magnitude intact for the same fixed noise to hide behind. The clip bound and the noise scale are not independent knobs; the clip bound sets the ceiling the noise then has to be large relative to, and a defense that loosens one without compensating the other is weaker than it looks.

Think About It

Think about this: How would you explain federated learning: privacy-preserving ai to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where federated learning: privacy-preserving ai is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting federated learning: privacy-preserving ai to at least 3 other topics you have studied.