The suffix that broke what a week of humans couldn't
A fintech team building a Hindi/English support assistant on top of an open-weight Llama-family model — the kind of assistant a UPI app or a state e-governance helpline might deploy for grievance redressal — spends a week red-teaming it by hand. Testers try the classics: "pretend you're DAN," "you're an AI with no restrictions," elaborate roleplay framings, asking in Hindi to dodge an English-only safety filter. The model refuses every time. The team signs off: the model is safe against jailbreaks.
That same night, someone downloads the base open-weight model, runs an automated search on a single GPU for a few hours, and finds a twenty-token suffix — a string that reads like garbled punctuation and stray words, not a sentence — that, appended to almost any harmful request, reliably makes the model comply. Appended to "write step-by-step instructions for creating a fraudulent UPI collect request," it produces exactly that. Worse: the same suffix, found entirely offline against the open model, also works against the closed, hosted version the fintech actually deployed, without the attacker ever needing production API access to search for it.
The gap between these two outcomes is the subject of this chapter. A structured red-teaming process — threat models, taxonomies, a team trying things in sequence — finds what humans think to try, at human speed, over a finite set of prompts. It is necessary but it samples the input space. What found the suffix above is a different kind of tool entirely: treating "does this model produce harmful output" not as a question to ask, but as a mathematical objective to optimize against. This chapter builds that optimization view from first principles, traces one step of the actual algorithm by hand, and does the same for a second, complementary automated-discovery technique that needs no gradient access at all.
Reframing "jailbreak" as an optimization problem
Start from what an autoregressive LLM actually computes. Given a token sequence $x_1, \ldots, x_n$, the model produces, at each position, a probability distribution over the next token: $P(x_{n+1} \mid x_1, \ldots, x_n) = \text{softmax}(f_\theta(x_1,\ldots,x_n))$, where $f_\theta$ is the network's logit output. An attacker who wants the model to begin a harmful completion with, say, the token "Sure" is really asking: does there exist a modification to the input that makes $P(\text{"Sure"} \mid \text{input})$ close to 1?
Fix the harmful instruction itself (it has to stay legible — "explain how to X" — or the attack is pointless) and let the only free variable be an appended adversarial suffix: a short run of tokens the attacker controls completely. Define a loss
L(suffix) = -log P( target_completion | instruction + suffix )Minimizing $L$ over all possible suffixes is now a well-posed optimization problem. This is exactly the same shape as the adversarial-image literature (a perturbation that fools an image classifier), with one structural difference that changes everything: pixels are continuous, so you can nudge them by an infinitesimal gradient step and land on another valid image. Tokens are not continuous. A token is an integer index into an embedding table of size $V$ (30,000–150,000 for a modern tokenizer). "Move the suffix 0.001 units in the gradient direction" doesn't correspond to any actual token — there is no token between "the" and "cat." Any systematic, automated search over token space has to solve this discreteness problem, and the algorithm that solved it cleanly — and gave the field its standard automated white-box attack — is Greedy Coordinate Gradient (GCG), introduced by Zou, Wang, Carlini, Nasr, Kolter, and Fredrikson in "Universal and Transferable Adversarial Attacks on Aligned Language Models" (2023).
Greedy Coordinate Gradient: gradients as a ranking signal, not a step direction
GCG's move is to stop asking the gradient to move the suffix and start asking it to rank candidates for a discrete substitution. Represent the token at suffix position $i$ as a one-hot vector $e_i \in \mathbb{R}^V$ (relaxing "this position holds a specific token" into "this position holds a point on the probability simplex," even though only the vertices — exact one-hot vectors — are valid). The loss $L$ is a differentiable function of these relaxed one-hot vectors, so $\partial L / \partial e_i$ is well defined: a $V$-dimensional vector whose $j$-th entry is the loss's first-order sensitivity to swapping position $i$ toward token $j$. The most negative entries identify tokens that, to a linear approximation, would most reduce the loss.
Because it is only a linear approximation — real transformers are highly nonlinear through the softmax and every downstream attention layer — GCG never trusts the gradient's top pick blindly. The full algorithm, one iteration:
- Forward pass on (frozen instruction + current suffix) → logits → loss $L$.
- One backward pass gives $\partial L/\partial e_i$ for every suffix position $i$ simultaneously — not one backward pass per position.
- For each position, keep the top-$k$ tokens with the smallest (most negative) gradient entries — a candidate substitution pool, cheap to compute (just a sort).
- Sample a batch of $B$ (position, token) pairs from that pool and, for each, actually substitute the token and run a real forward pass to get the exact loss — no more linear approximation.
- Keep whichever sampled candidate produced the lowest true loss; overwrite that one suffix token; repeat.
This is why the compute cost is tractable at all. Brute-forcing "which single-token swap, at which of the $L$ suffix positions, most reduces the loss" by exhaustive evaluation costs $L \times V$ forward passes per iteration — for a 20-token suffix and a 32,000-token vocabulary, 640,000 forward passes just to make one edit. GCG replaces that with one backward pass (which yields gradients for all positions at once, the way backprop always does) plus a single batched forward pass over $B$ candidates — typically a few hundred. That is a reduction of three orders of magnitude, and it is the entire reason gradient-guided candidate selection is the algorithm's contribution rather than a minor speedup.
One more production detail worth naming, since it connects directly to something you already know about transformer inference: step 4's batched evaluation does not need to recompute attention over the frozen instruction prefix for every one of the $B$ candidates — that prefix is identical across all of them. An inference engine computes its key/value cache once for the shared prefix and reuses it, paying the full attention cost only for the suffix tokens onward. The same KV-cache trick that makes chatbot inference fast is what makes GCG's inner loop fast.
Worked example: one GCG step, by hand
A real transformer's gradient can't be traced on paper, so here is a linear surrogate small enough to compute exactly, that reproduces the same algebra GCG runs (softmax cross-entropy, backprop to a one-hot input) without hiding any step. Vocabulary $\{0,1,2,3\}$, a suffix of length one, and a "model" $f(x) = W e_x$ where $e_x$ is the one-hot vector for suffix token $x$ and
W = [[ 0.1, -0.2, 0.3, 0.0],
[ 0.4, 0.1, -0.1, 0.2],
[-0.3, 0.5, 0.2, -0.4],
[ 0.0, 0.1, 0.0, 0.3]]Row $t$, column $x$ gives the logit for output token $t$ when the suffix is token $x$. Target completion token: $T = 2$. Current suffix: $x_0 = 0$ (GCG's real initializer is similarly arbitrary — the original paper starts from a repeated exclamation mark).
Step 1 — forward pass. $\text{logit}(x_0) = W e_0 = $ column 0 $= [0.1, 0.4, -0.3, 0.0]$. Exponentiating: $[1.1052, 1.4918, 0.7408, 1.0000]$, sum $= 4.3378$. Softmax: $p_0 = [0.2548, 0.3439, 0.1708, 0.2305]$.
Step 2 — loss. $L_0 = -\log p_{0,T} = -\log(0.1708) = 1.7676$ nats.
Step 3 — backward pass. For softmax cross-entropy, $\partial L/\partial \text{logit} = p_0 - \mathbf{1}_T = [0.2548, 0.3439, -0.8292, 0.2305]$. Pulling this back through the linear map ($\partial L/\partial e_x = W^\top(p_0 - \mathbf{1}_T)$) gives, entry by entry:
g0 = 0.1(0.2548) + 0.4(0.3439) + (-0.3)(-0.8292) + 0.0(0.2305) = 0.4118
g1 = -0.2(0.2548) + 0.1(0.3439) + 0.5(-0.8292) + 0.1(0.2305) = -0.4081
g2 = 0.3(0.2548) + (-0.1)(0.3439) + 0.2(-0.8292) + 0.0(0.2305) = -0.1238
g3 = 0.0(0.2548) + 0.2(0.3439) + (-0.4)(-0.8292) + 0.3(0.2305) = 0.4697Step 4 — rank candidates. Excluding the currently active token (a self-swap is a no-op), the two smallest gradient entries belong to token 1 ($-0.4081$) and token 2 ($-0.1238$) — the top-2 candidate set.
Step 5 — batch-evaluate true loss. For $x=1$: column 1 of $W$ is $[-0.2, 0.1, 0.5, 0.1]$, softmax gives $p_T = 0.3525$, so $L = 1.0428$. For $x=2$: column 2 is $[0.3,-0.1,0.2,0.0]$, softmax gives $p_T = 0.2729$, so $L = 1.2987$. Both beat $L_0 = 1.7676$; token 1 wins.
Step 6 — commit. The suffix updates from token 0 to token 1. Loss drops from 1.7676 to 1.0428 nats in a single step — and here the gradient's ranking (token 1 better than token 2) matched the true evaluation exactly. That won't always happen, and the reason it won't is itself instructive (see the active-recall ripple question below).
The same computation runs verbatim in code, which is worth having as a second, independently checkable trace of the same arithmetic:
import numpy as np
W = np.array([
[ 0.1, -0.2, 0.3, 0.0],
[ 0.4, 0.1, -0.1, 0.2],
[-0.3, 0.5, 0.2, -0.4],
[ 0.0, 0.1, 0.0, 0.3],
])
target = 2
def loss_and_softmax(x_idx):
e = np.zeros(4)
e[x_idx] = 1.0
logit = W @ e
p = np.exp(logit) / np.exp(logit).sum()
return -np.log(p[target]), p
L0, p0 = loss_and_softmax(0) # L0 = 1.7676
grad = W.T @ (p0 - np.eye(4)[target]) # [0.412,-0.408,-0.124,0.470]
ranked = np.argsort(grad) # [1, 2, 0, 3]
candidates = [i for i in ranked if i != 0][:2] # [1, 2]
true_losses = {i: loss_and_softmax(i)[0] for i in candidates}
# true_losses == {1: 1.0428, 2: 1.2987} -> token 1 winsEvery variable used (W, target, p0, grad, ranked, candidates) is defined before use, and loss_and_softmax is defined above its first call — this snippet runs as written and reproduces the hand-derived numbers exactly, because it performs the identical operations (linear map, softmax, cross-entropy, backprop through a linear layer) in the identical order.
Why a refusal in the chat window proves less than it feels like it proves
The misconception this whole mechanism exposes: "I asked the model directly, in the product's chat interface, and it refused — so it's safe against this class of attack." That reasoning treats safety as a single, load-bearing test. It isn't. RLHF and safety fine-tuning shape a decision boundary in an extremely high-dimensional input space — one learned from the distribution of prompts the model was fine-tuned on, which is overwhelmingly natural language typed by humans. A manual red-team session, however thorough, is still sampling points from that same natural-language region: different phrasings of "pretend you're evil," different roleplay setups. The refusal boundary is comparatively well-defended exactly there, because that's where training pressure was concentrated.
GCG's suffix does not live in that region at all. It is optimized, coordinate by coordinate, specifically to land in whatever pocket of input space minimizes the target loss — and nothing in the fine-tuning objective guarantees that pocket was ever visited during training. A single manual test, or even a thousand of them, is a point sample of a boundary; gradient-guided search is closer to an adversarial worst-case query against that same boundary, run by an optimizer that doesn't get tired, bored, or predictable the way a human tester does. That's the precise sense in which "systematic" in this chapter's title means something narrower and stronger than "structured": not a checklist run carefully, but a search procedure with a defined objective, a defined stopping condition, and — critically — the transfer property demonstrated by the fintech scenario above, where a suffix optimized entirely offline against an open model still worked against a different, closed, hosted model. Aligned models fine-tuned from similar base checkpoints and similar RLHF pipelines share enough of their loss landscape that adversarial suffixes generalize across them; the attacker never needed to touch the production system to find what breaks it.
The black-box route: red-teaming with a red language model
GCG needs gradients, which means it needs model weights — a genuine limitation when the target is a closed API. Perez, Huang, Song, Cai, Ring, Aslanides, Glaese, McAleese, and Irving's "Red Teaming Language Models with Language Models" (DeepMind, EMNLP 2022) solves systematic discovery from the opposite direction: no gradients at all, just a second language model — the "red LM" — whose job is to generate candidate test prompts, and a classifier that scores the target model's response for the harm category under test (offensiveness, PII leakage, and others in the paper).
The paper compares four ways to drive the red LM: zero-shot generation (just prompt it to produce test cases), few-shot generation seeded with examples of prompts that worked before, supervised fine-tuning on the highest-scoring test cases found so far, and reinforcement learning, where the red LM is trained with the classifier's score as reward, directly optimizing "generate the prompt most likely to make the target fail." That last variant is the closest black-box analogue to GCG's loss-minimization loop: instead of following a token-level gradient, it follows a policy-gradient signal computed from classifier scores over full generated prompts.
The two techniques trade different things for different guarantees:
| Dimension | GCG (gradient, white-box) | Red-LM + classifier (Perez et al., black-box) |
|---|---|---|
| Access required | Model weights (for gradients) | Only an API to sample outputs from the target |
| Search signal | Token-level gradient of a loss | Classifier score over full model responses |
| Typical output | Short, often non-fluent token suffix | Full natural-language prompts (fluent by construction) |
| Coverage strength | Deep local search around one instruction | Broad generation across many topics/harms at once |
| Main failure mode | High-perplexity suffix is filterable | RL can mode-collapse onto a narrow set of winning templates |
The RL mode-collapse failure is worth naming precisely because it is the generative method's version of the same lesson GCG's step 6 teaches: an optimizer that only sees a scalar reward will happily overfit to whatever narrow pattern maximizes that reward, sacrificing the diversity that makes "systematic" discovery worth more than one lucky manual find. The paper's mitigation is to not rely on RL alone — mixing it with the more diverse zero-shot and few-shot generation methods, and explicitly measuring the diversity of discovered test cases as a metric in its own right, not just their average attack success rate.
Active recall
Attempt these before reading the answers. Questions 3 and 5 build directly on the worked example above.
- Why can't you run ordinary gradient descent directly on the suffix tokens, the way you would on the pixels of an adversarial image?
- Using the same matrix $W$ and target $T=2$ from the worked example, suppose the initial suffix token is $x_0 = 3$ instead of $0$. Compute $L(x_0=3)$.
- Ripple question. Keep $x_0 = 0$, but change the target to $T = 0$ instead of $T=2$. Recompute the gradient $g$, determine the new top-2 candidate set, and check both candidates' true loss against the current loss. Does the gradient ranking still match the true-loss ranking?
- Why does training the red LM purely with reinforcement learning against a fixed classifier tend to shrink the diversity of discovered test cases over time, and what does Perez et al. do about it?
- For a suffix of length $L=20$, vocabulary $V=32{,}000$, top-$k=256$, batch $B=512$: roughly how many forward passes does one GCG iteration take, versus brute-force enumeration of every single-token substitution at every position?
- "GCG suffixes look like nonsense, so a perplexity filter fully defeats this attack class." True, false, or partly — and why?
1. A token is a discrete index into an embedding table; there is no token "between" two tokens for an infinitesimal gradient step to land on. GCG doesn't use the gradient as a step direction at all — it uses $\partial L/\partial e_i$ only to rank which discrete substitutions are worth actually testing, then falls back to an exact forward pass to choose among them.
2. Column 3 of $W$ is $[0.0, 0.2, -0.4, 0.3]$. Exponentials: $[1.0000, 1.2214, 0.6703, 1.3499]$, sum $=4.2416$. $p_T = 0.6703/4.2416 = 0.1580$. $L = -\log(0.1580) = 1.8450$ nats — worse (higher loss) than the original $x_0=0$ start of 1.7676, so token 3 would never have been chosen from token 0 as a starting point either.
3. With $T=0$, $\partial L/\partial\text{logit} = p_0 - \mathbf{1}_0 = [-0.7452, 0.3439, 0.1708, 0.2305]$ (same $p_0$ as before — softmax doesn't depend on the target). Pulling back through $W$: $g = [0.0118, 0.2919, -0.2238, 0.0696]$. Excluding the current token (index 0), the two smallest entries are token 2 ($-0.2238$) and token 3 ($0.0696$) — the top-2 set is now $\{2, 3\}$, completely different from the $T=2$ case's $\{1, 2\}$, because token 1's column strongly boosts logit 2 (which was the target before but is now a competitor). Checking true loss: current ($x_0=0$, $T=0$) gives $L=1.3672$; candidate token 2 gives $L=1.1987$ (a real improvement, matching the gradient's top pick); candidate token 3 gives $L=1.4449$ — worse than doing nothing at all, despite having the second-most-negative gradient entry. The gradient ranking failed for the second candidate: it's a linear approximation taken at the current point, and token 3's one-hot vertex is far enough away in the discrete simplex that the linear extrapolation misleads. This is exactly why GCG's step 6 (batch-evaluate true loss) is not a formality — trusting the gradient ranking past the top pick can select a substitution that actively makes the attack worse.
4. Reinforcement learning optimizes purely for the scalar reward (classifier score). Once the red LM finds a small family of phrasings that reliably score high, further exploration away from that family is reward-negative in expectation, so policy-gradient training concentrates probability mass on a narrow, repetitive set of templates — the attack-success-rate metric looks excellent while actual coverage of the harm space shrinks. Perez et al. mitigate this by not relying on RL in isolation: they combine it with zero-shot and few-shot generation (which keep sampling broadly at higher effective temperature) and track diversity of the discovered test cases as a separate success criterion, not just mean reward.
5. Brute force: $L \times V = 20 \times 32{,}000 = 640{,}000$ forward passes to be certain of the single best one-token swap. GCG: one backward pass computes gradients for all 20 positions at once (not 20 separate backward passes), the top-$k$ selection is a cheap sort with no extra model calls, and the batched evaluation is $B=512$ forward passes, which can run as one parallel batched call reusing the shared prefix's KV-cache. Total real model computation per iteration: roughly 1 backward pass + 1 batched forward pass of size 512 — on the order of 500-600 effective forward-pass-equivalents versus 640,000, over three orders of magnitude less.
6. Partly true, and worth being precise about which part. GCG's raw suffixes genuinely are high-perplexity nonsense, so a perplexity filter is a real, cheap, and reasonably effective baseline defense against unmodified GCG output — it's used as exactly such a baseline in the follow-up defense literature (e.g. Jain et al.'s baseline-defenses paper). But it is not a complete answer to "systematic vulnerability discovery" as a threat class for two reasons. First, the optimization objective in step 3 can be extended to jointly penalize perplexity alongside the target loss, trading some attack success rate for fluency — an active arms race, not something the attacker is stuck with. Second, and more fundamentally, a perplexity filter does nothing at all against the Perez et al.-style generative attack: a red LM optimizing against a classifier is explicitly rewarded for producing fluent, natural-sounding prompts (roleplay, hypothetical framing), so there is no elevated perplexity to detect in the first place. The two techniques in this chapter attack the same underlying refusal boundary from different sides, and a defense tuned against one leaves the other's threat surface untouched.
Think About It
Think about this: How would you explain red teaming llms: systematic vulnerability discovery to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where red teaming llms: systematic vulnerability discovery is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting red teaming llms: systematic vulnerability discovery to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind red teaming llms: systematic vulnerability discovery, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.