A team building a regional-language customer-support assistant — say, for a telecom or insurance platform serving tens of millions of users across Hindi, Tamil, and Bengali — fine-tunes an open 7-billion-parameter model overnight on a single GPU using LoRA. Training memory is solved: the adapter matrices are a few megabytes, the base model stays frozen, and the whole job fits on hardware a college lab can afford. Then the model has to go live. During a festival-season traffic spike the assistant must serve thousands of concurrent conversations, and the arithmetic changes completely. Each replica of the model, stored in 16-bit floating point, needs 7,000,000,000 parameters × 2 bytes = 14,000,000,000 bytes ≈ 14 GB just to hold the weights — before a single token of conversation is processed, before the KV-cache for ongoing conversations is allocated. A single 40 GB GPU holds barely two such replicas once you leave room for activations and cache, and every extra GPU is real monthly cloud spend. LoRA solved the training-time memory problem. It did nothing about the serving-time memory problem, because merging a LoRA adapter back into the base weights, or even just leaving it attached, still leaves you with a 7-billion-parameter model at inference. This chapter is about what happens after tuning and before serving: parameter-efficient tuning methods that live in a different part of the design space than LoRA, and model compression techniques — quantization, distillation, structured pruning — that shrink an already-trained model so it can actually be deployed cheaply.
Beyond low-rank updates: how else can you tune fewer parameters?
LoRA works by injecting a trainable low-rank correction into existing weight matrices while freezing everything else — a change to the network's weights. That is one point in a much larger design space. Three other families of parameter-efficient fine-tuning (PEFT) methods change something other than the weights, and are worth knowing precisely because they fail and succeed under different conditions than LoRA does.
Prompt tuning (Lester, Al-Rfou & Constant, "The Power of Scale for Parameter-Efficient Prompt Tuning", EMNLP 2021) touches no weight matrix at all. It prepends a small number P of trainable continuous embedding vectors — a "soft prompt" — to the input sequence's embeddings, and trains only those P vectors while the entire backbone, including the token embedding table, stays frozen. The model is steered purely by what it sees at the input, the same way a well-chosen natural-language prompt steers a frozen model, except the prompt is now a set of learned real-valued vectors rather than discrete tokens, so gradient descent can search a continuous space instead of the discrete space of English words.
Prefix tuning (Li & Liang, "Prefix-Tuning: Optimizing Continuous Prompts for Generation", ACL 2021) goes one layer of the model further. Instead of only prepending vectors to the input embeddings, it prepends trainable key and value vectors directly into the self-attention computation at every transformer layer, not just the first. Every layer's attention block therefore attends to its own learned "virtual tokens," so the trainable signal can reshape the representation at each depth of the network, not merely bias what the first layer starts from. This is strictly more expressive than prompt tuning, at the cost of more trainable parameters, as the worked example below quantifies.
BitFit (Ben Zaken, Ravfogel & Goldberg, "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models", 2021) takes the opposite approach: it changes nothing about the input or the attention mechanism, and instead unfreezes only the bias terms already present in every linear and layer-norm sublayer, training nothing else. Biases are a tiny fraction of a transformer's parameters, yet the paper shows this captures a surprising amount of task adaptation — evidence that much of what "fine-tuning" accomplishes on many tasks is a shift in each layer's operating point rather than a change in the linear transformation itself.
(IA)³ (Liu et al., "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning", NeurIPS 2022) learns three per-channel rescaling vectors per transformer layer, one each for the attention keys, the attention values, and the feed-forward intermediate activations. These vectors multiply element-wise into the existing activations rather than adding a new matrix product, so the parameter count is even smaller than a low-rank LoRA update, and — unlike prompt or prefix tuning — it adds no extra tokens to the sequence, so it costs nothing extra at inference time.
The structural point worth holding onto: LoRA and adapters modify weights; prompt and prefix tuning modify the input context the frozen weights see; BitFit and (IA)³ modify small existing parameters (biases, scales) rather than adding new structure. Because prompt and prefix tuning lengthen every forward pass by P extra positions, they trade a smaller training footprint for a small but permanent inference-time cost — while a merged LoRA adapter costs nothing extra at serving time. Choosing among these is a real systems decision, not just a training-cost decision.
Worked example: how many parameters does each method actually add?
Take a model shaped like a 7-billion-parameter LLM: hidden size d_model = 4096, L = 32 transformer layers, and a chosen prompt/prefix length of P = 20 virtual tokens.
d_model = 4096
n_layers = 32
P = 20 # number of virtual tokens
full_model_params = 7_000_000_000
prompt_tuning_params = P * d_model
prefix_tuning_params = 2 * n_layers * P * d_model # one K vector and one V vector per layer
print(prompt_tuning_params) # 81920
print(prefix_tuning_params) # 5242880
print(f'{prompt_tuning_params / full_model_params * 100:.5f} percent') # 0.00117 percent
print(f'{prefix_tuning_params / full_model_params * 100:.5f} percent') # 0.07490 percent
print(prefix_tuning_params / prompt_tuning_params) # 64.0
Prompt tuning trains 81,920 parameters — about 0.0012% of the backbone — because it is nothing but a P × d_model embedding table. Prefix tuning trains 5,242,880 parameters, about 0.075% of the backbone, sixty-four times more than prompt tuning, because it needs its own key and value vector pair injected at each of the 32 layers rather than a single shared input. Both numbers are three to four orders of magnitude smaller than full fine-tuning's 7 billion, but they are not interchangeable: prefix tuning's extra capacity typically closes more of the gap to full fine-tuning on harder generation tasks, which is exactly why Li and Liang built it as a strict generalization of the input-only idea rather than stopping at prompt tuning.
Compressing a trained model: quantization
Every method above still produces a model whose weights are stored as 16-bit (or 32-bit) floating-point numbers. Quantization discards precision deliberately: it stores each weight as a low-bit-width integer plus a small amount of shared metadata (a scale, sometimes a zero-point), and reconstructs an approximate floating-point value only when needed for computation. For a signed b-bit integer representation, the usable range is [-2^(b-1), 2^(b-1) - 1] — for 4-bit integers, [-8, 7], sixteen distinct levels. The simplest scheme, symmetric round-to-nearest (RTN) quantization, picks one scale per group of weights from the largest magnitude in that group:
def quantize_symmetric(w, bits=4):
qmax = 2 ** (bits - 1) - 1 # 7 for 4-bit
qmin = -2 ** (bits - 1) # -8 for 4-bit
scale = max(abs(x) for x in w) / qmax
q = [round(x / scale) for x in w]
q = [max(qmin, min(qmax, qi)) for qi in q]
dq = [qi * scale for qi in q]
return q, dq, scale
A misconception worth correcting
The intuitive picture of quantization is that it makes every weight a little blurrier by roughly the same amount — a uniform loss of resolution, so a 4-bit model should be a uniformly "worse" version of the 16-bit one. That picture is wrong, and the reason it is wrong is the actual mechanism the diagram below traces: the scale is set by the single largest-magnitude weight in the group. If one weight is an outlier — and empirical work on large transformers (Dettmers, Lewis, Belkada & Zettlemoyer, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", NeurIPS 2022) shows that a small number of "salient" weight channels, tied to a few high-magnitude activation dimensions, reliably dominate a layer's output — that one outlier stretches the scale so far that every ordinary-sized weight collapses onto one or two of the sixteen available integer levels. The error is not spread evenly; it is concentrated almost entirely on the weights that were not outliers, while the outlier itself is quantized almost exactly. Naive uniform rounding does not fail gracefully across the board — it fails catastrophically on the majority of values and barely at all on the few that set the scale.
Full worked example: six weights, one outlier
Consider one output channel's weights, drawn from a real-looking row of a weight matrix: w = [0.10, -0.15, 0.22, -0.08, 0.31, 3.80]. The first five are ordinary; the sixth, 3.80, is the kind of outlier weight LLM.int8() identifies as tied to a salient activation channel.
Step 1 — naive 4-bit symmetric quantization. The scale is set by the largest magnitude across all six weights:
w = [0.10, -0.15, 0.22, -0.08, 0.31, 3.80]
q, dq, scale = quantize_symmetric(w, bits=4)
print(round(scale, 4)) # 0.5429 (= 3.80 / 7)
print(q) # [0, 0, 0, 0, 1, 7]
print([round(v, 4) for v in dq]) # [0.0, 0.0, 0.0, 0.0, 0.5429, 3.8]
Every weight below 0.271 (half the scale) rounds to integer level 0, which dequantizes to exactly 0.0. Four of the five ordinary weights — 0.10, −0.15, 0.22, and −0.08 — fall in that band and are wiped out entirely: 100% relative error each. The fifth, 0.31, rounds to level 1 and dequantizes to 0.5429, a 75% relative error. Only the outlier, 3.80, lands exactly on level 7 with zero error, precisely because it is the value that defined the scale in the first place. Five of six weights are destroyed to preserve the one that needed the least help.
Step 2 — isolate the outlier, requantize the rest. This is the core idea shared by LLM.int8()'s mixed-precision decomposition and, in a related but distinct form, AWQ's per-channel scaling: keep the outlier out of the scale computation entirely.
w_normal = [0.10, -0.15, 0.22, -0.08, 0.31]
q2, dq2, scale2 = quantize_symmetric(w_normal, bits=4)
print(round(scale2, 5)) # 0.04429 (= 0.31 / 7)
print(q2) # [2, -3, 5, -2, 7]
print([round(v, 4) for v in dq2]) # [0.0886, -0.1329, 0.2214, -0.0886, 0.31]
With the outlier removed, the scale shrinks by a factor of roughly 12.3 (from 0.5429 to 0.04429), and every one of the five weights now lands on its own distinct integer level. The relative errors are 11.4%, 11.4%, 0.6%, 10.7%, and 0.0% — worst case 11.4%, against a worst case of 100% a moment ago. The outlier itself is simply stored separately, either kept at full 16-bit precision (LLM.int8()'s approach) or, in AWQ (Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", 2023), scaled up before quantization and rescaled back down in the corresponding activation, so the whole matrix can still be stored at a uniform bit width without a hardware-unfriendly mix of precisions.
Beyond round-to-nearest: GPTQ, AWQ, and QLoRA
The round-to-nearest scheme above quantizes every weight independently and never revisits a decision once made. GPTQ (Frantar, Ashkboos, Hoefler & Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023) does better by quantizing a weight matrix one column at a time and, after rounding each column, adjusting the still-unquantized remaining columns to compensate for the error just introduced — using an approximate inverse-Hessian computed from a small calibration set of real activations, in the spirit of the older Optimal Brain Surgeon pruning method. Because the compensation happens layer by layer with no gradient updates and no labeled data, GPTQ needs no retraining, yet it pushes usable quantization down to 4 bits and below with far smaller accuracy loss than naive rounding. AWQ takes a different route to the same destination: rather than compensating after the fact, it identifies the small set of salient weight channels ahead of time (the outlier-detection idea in the worked example above, generalized from single weights to whole channels correlated with high-magnitude activations) and scales those channels up before quantization while scaling the corresponding activations down by the same factor — protecting their resolution without ever leaving uniform bit-width storage or requiring mixed-precision hardware kernels.
QLoRA (Dettmers, Pagnoni, Holtzman & Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs", NeurIPS 2023) is the answer to the opening scenario directly: it quantizes the frozen backbone to 4 bits using a custom data type called NF4 ("4-bit NormalFloat"), whose sixteen levels are spaced to match a normal distribution rather than spaced uniformly — appropriate because trained weights are empirically close to normally distributed, unlike the uniform grid used above — and then trains LoRA adapters on top of that frozen 4-bit backbone, with the adapter arithmetic itself carried out in 16-bit precision. Training and serving-side compression, treated as two separate stages throughout this chapter, become one pipeline: a 7-billion-parameter model that needed 14 GB in FP16 fits comfortably in under 5 GB in NF4, on a single consumer GPU, while still being tunable.
Compressing a trained model: knowledge distillation
Quantization keeps the same architecture and shrinks each number's precision. Knowledge distillation (Hinton, Vinyals & Dean, "Distilling the Knowledge in a Neural Network", NeurIPS Deep Learning Workshop 2015) instead trains a smaller "student" network from scratch to imitate a larger "teacher," and the trick that makes this work well is what the student is trained to imitate. Training the student only on the teacher's hard predicted label throws away almost all the information the teacher's output distribution carries about which wrong answers are "less wrong" than others. Hinton's fix is to soften the teacher's output with a temperature T before computing the target the student learns from:
p_i(T) = exp(z_i / T) / sum_j exp(z_j / T)
where z are the pre-softmax logits. At T = 1 this is the ordinary softmax; raising T flattens the distribution and reveals the teacher's relative confidence across the wrong classes — its "dark knowledge." Take a three-way intent classifier over logits z_teacher = [4.0, 1.0, 0.5] and a partially-trained student with z_student = [3.0, 1.5, 0.8]:
# T = 1 (ordinary softmax)
teacher@T1 = [0.9259, 0.0461, 0.0280]
student@T1 = [0.7497, 0.1673, 0.0831]
KL(teacher || student) at T=1 ≈ 0.1057
# T = 5 (softened)
teacher@T5 = [0.4889, 0.2683, 0.2428]
student@T5 = [0.4193, 0.3106, 0.2701]
KL(teacher || student) at T=5 ≈ 0.0099
At T = 1, the teacher's distribution is so peaked (92.6% on the top class) that classes 2 and 3 are barely distinguishable in the target — the student gets almost no signal about which of the two wrong intents is the "nearer miss." At T = 5, the teacher's true relative preference for class 2 over class 3 (26.8% vs 24.3%, not identical) becomes visible, and the KL divergence the student is minimizing carries that information. The gradient from a softened target scales down roughly as 1/T², so Hinton's distillation loss multiplies the soft-target term by T² before combining it with the ordinary hard-label cross-entropy loss, keeping the two gradient magnitudes comparable rather than letting the softened term vanish. The student that results is typically a different, smaller architecture — fewer layers, smaller hidden size — trained once, rather than the same architecture at lower numeric precision.
Compressing a trained model: structured pruning
Pruning removes weights outright rather than reducing their precision. Naive magnitude pruning — zeroing out the smallest-magnitude weights in a matrix up to some target sparsity — is cheap to compute but usually needs retraining to recover accuracy, and even then, an unstructured sparsity pattern (scattered individual zeros) rarely produces a real speedup, because commodity GPU matrix-multiply hardware is built for dense tiles; skipping individual scattered zeros usually costs more in irregular memory access than it saves in arithmetic. SparseGPT (Frantar & Alistarh, "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot", ICML 2023) solves the accuracy side the same way GPTQ solves quantization's accuracy side: it prunes a weight matrix using approximate second-order (Hessian) information from calibration data and, after removing each weight, adjusts the remaining weights in that row to compensate — all in one pass, with no retraining, reaching 50%+ sparsity on large transformers with modest accuracy loss. The hardware side still needs a structured pattern to pay off — NVIDIA's Ampere and later GPUs accelerate a specific 2:4 pattern (exactly two zeros in every group of four weights), which is regular enough for the hardware to skip, unlike arbitrary unstructured sparsity. Pruning and quantization compose: a model can be pruned to 2:4 structured sparsity and then quantized, compounding the memory and compute savings, though each step's compensation math (SparseGPT's Hessian-based adjustment, GPTQ's Hessian-based adjustment) has to be re-run against the model as it stands after the previous step, not against the original dense, full-precision weights.
Active recall
Attempt each question before reading the answer beneath it.
1. A model uses prefix tuning with P = 10 virtual tokens on a backbone with d_model = 2048 and L = 24 layers. How many trainable parameters does this add, and how does that compare to prompt tuning with the same P and d_model?
2. In the Step-1 quantization worked example, suppose the outlier weight w6 were 1.90 instead of 3.80 — half its original magnitude — while w1 through w5 stay the same. Recompute the naive scale and the quantization level each of w1–w5 lands on. Does the worst-case relative error among w1–w5 improve, stay the same, or get worse? Does anything change for the Step-3 (outlier-isolated) result?
3. Why does raising the distillation temperature T from 1 to 5 in the worked example shrink the KL divergence between teacher and student, and why is a shrinking KL divergence not, by itself, evidence that the student is learning more from the teacher at higher T?
4. A colleague proposes pruning 60% of a weight matrix's individual entries by magnitude, expecting a proportional 60% inference speedup on a standard GPU. What is wrong with this expectation, and what would have to change about the pruning pattern for the speedup to materialize?
5. Why does (IA)³ add zero extra inference latency compared to prompt tuning or prefix tuning, given that all three are described as parameter-efficient fine-tuning methods?
6. QLoRA quantizes the frozen backbone to 4-bit NF4 and trains LoRA adapters in 16-bit precision on top of it. Why must the adapters stay at higher precision even though the backbone they're attached to is 4-bit?
Answers
1. Prefix tuning: 2 × L × P × d_model = 2 × 24 × 10 × 2048 = 983,040 parameters (one learned K and one learned V vector per layer per virtual token). Prompt tuning: P × d_model = 10 × 2048 = 20,480 parameters. Prefix tuning uses 48 times more trainable parameters here — the same 2×L multiplier as the chapter's worked example, since that factor depends only on injecting a K/V pair at every one of the L layers, not on the specific d_model or P chosen.
2. New scale = 1.90 / 7 ≈ 0.2714. Quantizing: w1=0.10 → round(0.368) = 0, w2=-0.15 → round(-0.553) = -1, w3=0.22 → round(0.811) = 1, w4=-0.08 → round(-0.295) = 0, w5=0.31 → round(1.142) = 1. Dequantized: [0, -0.271, 0.271, 0, 0.271]. Relative errors: 100%, 81%, 23%, 100%, 12%. The worst case is unchanged at 100% — w1 and w4 still land exactly on level 0 and are still completely destroyed — even though the average error across the five weights improved because w2, w3, and w5 now land on nonzero levels. Halving the outlier helps some weights without fixing the fundamental failure mode. Nothing changes for Step 3: it explicitly excludes w6 from the scale computation, so it never sees the outlier's value at all — this invariance to the outlier's magnitude is exactly why outlier isolation is the robust fix rather than a magnitude-dependent one.
3. Raising T divides every logit by a larger number before the softmax, which pulls all the exponentiated values closer to each other and flattens both distributions toward uniform. Two distributions that are each closer to uniform are automatically closer to each other in KL divergence, regardless of whether the student has actually learned the teacher's fine-grained preferences better. What higher T actually contributes is not a smaller number to optimize but more information in the target itself — the relative ordering and spacing among the non-top classes becomes visible in the gradient rather than being rounded away by a near-one-hot target. The T² multiplier on the soft-target loss term exists precisely to stop the shrinking KL divergence from being mistaken for a shrinking learning signal.
4. The expectation confuses removing 60% of the values with removing 60% of the work. Magnitude pruning by default produces an unstructured, scattered zero pattern; dense GPU matrix-multiply kernels are built to stream contiguous tiles of memory efficiently and get no benefit from skipping individually scattered zeros — the irregular memory access pattern needed to detect and skip them typically costs more than it saves. For the speedup to show up, the sparsity has to be structured — for instance the 2:4 pattern (exactly two zeros in every four consecutive weights) that Ampere-generation-and-later NVIDIA GPUs have dedicated hardware support for — and accuracy typically needs to be recovered with a compensation method like SparseGPT rather than plain magnitude pruning.
5. Prompt and prefix tuning both add extra tokens (P input embeddings, or P key/value pairs per layer) that the model must process on every single forward pass, permanently lengthening the effective sequence the attention mechanism operates over. (IA)³ instead learns rescaling vectors that multiply into the existing key, value, and feed-forward activations already computed on the real input tokens — once training is done, those vectors can be folded into the corresponding weight matrices (much like a LoRA adapter can be merged), so no extra tokens and no extra runtime computation are added at all.
6. The LoRA update at each layer is a low-rank correction added to that layer's (dequantized) output, and gradient descent on the adapter matrices needs reasonably precise arithmetic to accumulate small updates over many training steps without those updates being swallowed by rounding error — 4-bit integers cannot represent the small, smoothly varying gradients that training relies on. The frozen 4-bit backbone only ever needs to be dequantized on the fly for the forward pass's matrix multiplications, where the fixed NF4 levels are accurate enough because those weights are never updated; the adapters, which are being trained, are the part of the computation that actually needs precision headroom.
Think About It
Think about this: How would you explain parameter-efficient tuning and model compression to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where parameter-efficient tuning and model compression is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting parameter-efficient tuning and model compression to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind parameter-efficient tuning and model compression, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.