Parameter-Efficient Fine-tuning: Adapters and LoRA

AI4Bharat's IndicTrans and IndicBERT efforts face a specific arithmetic problem. Suppose a lab wants one 7-billion-parameter base model — say, a Llama-2-7B-class transformer with hidden dimension 4096 and 32 transformer blocks — to serve high-quality translation and classification for all 22 scheduled Indian languages. Full fine-tuning trains a separate copy of every one of the model's ~7 billion parameters per language. Stored in 16-bit precision, one fine-tuned copy is roughly 14 GB. Twenty-two of them is about 308 GB of checkpoints, and training each one with the Adam optimizer needs GPU memory for the weights, the gradients, and two optimizer moment buffers per parameter — commonly accounted at around 16 bytes per trainable parameter in mixed-precision training (2 bytes for fp16 weights, 2 for fp16 gradients, 4+4 for the fp32 Adam momentum and variance, 4 for an fp32 master copy). At 7 billion parameters that is roughly 7×10⁹ × 16 bytes ≈ 112 GB just to hold the weights, gradients, and optimizer state for one language — before any activations. That does not fit on a single GPU, and doing it 22 times is a cluster-scale expense for what is, in each case, a narrow adaptation of an already-capable model.

Parameter-efficient fine-tuning (PEFT) asks a sharper question: instead of moving all 7 billion parameters, can we freeze the pretrained weights entirely and learn a much smaller set of new parameters that steers the frozen model toward the new language or task? This chapter works through the two techniques that established this idea and still anchor it in production systems: bottleneck adapters (Houlsby et al., "Parameter-Efficient Transfer Learning for NLP," ICML 2019) and Low-Rank Adaptation, LoRA (Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," ICLR 2022). Both cut trainable parameters by orders of magnitude. They differ in one architectural choice — serial insertion versus parallel addition — and that single choice has consequences for inference latency, deployability, and how many task-specific "add-ons" a single frozen base can serve at once, which is exactly the AI4Bharat problem.

Adapter modules: a serial bottleneck inside every layer

Houlsby et al.'s adapter inserts a small new module directly into the computation graph of each transformer sublayer, after the multi-head attention block and after the feed-forward block. Each adapter module is a bottleneck autoencoder: it projects the sublayer's d-dimensional output down to a small dimension m, applies a nonlinearity, projects back up to d, and adds the result back via a residual (skip) connection — so the module can learn to be close to the identity function when nothing needs to change. Everything else in the transformer (the attention weights, the feed-forward weights, the embeddings) is frozen; only the adapters' down-projection, up-projection, and their biases are trained.

The parameter count for a single adapter with input/output dimension d and bottleneck dimension m is:

params_adapter = (d × m) + (m × d) + m + d = 2dm + d + m

For d = 4096 and a typical bottleneck m = 64: 2 × 4096 × 64 + 4096 + 64 = 524,288 + 4,160 = 528,448 parameters per adapter. Houlsby's design places two adapters per transformer block (one after attention, one after the feed-forward sublayer), so one block costs 2 × 528,448 = 1,056,896 parameters. Across 32 blocks: 1,056,896 × 32 = 33,820,672 — about 33.8 million trainable parameters, or roughly 0.48% of a 7-billion-parameter model. That is already a huge win over full fine-tuning's 100%, but it is not the smallest possible win, and the reason why sets up LoRA.

LoRA: reparametrizing the weight update, not the layer

LoRA starts from a different observation. Fine-tuning updates a weight matrix W₀ (shape d_out × d_in) to W₀ + ΔW. Hu et al.'s hypothesis, borrowing from evidence that fine-tuning updates tend to have low "intrinsic rank," is that ΔW does not need full rank d to capture the adaptation a downstream task requires. So instead of learning a full d_out × d_in update, LoRA factorizes it as the product of two small matrices:

ΔW = B·A,   where A is (r × d_in), B is (d_out × r), and r ≪ min(d_in, d_out)

The forward pass for a linear layer becomes h = W₀x + (α/r)·B(Ax), where W₀ stays frozen and only A and B are trained. α is a fixed scaling constant (a hyperparameter, not learned) that lets you tune how strongly the low-rank path contributes without having to re-tune the learning rate every time you change r.

Parameter count for one d×d matrix with rank r: A contributes r×d_in, B contributes d_out×r, so total = r(d_in + d_out). When d_in = d_out = d, this is 2rd. For d = 4096, r = 8: 2 × 8 × 4096 = 65,536 parameters — versus 16,777,216 for the full d×d matrix. The compression ratio is d²/(2rd) = d/(2r) = 4096/16 = 256×. The original LoRA paper applies this only to the query and value projection matrices in self-attention (leaving keys and the feed-forward layers frozen, based on their ablations finding this the best accuracy-per-parameter tradeoff), so the per-layer trainable cost is 2 × 65,536 = 131,072, and across 32 layers: 131,072 × 32 = 4,194,304 — about 4.19 million parameters, or 4,194,304 / 7×10⁹ ≈ 0.06% of the model. Compare this to the adapter's 33.8 million: at these commonly used settings (m = 64 for adapters, r = 8 for LoRA), LoRA needs about 33,820,672 / 4,194,304 ≈ 8× fewer trainable parameters than bottleneck adapters, for comparable downstream accuracy in both papers' reported results.

Why B starts at zero and A does not

LoRA initializes A with small random values (e.g., a scaled Gaussian) and B with all zeros. This is not arbitrary. If both were initialized randomly, ΔW = BA would be nonzero at step 0, immediately perturbing a model that was already well-calibrated from pretraining, before a single gradient step of task-specific evidence has been seen — a worse, noisier starting point. If both were initialized to zero, training would stall: the gradient with respect to A is ∂L/∂A = Bᵀ·(upstream gradient), and since B = 0 that gradient is exactly zero, so A never moves and the layer is dead. Initializing only B to zero solves both problems at once: ΔW = BA = 0 at initialization (the model starts numerically identical to the frozen pretrained model, so training begins from a known-good point), while the gradient with respect to B, ∂L/∂B = (upstream gradient)·(Ax)ᵀ, is generally nonzero because A and x are both nonzero. So B moves away from zero on the very first backward pass; once B is nonzero, A's gradient (which depends on B) becomes nonzero too. The asymmetry is a one-step-delayed but fully functioning warm start, not a stalled parameter.

Diagram: parallel low-rank update versus serial bottleneck

Tracing LoRA's forward pass in code

The parameter math above should show up exactly when you count parameters in an actual module. Here is a minimal, self-contained LoRA-wrapped linear layer with the shapes traced step by step:

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, r=8, alpha=16):
        super().__init__()
        self.base = nn.Linear(in_features, out_features, bias=False)
        self.base.weight.requires_grad_(False)      # frozen pretrained weight
        self.A = nn.Parameter(torch.randn(r, in_features) * 0.01)  # random init
        self.B = nn.Parameter(torch.zeros(out_features, r))        # zero init
        self.scaling = alpha / r

    def forward(self, x):
        base_out = self.base(x)                     # (batch, out_features)
        lora_out = (x @ self.A.T) @ self.B.T          # (batch, out_features)
        return base_out + self.scaling * lora_out

layer = LoRALinear(in_features=4096, out_features=4096, r=8, alpha=16)
trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in layer.parameters() if not p.requires_grad)
print(trainable, frozen)

Trace it by hand before trusting the print statement. self.base is an nn.Linear(4096, 4096, bias=False), so its weight tensor has shape (4096, 4096) = 16,777,216 elements, and requires_grad_(False) makes every one of them frozen. self.A has shape (r=8, in_features=4096) → 8 × 4096 = 32,768 elements. self.B has shape (out_features=4096, r=8) → 4096 × 8 = 32,768 elements. Both A and B default to requires_grad=True as fresh nn.Parameter tensors, so trainable = 32,768 + 32,768 = 65,536, matching the hand-derived 2rd = 2 × 8 × 4096 exactly. print(trainable, frozen) outputs 65536 16777216. For the forward pass, if x has shape (batch, 4096): base_out is (batch, 4096); x @ self.A.T multiplies (batch, 4096) by (4096, 8), giving (batch, 8); that result times self.B.T, shape (8, 4096), gives (batch, 4096) — matching base_out's shape exactly, so the final addition is a plain elementwise sum with no broadcasting surprises.

Correcting a misconception: "PEFT always adds inference overhead"

A common misreading is that any parameter-efficient method — adapters, LoRA, doesn't matter — must permanently add a small extra computation to every forward pass, since you're literally adding new layers to the network. This is true for bottleneck adapters, but it is specifically false for LoRA after training, and the reason is the GELU nonlinearity sitting inside the adapter but not inside LoRA's update path. LoRA's contribution, (α/r)·BAx, is a purely linear function of x (a matrix product of matrices, with no nonlinearity between A and B), so it can be algebraically folded into the frozen weight before deployment: W' = W₀ + (α/r)BA. Once merged, the model is an ordinary dense network with exactly the same architecture, latency, and memory footprint as the original pretrained model — nothing marks it as having been LoRA-tuned. A bottleneck adapter cannot do this: its down-projection, GELU, and up-projection compose a nonlinear function, and nonlinear functions cannot be collapsed into a single frozen matrix. So the adapter's extra hop — small, but nonzero — is permanently in the critical path at inference time, for every request, forever. This is precisely why production LLM serving stacks (vLLM's multi-LoRA serving, Hugging Face's PEFT library) default to LoRA over adapters when latency matters: you get the adapter's parameter savings during training, plus the option of zero-overhead deployment that adapters structurally cannot offer.

That mergeability cuts both ways for the AI4Bharat scenario. If you are serving one language at a time, merge and deploy a plain dense model per language — 14 GB each, standard serving stack, zero LoRA-specific code at inference. If you need to serve all 22 languages from one GPU simultaneously (a realistic multi-tenant setting), keep A and B unmerged per language and swap them per incoming request: the frozen 14 GB base stays resident once, and each language adds only its own A, B pair. In fp16, one language's {A, B} pair at r=8 across the 32 query/value matrices is 4,194,304 parameters × 2 bytes = 8,388,608 bytes, exactly 8 MiB. Twenty-two languages' adapters together are about 22 × 8 MiB ≈ 176 MiB — compared with 22 × 14 GB ≈ 308 GB for 22 separately fine-tuned full copies. One frozen base plus a rack of megabyte-scale adapters is the entire economic argument for LoRA in a multilingual serving setting.

Extending the idea: QLoRA and quantized bases

Dettmers, Pagnoni, Holtzman, and Zettlemoyer's QLoRA (NeurIPS 2023) pushes the same idea one step further by quantizing the frozen base weights themselves, down to 4-bit precision using a scheme called NF4 (4-bit NormalFloat), with a second round of quantization applied to the quantization constants ("double quantization") to shave off further memory, plus a paged-optimizer mechanism that offloads memory spikes to CPU RAM. Only the LoRA A and B matrices are trained in full precision (typically bfloat16); the frozen base stays 4-bit throughout. At 4 bits per parameter, a 7B base model's weights shrink from 14 GB (fp16) to about 3.5 GB — small enough that training now fits comfortably on a single consumer GPU rather than requiring a multi-GPU node, without materially hurting task accuracy relative to full-precision fine-tuning, per QLoRA's reported results. This is the mechanism by which teams routinely fine-tune 65B-scale models on one 48 GB GPU, and the underlying update rule is exactly the ΔW = BA of this chapter — QLoRA changes only how the frozen W₀ is stored, not how the trainable correction is computed.

Active recall

Attempt each question before reading its answer.

Q1. For d = 4096 and rank r = 16 (instead of 8), how many trainable parameters does a single LoRA-adapted W_q matrix have?

Q2. If LoRA at r = 16 is applied to W_q and W_v across all 32 layers of the same 7B model, what is the total trainable parameter count, and what fraction of the model is that?

Q3. Starting from the original r = 8, q/v-only configuration (4,194,304 trainable parameters, 0.06% of the model), suppose you instead apply LoRA at r = 8 to all four attention projections (W_q, W_k, W_v, W_o, each 4096×4096) and both feed-forward matrices (an up-projection 4096→16384 and a down-projection 16384→4096) in every layer. Recompute the total trainable parameters and the fraction of the 7B model, and trace through what else changes as a result — optimizer memory, deploy-time merge cost, and overfitting risk.

Q4. True or false: "After training, A and B must always be shipped and loaded separately from the base model, exactly like a bottleneck adapter." Justify your answer using the AI4Bharat multilingual scenario.

Q5. Why does LoRA initialize B to all zeros while initializing A randomly, rather than initializing both to zero or both randomly?

Worked answers

A1. Trainable parameters = 2rd = 2 × 16 × 4096 = 131,072. This is exactly double the r = 8 case (65,536), because parameter count is linear in r for fixed d — doubling the rank doubles both A's and B's parameter counts.

A2. Per layer: 2 matrices (q, v) × 131,072 = 262,144. Across 32 layers: 262,144 × 32 = 8,388,608 trainable parameters. As a fraction of 7×10⁹: 8,388,608 / 7,000,000,000 ≈ 0.12% — exactly double the r = 8 fraction of 0.06%, consistent with A1's observation that everything scales linearly in r.

A3. Attention: each of the 4 matrices is square 4096×4096, so each costs 2rd = 2 × 8 × 4096 = 65,536; four matrices = 262,144 per layer. Feed-forward: for a rectangular (in, out) matrix, LoRA cost is r(in + out). Up-projection (4096→16384): 8 × (4096 + 16384) = 8 × 20,480 = 163,840. Down-projection (16384→4096): same dimension sum, also 163,840. Two FFN matrices = 327,680 per layer. Total per layer = 262,144 + 327,680 = 589,824. Across 32 layers: 589,824 × 32 = 18,874,368 trainable parameters — exactly 4.5× the original 4,194,304 (18,874,368 / 4,194,304 = 4.5). As a fraction of 7B: 18,874,368 / 7,000,000,000 ≈ 0.27%, up from 0.06%. Ripple effects: (i) optimizer memory scales with trainable parameter count, so the ~64 MiB of optimizer overhead in the original configuration becomes ~64 × 4.5 ≈ 288 MiB — still trivial next to the 112 GB a full fine-tune would need, so this stays a non-issue; (ii) merge cost at deployment is a one-time offline matrix addition per adapted matrix, done once before serving starts, not per inference request — going from 2 to 6 adapted matrices per layer (192 matrices total instead of 64) makes that offline step roughly 3× longer in wall-clock terms but changes nothing about serving latency, which stays zero-overhead either way; (iii) overfitting risk genuinely increases — 18.87M trainable parameters is meaningfully more capacity to memorize a small per-language fine-tuning set (often only a few thousand examples for lower-resource Indian languages), so this wider configuration would typically call for a lower rank, dropout on the LoRA path, or restricting the extra matrices to a subset of layers rather than applying them everywhere.

A4. False. Because ΔW = BA is purely linear, it can be merged into W₀ once training finishes: W' = W₀ + (α/r)BA. If you are deploying a single language, you merge and ship an ordinary dense checkpoint indistinguishable from a fully fine-tuned model, with A and B gone from the deployed artifact entirely. Keeping A and B unmerged and loaded separately is a choice you make only when you want to serve multiple languages from one resident frozen base — exactly the AI4Bharat case — trading a small per-request matmul for not needing 22 separate 14 GB checkpoints in memory. Separate loading is an option LoRA's structure enables, not a requirement it imposes.

A5. With B = 0 and A random: ΔW = BA = 0 at initialization, so the model starts numerically identical to the frozen pretrained model — no random perturbation degrades its calibrated behavior before training begins. Gradients still flow: ∂L/∂B = (upstream gradient)·(Ax)ᵀ is generally nonzero because A and x are nonzero, so B moves away from zero on the first backward pass; once B ≠ 0, ∂L/∂A = Bᵀ·(upstream gradient) becomes nonzero too, so A starts updating from the second step onward. If both A and B started at zero, ∂L/∂A would be identically zero forever (it depends on B), permanently stalling A. If both started random, ΔW would be nonzero from step 0, immediately and arbitrarily perturbing an already-good pretrained model before any task-specific gradient signal had been seen.

Think About It

Think about this: How would you explain parameter-efficient fine-tuning: adapters and lora to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where parameter-efficient fine-tuning: adapters and lora is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting parameter-efficient fine-tuning: adapters and lora to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind parameter-efficient fine-tuning: adapters and lora, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.