Large Language Model Fine-tuning and LoRA

An Indian ed-tech platform preparing students for JEE and NEET wants four subject-specialist tutoring assistants: Physics, Chemistry, Mathematics, Biology, each fine-tuned on that subject's problem sets, common misconceptions, and answer-explanation style. The obvious approach is to fully fine-tune an open 7-billion-parameter model four times and deploy four separate copies. Before a single student sends a query, that plan already needs four full model weight sets in memory to serve concurrently, and it needed four times the GPU memory just to train them. Every additional subject multiplies both numbers again. This chapter is about the two techniques a production team actually reaches for to escape this arithmetic — shrinking the trainable footprint of fine-tuning until it is a rounding error next to the frozen base, and quantizing the frozen base itself — and about what happens once you need to serve many such fine-tuned personalities off one shared copy of the underlying model at once.

Recap: what LoRA freezes and what it trains

You have already seen the core mechanism: instead of updating a weight matrix W directly, Low-Rank Adaptation freezes W and learns a correction as the product of two much smaller matrices, B and A, so the effective weight during the forward pass becomes W + BA. This chapter does not re-derive that decomposition. It builds on it to answer the questions a team actually hits once LoRA leaves the notebook and has to run on real GPUs, serve real traffic, and coexist with three other fine-tuned personalities on the same box: exactly how much memory does this save and where does the saving come from, how do you shrink the frozen base itself without breaking training, and how do you serve many separately fine-tuned adapters at once without paying for many separate models.

The real memory ledger: full fine-tuning vs LoRA vs QLoRA

Take a 7-billion-parameter transformer trained with mixed-precision Adam — the standard recipe used for Llama-2-7B and similar models (Touvron et al., 2023, "Llama 2: Open Foundation and Fine-Tuned Chat Models"). For every parameter that is actually being trained, mixed-precision Adam needs to keep, per parameter:

bf16 weight              2 bytes
bf16 gradient             2 bytes
fp32 master weight        4 bytes   (kept for numerically stable updates)
fp32 Adam first moment    4 bytes   (m)
fp32 Adam second moment   4 bytes   (v)
--------------------------------
total                    16 bytes / trainable parameter

For full fine-tuning, every one of the 7 billion parameters is trainable, so the weight-and-optimizer memory alone is 7×10⁹ × 16 bytes = 1.12×10¹¹ bytes = 112 GB — before a single activation is stored. That does not fit on one 80 GB A100 or H100, which is exactly why full fine-tuning a 7B model routinely needs multi-GPU sharding (ZeRO, FSDP) even for a model that would run comfortably for inference on a single consumer card.

Now do the same ledger for LoRA applied to the query and value projections only, with rank r = 8, on a 32-layer, d = 4096 architecture (Llama-2-7B's dimensions; the 7B model uses ordinary multi-head attention, not grouped-query attention, which Llama 2 reserves for its 34B and 70B variants). The frozen base weights need none of the 14 extra bytes above — they never receive a gradient or an optimizer update, so they sit in memory as bf16 storage only:

Frozen base (bf16, no grad/optimizer): 7×10⁹ × 2 bytes  = 14.00 GB

The only parameters that need the full 16-byte treatment are the LoRA matrices themselves. Each targeted matrix is [4096, 4096]; LoRA adds B ∈ ℝ^{4096×8} and A ∈ ℝ^{8×4096}, contributing 8×4096 + 4096×8 = 65,536 trainable parameters. Two matrices per layer (query and value) across 32 layers gives:

65,536 params/matrix × 2 matrices/layer × 32 layers = 4,194,304 trainable params (≈4.19M)
4,194,304 × 16 bytes = 67,108,864 bytes ≈ 0.067 GB

Total for LoRA on a bf16 base: 14.00 + 0.067 ≈ 14.07 GB — an 87% reduction from 112 GB, and the trainable-parameter count is 4.19M against 7 billion, or 0.06% of the model. Quantizing the frozen base itself (QLoRA, covered next) drops the 14.00 GB frozen-weight term to roughly 3.6 GB, bringing the total to about 3.7 GB — small enough to fine-tune a 7B model on a single 8–12 GB consumer GPU, which is precisely the claim in Dettmers, Pagnoni, Holtzman, and Zettlemoyer's 2023 paper "QLoRA: Efficient Finetuning of Quantized LLMs" (NeurIPS 2023).

Quantizing the frozen weights: NF4 and double quantization

QLoRA's contribution is orthogonal to LoRA's parameter-count reduction — it attacks the other large term in the ledger, the 14 GB of frozen base weights, by storing them in 4-bit NormalFloat (NF4) instead of bf16. NF4 is a quantization data type built so that its 16 representable values are spaced to match the quantiles of a zero-centered normal distribution, which is a much better fit for pretrained transformer weights than a uniform 4-bit grid, because most weight values cluster near zero. During the forward and backward pass, each 4-bit block is dequantized back to bf16 on the fly for the actual matrix multiply — the compute happens in bf16, only the storage is 4-bit. This matters for the misconception below: nothing about training is done "in 4-bit."

Quantization is applied in blocks (QLoRA uses a block size of 64 weights) and each block needs its own scaling constant (an "absmax") to map the block's actual value range onto the 16 NF4 levels. Naively, that constant is stored in fp32:

Single-level NF4:      4 bits/weight + 32 bits / 64 weights = 4 + 0.5   = 4.5   bits/param

Double quantization treats those per-block fp32 constants as data worth quantizing too: it 8-bit-quantizes them in a second pass, using a coarser block size of 256 constants per second-level block:

Double-quantized:      4 bits/weight + 8 bits/64 weights + 32 bits/(64×256)
                      = 4 + 0.125 + 0.001953
                      ≈ 4.127 bits/param
Savings over single-level:  4.5 − 4.127 = 0.373 bits/param

That 0.373-bit-per-parameter savings is the exact figure the QLoRA paper reports, and deriving it from block sizes alone (rather than just quoting it) is worth doing once so it stops looking like a magic constant. Converting the double-quantized rate to bytes: 4.127 / 8 = 0.5159 bytes/param, so the frozen base of a 7B model costs 7×10⁹ × 0.5159 ≈ 3.61 GB, matching the ~3.6 GB used in the ledger above.

Paged optimizers: surviving the memory spikes

Even after LoRA and NF4 shrink the steady-state memory footprint to a few gigabytes, GPU memory usage during training is not flat — it spikes momentarily whenever a long sequence in a batch forces a larger-than-usual activation buffer, or whenever gradient checkpointing recomputes an unusually deep segment of the graph. On a small consumer GPU, a spike that would have been comfortably absorbed on an 80 GB card can trigger an out-of-memory crash even though the average usage is well under budget. QLoRA borrows the operating-system idea of paging: it allocates optimizer state in NVIDIA's unified memory, which lets pages of that state be evicted to CPU RAM automatically when the GPU is under pressure and paged back in when needed, rather than the process crashing outright. It costs a little latency only during the rare spike, and nothing in the common case — which is exactly the deal an OS makes with virtual memory and disk swap.

Choosing what to adapt: targets and rank

Rank and target-module choice trade trainable capacity for memory and merge cost in a way that is easy to get wrong by intuition, so it is worth computing rather than guessing. The general formula for the trainable parameters LoRA adds to one matrix of shape [d_out, d_in] at rank r is r × (d_in + d_out) — this is rank × d_out for B plus rank × d_in for A. A small calculator makes it easy to compare configurations without re-deriving the arithmetic each time:

def lora_trainable_params(layers, targets, rank, d_model=4096, d_ffn=11008):
    """
    targets: any subset of {"q_proj","k_proj","v_proj","o_proj","down_proj"}.
    Shapes assumed for Llama-2-7B: q/k/v/o_proj are [4096, 4096];
    down_proj is [4096, 11008] (MLP intermediate size 11008).
    Returns total trainable LoRA parameters across all layers.
    """
    shapes = {
        "q_proj": (d_model, d_model),
        "k_proj": (d_model, d_model),
        "v_proj": (d_model, d_model),
        "o_proj": (d_model, d_model),
        "down_proj": (d_model, d_ffn),
    }
    per_layer = 0
    for name in targets:
        d_out, d_in = shapes[name]
        per_layer += rank * (d_in + d_out)
    return per_layer * layers

baseline = lora_trainable_params(32, ["q_proj", "v_proj"], rank=8)
expanded = lora_trainable_params(32, ["q_proj", "v_proj", "o_proj", "down_proj"], rank=16)
print(baseline, expanded, expanded / baseline)

Tracing this by hand: baseline matches the 4,194,304 computed in the memory ledger above. For expanded, q/v/o at rank 16 each contribute 16×(4096+4096)=131,072 per layer (393,216 for all three), and down_proj contributes 16×(4096+11008)=241,664 per layer; summed and multiplied by 32 layers gives 20,316,160. The printed ratio is exact in binary floating point because it reduces to 155/32: the program prints 4194304 20316160 4.84375.

Two things follow from this. First, going from 2 targeted matrices at rank 8 to 4 matrices at rank 16 does not simply double capacity twice (to 4×) — it lands at 4.84×, because down_proj's rectangular shape (4096×11008) adds more per-rank parameters than a square 4096×4096 matrix does. Second, even at the larger configuration, 20.3 million trainable parameters is still 0.29% of a 7B model — LoRA's rank and target choices move the trainable count by single-digit multiples, never by orders of magnitude, which is the entire point of the technique. A related line of work, DoRA (Liu et al., 2024, "DoRA: Weight-Decomposed Low-Rank Adaptation," ICML 2024), changes what gets adapted rather than how many parameters are used: it decomposes each frozen weight matrix into a magnitude vector and a direction matrix, trains the magnitude directly — a full-rank update, since it is only a vector of size d_out and cheap to train outright — and applies the LoRA-style low-rank update to the direction component on top of a frozen base direction, which the authors report closes more of the gap to full fine-tuning at matched rank without adding any inference-time cost — a different mechanism from picking targets or rank, but one that composes with everything in this section.

Merging adapters vs keeping them separate

Once training finishes, there are two ways to deploy a LoRA adapter. Merging computes W' = W + BA once, offline, and replaces the frozen weight with the result — inference then runs on an ordinary dense matrix of exactly the original shape, with zero extra latency or memory versus the un-adapted model. Keeping the adapter separate means every forward pass computes xW + x(BA) as two matrix multiplies instead of one, which is slightly slower per request but lets a single deployed base model serve many different adapters by swapping which B, A pair it multiplies against per request.

Merge cost itself is cheap and one-time: computing B (d_out×r) @ A (r×d_in) costs 2 × d_out × r × d_in floating-point operations (multiply-add counted as two ops). For the baseline configuration (q, v at rank 8), each matrix costs 2×4096×8×4096 = 268,435,456 FLOPs; two matrices per layer across 32 layers totals 17.18 GFLOPs. For the expanded configuration (q, v, o at rank 16, plus down_proj at rank 16, d_in=11008), the square matrices cost 2×4096×16×4096 ≈ 537 MFLOPs each and down_proj costs 2×4096×16×11008 ≈ 1.44 GFLOPs; summed and multiplied by 32 layers, total merge cost is ≈97.71 GFLOPs. The ratio between the two merge costs is 97.71/17.18 ≈ 5.7× — noticeably more than the 4.84× parameter-count ratio computed above, because down_proj's rectangular shape contributes disproportionately more FLOPs per trainable parameter than it contributes trainable parameters. Either way, on a GPU delivering on the order of 10¹⁴ FLOPs/second, both merges finish in well under a millisecond of raw compute — merging is never the bottleneck; the decision of whether to merge is entirely about whether you need to swap adapters at request time.

Serving many adapters at once: the multi-tenant problem

This is exactly the situation the JEE tutoring platform is in: Physics, Chemistry, Mathematics, and Biology adapters, each trained separately, need to serve concurrent traffic from one deployment. Merging each adapter into its own copy of the base model would need 4 × 14 GB ≈ 56 GB of bf16 weights resident at once (or four separate GPUs), which defeats the memory savings LoRA was supposed to deliver at inference time. The alternative — used in systems like S-LoRA (Sheng et al., "S-LoRA: Serving Thousands of Concurrent LoRA Adapters," MLSys 2024) — keeps exactly one frozen, possibly quantized, copy of the base weights resident, and keeps every tenant's adapter matrices as small separate objects in GPU memory. A batch of incoming requests, even when each request needs a different adapter, is served with the shared xW term computed once per batch and each request's x(BᵢAᵢ) correction added in, using batched, grouped matrix multiplication kernels designed to handle adapters of different ranks in the same batch efficiently. The panel below shows this structure directly.

Notice what makes this batching possible at all: because every request shares the same xW term, the expensive part of the computation (the full-rank matrix multiply against a 4096-dimensional weight) is done once per batch regardless of how many different adapters are represented in it. Only the small, per-tenant x(BᵢAᵢ) correction differs — and at rank 8, that correction is roughly two orders of magnitude cheaper than the shared term. This is why a platform can go from four subjects to forty without needing forty times the GPU memory: the frozen base is paid for once, and each additional subject costs on the order of tens of megabytes, not gigabytes.

Correcting a common misconception

Students who have just learned that LoRA trains 0.06% of a model's parameters often conclude that fine-tuning must therefore run roughly 1,600× faster too, since gradients are computed for so much less. That is not right, and the ledger above shows exactly why. Forward-pass compute is unchanged: every activation still has to flow through every frozen layer to reach the output, whether or not that layer's weights are being updated. Backward-pass compute drops only partially — for a frozen matrix, you skip computing the weight gradient dL/dW, but you still have to compute the input gradient dL/dx so that error signal can keep flowing backward to earlier LoRA adapters. Skipping one of two backward matmuls per frozen layer is a real but modest saving, nowhere near 1,600×. The overwhelming majority of the memory reduction — and essentially none of the compute-time reduction — comes from a completely different place: 7 billion frozen parameters no longer need the extra 14 bytes each (fp32 master weight plus two fp32 Adam moments) that full fine-tuning requires for every trainable parameter. That single elimination is what turns a 112 GB training footprint into 14 GB. LoRA is primarily a memory optimization with a secondary, smaller compute benefit — not a training-time optimization with memory as a side effect.

Active recall

Attempt each question before reading its answer.

1. After merging a LoRA adapter into a base model (W' = W + BA), is inference on the merged model slower, faster, or the same speed as inference on the original un-adapted model?
2. LoRA shrinks trainable-parameter memory; QLoRA shrinks frozen-weight memory. Why do these two savings add up rather than overlap or compete?
3. Using the formula r × (d_in + d_out) per matrix, compute the trainable parameter count for LoRA applied to v_proj only, across all 32 layers of Llama-2-7B, at rank r = 8. Express the result as a percentage of the model's 7 billion total parameters.
4. The JEE platform wants to add a fifth subject (Biology) by keeping each subject's adapter merged into its own full copy of the base model, rather than serving adapters unmerged off a shared base. What GPU memory cost does this incur per additional subject, and how does that compare to the unmerged, shared-base approach?
5. Ripple question. Starting from the original configuration (LoRA on q_proj and v_proj only, rank 8), the team now (a) adds o_proj and down_proj as targets and (b) raises rank from 8 to 16. Trace the effect on: trainable parameter count, trainable-parameter memory footprint, whether the adapted model still fits on a 16 GB GPU alongside a QLoRA-quantized base, and one-time merge compute cost.
6. True or false, with correction: "QLoRA fine-tunes the model directly in 4-bit precision, so the gradients themselves are computed in 4-bit."

Answers.

1. The same speed. Once merged, W' has exactly the same shape and dtype as the original W, so the forward pass is one ordinary dense matrix multiply — identical cost to the un-adapted model. The extra compute only exists in the unmerged configuration, where inference computes xW + x(BA) as two separate matmuls to keep the adapter swappable per request.

2. Because they target different terms in the memory ledger. LoRA's saving comes from shrinking the trainable-parameter count, which shrinks the 16-bytes-per-trainable-parameter (gradient + optimizer state) term. QLoRA's saving comes from shrinking the storage cost of the frozen weights themselves, from 2 bytes/param (bf16) to about 0.516 bytes/param (double-quantized NF4). Frozen weights never carry gradient or optimizer-state cost regardless of quantization, and trainable LoRA matrices are stored at full precision regardless of how the frozen base is quantized — the two techniques touch disjoint parameters and disjoint memory categories, so their savings are additive: 14.00 GB → 3.61 GB for the base, plus 0.067 GB for the adapter either way.

3. One matrix (v_proj, d_in = d_out = 4096) per layer, rank 8: 8 × (4096+4096) = 65,536 per layer. Across 32 layers: 65,536 × 32 = 2,097,152 (≈2.10M) trainable parameters. As a percentage of 7 billion: 2,097,152 / 7×10⁹ ≈ 0.030%.

4. Merging each subject's adapter into its own full bf16 copy costs 14.00 GB of GPU memory per additional subject (the full frozen-base weight, since merging destroys the shared-base property — each merged copy is now an independent dense model). Five subjects this way costs 5 × 14.00 = 70 GB. Serving adapters unmerged off one shared base costs 14.00 GB once (or ≈3.6 GB if that shared base is QLoRA-quantized) plus ≈0.067 GB per additional subject's adapter — five subjects costs roughly 3.6 + 5×0.067 ≈ 3.94 GB, about eighteen times less memory than the merged-copies approach, and the gap widens with every subject added.

5. Parameter count: the calculator from the "Choosing what to adapt" section gives exactly this comparison: baseline (q, v at r=8) is 4,194,304; expanded (q, v, o, down_proj at r=16) is 20,316,160 — a 4.84375× increase, larger than either the 2× from doubled rank or the 2× from doubled matrix count alone, because down_proj's rectangular 4096×11008 shape adds disproportionately many parameters per unit of rank. Trainable-parameter memory: at 16 bytes/trainable-parameter, this rises from 67 MB to 20,316,160 × 16 ≈ 325 MB — still tiny in absolute terms. Fits on 16 GB? Yes, comfortably: a QLoRA base costs ≈3.6 GB, the expanded adapter's trainable memory costs ≈0.33 GB, leaving well over 10 GB of headroom for activations even without aggressive gradient checkpointing — the configuration change moved the total by a few hundred megabytes, nowhere near the 16 GB ceiling. Merge compute: this is the one place the ripple is proportionally larger than the parameter-count ripple — merge FLOPs rise from ≈17.18 GFLOPs to ≈97.71 GFLOPs, a 5.7× increase, again disproportionately driven by down_proj's asymmetric shape. Even so, 97.71 GFLOPs is roughly a millisecond of compute on a modern accelerator and happens once after training completes — it never appears in the per-request latency budget.

6. False. The frozen base weights are stored in 4-bit NF4, but every forward and backward matrix multiply dequantizes the relevant block back to bf16 before the multiply happens — the arithmetic is done in bf16, not 4-bit. The LoRA adapter matrices (B and A) are never quantized at all; they are stored and updated at bf16/fp32 precision throughout training, exactly as in ordinary LoRA. Quantization in QLoRA is a storage-format decision for the frozen weights only, not a computation-precision decision for training.

Think About It

Think about this: How would you explain large language model fine-tuning and lora to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where large language model fine-tuning and lora is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting large language model fine-tuning and lora to at least 3 other topics you have studied.