Fine-tuning and Instruction Tuning

Suppose a team is building a Hindi-language helpdesk assistant for a railway-booking service, the kind of system that has to answer "मेरा टिकट कन्फर्म हुआ या नहीं?" with a short, polite, correctly formatted reply. They download an open pretrained model such as Llama-2-7B or one of the Indic base models built on top of it, the kind of model AI4Bharat trains for Indian languages. They type in a query. The model does not answer it. It continues it: it appends a plausible-looking follow-up question, or drifts into a forum-style thread, or starts writing a second, unrelated query in the same style, because that is what next-token prediction on web text taught it to do when it sees a line that looks like the start of a conversation. The model is not broken. It is doing exactly what it was trained to do: predict the statistically likely continuation of a sequence. Answering a question politely, in one language, in a fixed format, was never the training objective. That gap between "predicts plausible continuations of internet text" and "follows an instruction the way a helpful assistant would" is precisely what fine-tuning, and specifically instruction tuning, exists to close.

Fine-tuning as a continuation of gradient descent, not a new algorithm

Pretraining and fine-tuning are the same procedure, gradient descent on a loss function computed from the model's own weights, run on different data with different objectives at different points in the model's life. Pretraining exposes a transformer to trillions of tokens of raw text with a single self-supervised objective: predict the next token, or in the case of masked models, predict a hidden token. This is cheap to supervise (the label is just "the next word", freely available in any text) which is what makes it possible to scale pretraining to enormous corpora. Fine-tuning takes those pretrained weights as an initialization, rather than random weights, and continues training on a much smaller, curated dataset built for a specific downstream distribution: legal documents, customer-support transcripts, or, in the case that matters most for modern LLMs, pairs of instructions and desired responses.

Three things distinguish fine-tuning from pretraining in practice. The learning rate is typically one to two orders of magnitude smaller (pretraining runs often use peak learning rates around 1 to 3 × 10^-4; full fine-tuning typically uses 1 to 5 × 10^-5), because the model already sits in a good region of the loss landscape and large steps risk destroying that. The dataset is orders of magnitude smaller, thousands to low millions of examples rather than trillions of tokens. And the objective can change: pretraining is almost always next-token prediction on unstructured text, while fine-tuning can supervise on structured input-output pairs, apply a loss only to the response tokens (masking the instruction so the model is not penalized for "predicting" text it was simply given), or, in reinforcement-learning-based tuning, use a reward signal instead of a token-level cross-entropy loss at all.

Full fine-tuning, and why it is expensive at scale

The most direct approach is full fine-tuning: unfreeze every parameter and backpropagate through the entire network on the new data. This works, and for smaller models or narrow domains it is often the right choice. The problem is memory, and it is worth deriving the number rather than asserting it.

Training a model with mixed-precision Adam requires storing more than just the weights. For every parameter you need, in the standard mixed-precision recipe: the fp16 (2-byte) working copy of the weight used for the forward and backward pass, the fp16 (2-byte) gradient, and, because Adam accumulates first and second moment estimates and mixed precision keeps a master copy of the weights in full precision for numerically stable updates, three additional fp32 (4-byte) tensors: the master weight, the momentum term m, and the variance term v. That is 2 + 2 + 4 + 4 + 4 = 16 bytes of state per trainable parameter, a figure that matches the model-state memory analysis in Rajbhandari et al.'s ZeRO paper (2020), which uses exactly this 16Ψ bytes accounting (Ψ being the parameter count) to motivate sharding optimizer state across GPUs.

Apply that to a 7-billion-parameter model, the scale of the original LLaMA-7B (Touvron et al., 2023):

params            = 7,000,000,000
bytes_per_param   = 16          # 2 (fp16 wt) + 2 (fp16 grad) + 4 (fp32 wt) + 4 (m) + 4 (v)
training_memory   = params * bytes_per_param
                  = 7e9 * 16
                  = 112,000,000,000 bytes
                  = 112 GB

112 GB of model state, before a single activation tensor is stored, does not fit on one 80 GB H100, let alone a 24 GB consumer GPU. Compare this to inference on the same model, which only needs the fp16 weights: 7e9 × 2 bytes = 14 GB, comfortable on a single high-end GPU. Full fine-tuning costs roughly eight times the memory of just running the model, which is why full fine-tuning of a multi-billion-parameter model normally requires multiple GPUs and a sharding framework (ZeRO or PyTorch's Fully Sharded Data Parallel), not because the model got bigger, but because training keeps five copies of state per parameter instead of one.

The misconception: fine-tuning does not mainly teach new facts

The natural way to picture fine-tuning is "the model reads a new textbook and learns new things," the same mental model as a student studying a new subject. This is misleading, and it is worth correcting explicitly because it leads to bad engineering decisions. A pretrained model in the multi-billion parameter range has already been exposed to a large fraction of the world's easily digitized text. Fine-tuning on a few thousand or even a few million examples is a rounding error against that exposure in terms of raw information volume. What a well-designed fine-tuning run changes is not primarily what the model knows, but how the model behaves: which of its already-learned capabilities it surfaces, in what format, under what conditions, and with what tone. Instruction tuning does not teach a 7B model arithmetic it never had; it teaches the model that when a user's message looks like a request, the correct behavior is to address that request directly, briefly, and in the requested format, instead of continuing it as if it were the middle of an unrelated document.

The practical consequence is that fine-tuning is a poor tool for injecting fresh, reliable, individually-checkable facts (today's train timings, a specific customer's account balance, a policy that changed last week). A model can be fine-tuned on a fact and still contradict it under a slightly rephrased query, or "forget" it after further training, because the fact is stored as a diffuse, imprecise adjustment to billions of weights rather than as a retrievable record. For that use case, retrieval-augmented generation, giving the model the fact as text in its context window at inference time, is more reliable than trying to bake the fact into the weights. Fine-tuning earns its keep on behavior, format, style, safety refusals, and task-following, which is exactly the instruction-following problem this chapter is about.

Catastrophic forgetting

The flip side of "fine-tuning reshapes behavior" is that it can reshape behavior you wanted to keep. Fine-tune too long, on too narrow a distribution, at too high a learning rate, and the model's general-purpose skills can degrade even as its performance on the fine-tuning task improves. This phenomenon predates deep learning by decades: McCloskey and Cohen (1989) named it catastrophic interference while studying simple connectionist networks that, after learning a second set of associations, abruptly lost their ability to perform the first. The same failure mode shows up in LLM fine-tuning: a model narrowly fine-tuned on, say, customer-support tickets for a single product can lose fluency at unrelated tasks it previously handled well, like multi-step arithmetic or code generation, because gradient updates optimized purely for the narrow objective are free to disturb any weight, including ones load-bearing for capabilities the narrow data never exercises.

Three standard mitigations follow directly from the mechanism. Lower the learning rate and use fewer epochs, so each update is a smaller perturbation to the pretrained weights. Mix general-purpose instruction data back into the fine-tuning set (a "replay" strategy), so the objective itself keeps rewarding retained general behavior, not only the narrow task. And prefer parameter-efficient fine-tuning, covered next, which by construction leaves the vast majority of the pretrained weights untouched, so there is structurally less surface area for forgetting to happen on.

Instruction tuning: supervised fine-tuning on (instruction, response) pairs

Instruction tuning is not a different training algorithm from fine-tuning; it is fine-tuning where the dataset is reshaped into natural-language instructions paired with the responses a helpful assistant should give, drawn from many different tasks at once, rather than one narrow task. Wei et al. (2022), in the FLAN paper "Finetuned Language Models Are Zero-Shot Learners," fine-tuned a 137-billion-parameter pretrained model on more than sixty NLP datasets, each rewritten as natural-language instruction templates and grouped into task clusters, and then evaluated the model's zero-shot performance on clusters that were held out entirely during fine-tuning. The result that made the paper influential: instruction tuning on a diverse set of tasks improved zero-shot performance on tasks the model had never seen an instruction for during fine-tuning, evidence that the model was learning a general skill, "read an instruction and do what it says," rather than memorizing sixty separate task formats. Chung et al. (2022) extended this in "Scaling Instruction-Finetuned Language Models" (the Flan-T5/Flan-PaLM line), scaling the instruction collection up to over a thousand tasks and showing that both the number of fine-tuning tasks and the base model's size independently improve downstream generalization, and that mixing in chain-of-thought exemplars during instruction tuning specifically improves multi-step reasoning at inference time.

OpenAI's InstructGPT paper (Ouyang et al., 2022, "Training language models to follow instructions with human feedback") is the version of this idea that shaped ChatGPT-era systems, and it makes the "behavior, not facts" point concrete. It used a three-stage pipeline on top of pretrained GPT-3: supervised fine-tuning (SFT) on roughly thirteen thousand human-written demonstrations of instruction-following, then a reward model trained on human rankings of multiple model outputs for the same prompt, then reinforcement learning (PPO) against that reward model. Instruction tuning is specifically that first stage, the supervised one; the paper's most quoted finding is that human evaluators preferred outputs from the 1.3-billion-parameter InstructGPT model, a model over a hundred times smaller, to outputs from the 175-billion-parameter base GPT-3. Raw scale bought GPT-3 more latent capability; instruction tuning was what made a small fraction of that capability reliably accessible through a plain-language request. That single result is the cleanest evidence available that instruction tuning's main effect is behavioral, since no amount of supervised fine-tuning on thirteen thousand examples could plausibly out-teach GPT-3's pretraining corpus on raw world knowledge.

Parameter-efficient fine-tuning: LoRA

Full fine-tuning's 16-bytes-per-parameter memory cost makes it impractical to fine-tune the largest models on modest hardware, and it also fine-tunes far more capacity than most tasks need. Hu et al. (2021/2022), in "LoRA: Low-Rank Adaptation of Large Language Models," proposed freezing the entire pretrained weight matrix and learning only a small additive update, factored as the product of two low-rank matrices, injected alongside it.

Take a single weight matrix inside a transformer's attention block, W ∈ ℝ^d×d, mapping a hidden state to a query, key, or value projection. LoRA freezes W entirely and introduces two new trainable matrices, A ∈ ℝ^r×d (a down-projection to a small rank r) and B ∈ ℝ^d×r (an up-projection back to the original dimension), where r ≪ d. The forward pass becomes:

h = W @ x + (alpha / r) * (B @ (A @ x))

with alpha a fixed scaling constant. Only A and B receive gradients; W stays exactly as pretraining left it. At the start of training, B is initialized to all zeros (and A to small random values), so the LoRA term contributes nothing and the fine-tuned model is initially identical to the pretrained one, a useful safety property: fine-tuning starts from a known-good behavior and moves away from it only as far as gradient descent pushes it.

Worked example: how much does this actually save?

Take a realistic attention projection with the hidden dimension of a 7B-class model, d = 4096, and a common LoRA rank, r = 8.

full_params     = d * d
                = 4096 * 4096
                = 16,777,216

lora_A_params   = r * d = 8 * 4096 = 32,768
lora_B_params   = d * r = 4096 * 8 = 32,768
lora_params     = lora_A_params + lora_B_params
                = 65,536

reduction       = full_params / lora_params
                = 16,777,216 / 65,536
                = 256

Fine-tuning this one matrix with LoRA trains 256 times fewer parameters than fine-tuning it directly. This is not specific to one matrix: the general formula falls straight out of the arithmetic above, since full_params = d² and lora_params = 2rd, so the reduction factor is d² / 2rd = d / 2r. Checking: 4096 / (2 × 8) = 4096 / 16 = 256, which matches the direct computation exactly.

Extend this to a full model in the style of LoRA's original experiments, which applied the technique only to the query and value projections in every attention layer. A 32-layer, d = 4096 model, LoRA on W_q and W_v in each layer, r = 8:

matrices_per_layer = 2                      # W_q and W_v
params_per_matrix   = 65,536                # from above
layers               = 32

total_lora_params = matrices_per_layer * params_per_matrix * layers
                  = 2 * 65,536 * 32
                  = 4,194,304                # ~4.19 million

# against a 7-billion-parameter base model:
fraction_trainable = 4,194,304 / 7,000,000,000
                    = 0.0006   (0.06%)

Roughly 0.06% of the model's parameters are trained; everything else is loaded once and left untouched. Since Adam's optimizer state only has to be kept for trainable parameters, the 16-bytes-per-parameter cost that made full fine-tuning need 112 GB now applies to about 4.19 million parameters instead of 7 billion: 4,194,304 × 16 bytes ≈ 67 MB of optimizer state, against 14 GB just to hold the frozen fp16 weights in memory for the forward and backward pass. Total memory is now dominated by the frozen weights and activations, not by training state, which is why LoRA fine-tuning of a 7B model fits comfortably on a single 24 GB GPU where full fine-tuning could not. Hu et al. report a similar order of magnitude at GPT-3's scale: applying LoRA reduced trainable parameters by roughly 10,000 times and GPU memory requirements by roughly 3 times relative to full fine-tuning with Adam. Dettmers et al.'s QLoRA (2023) pushes this further by storing the frozen base weights themselves in a 4-bit quantized format (their 4-bit NormalFloat, NF4) while keeping the LoRA adapters in higher precision, which the paper uses to fine-tune a 65-billion-parameter model on a single 48 GB GPU without losing accuracy relative to full 16-bit fine-tuning.

Tracing the computation by hand

Formulas are easy to nod along to and easy to implement wrong. Trace one concrete forward pass through a LoRA layer with small enough numbers to check by hand, using rank r = 1 (real deployments typically use r between 4 and 64; r = 1 is chosen here purely so every multiplication fits on one line) and d = 4.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base_weight, r, alpha):
        super().__init__()
        d_out, d_in = base_weight.shape
        self.W = nn.Parameter(base_weight.clone(), requires_grad=False)  # frozen
        self.A = nn.Parameter(torch.zeros(r, d_in))
        self.B = nn.Parameter(torch.zeros(d_out, r))
        self.scale = alpha / r

    def forward(self, x):
        frozen_out = self.W @ x
        lora_out = self.B @ (self.A @ x)
        return frozen_out + self.scale * lora_out

W = torch.tensor([[1., 0., 0., 1.],
                   [0., 1., 1., 0.],
                   [1., 1., 0., 0.],
                   [0., 0., 1., 1.]])

layer = LoRALinear(W, r=1, alpha=2.0)
with torch.no_grad():
    layer.A.copy_(torch.tensor([[1., -1., 1., -1.]]))
    layer.B.copy_(torch.tensor([[1.], [0.], [-1.], [2.]]))

x = torch.tensor([1., 0., 1., 0.])
h = layer(x)
print(h)

Trace it by hand, term by term. First the frozen path, W @ x: row 1 of W is [1,0,0,1], dotted with x=[1,0,1,0] gives 1·1+0·0+0·1+1·0 = 1. Row 2, [0,1,1,0]·[1,0,1,0] = 0+0+1+0 = 1. Row 3, [1,1,0,0]·[1,0,1,0] = 1+0+0+0 = 1. Row 4, [0,0,1,1]·[1,0,1,0] = 0+0+1+0 = 1. So W @ x = [1, 1, 1, 1].

Now the adapter path. A @ x: A is the single row [1,-1,1,-1], dotted with x=[1,0,1,0]: 1·1 + (-1)·0 + 1·1 + (-1)·0 = 1 + 1 = 2. That scalar, 2, is the compressed r-dimensional (here r=1) representation. Then B @ (A @ x): B = [1, 0, -1, 2]ᵀ scaled by that scalar 2 gives [2, 0, -2, 4]. Finally the scale factor alpha / r = 2 / 1 = 2 multiplies this vector: 2 × [2, 0, -2, 4] = [4, 0, -4, 8].

Adding the two paths, h = [1,1,1,1] + [4,0,-4,8] = [5, 1, -3, 9]. That tensor, printed as tensor([ 5., 1., -3., 9.], grad_fn=<AddBackward0>), is what the code above actually prints: PyTorch right-pads the positive entries with a leading space so they line up with the minus sign in -3, and a grad_fn is attached because self.A and self.B are declared as nn.Parameter(...) with the default requires_grad=True and the forward call is not wrapped in torch.no_grad(), so the output stays attached to the autograd graph even though the frozen W itself, having requires_grad=False, contributes to the output but never receives a gradient during backward().

The trainable parameter count for this toy layer is |A| + |B| = 4 + 4 = 8, against |W| = 16, a reduction factor of d/(2r) = 4/2 = 2, consistent with the general formula, just less dramatic than the 256× case above because d is small here.

Active recall

Attempt these before reading the answers below.

1. Why does instruction tuning improve a model's zero-shot performance on tasks it never saw an instruction for during fine-tuning, given that the base model was already pretrained on trillions of tokens that surely contained many examples of instructions and answers?
2. A team applies LoRA with r = 16 to the W_v matrix only, in every one of 40 transformer layers, on a model with d = 4096. Compute the number of trainable parameters, and compare it to fully fine-tuning just those 40 W_v matrices.
3. A team fine-tunes a general-purpose assistant model for 10 epochs on a narrow internal ticketing dataset at a relatively high learning rate. It becomes excellent at ticket triage but starts failing basic arithmetic it previously handled correctly. Name the phenomenon and give two concrete mitigations.
4. Take the worked LoRA hand-trace above (W, A, B, x, alpha=2, r=1). Suppose the team instead sets r = 2, keeping alpha = 2, with the new A = [[1,-1,1,-1],[0,1,0,1]] and new B with columns [1,0,-1,2] and [1,1,0,0]. Recompute everything that changes: the shapes of A and B, the trainable parameter count, the scale factor, and the final output h.
5. A colleague argues: "We fine-tuned our model on this week's exam timetable, so now it reliably knows the timetable." What is wrong with this framing, and what would you recommend instead?
6. Why does full fine-tuning of a 7B model need roughly 112 GB of memory for weights, gradients, and optimizer state, while running that same model for inference needs only about 14 GB?

Answers.

1. Pretraining text does contain instructions and answers, but scattered across a huge diversity of surface forms, and the pretraining objective never explicitly rewards "recognize this as an instruction, then comply with it" as a general behavior; it only rewards predicting the next token, which for instruction-shaped text in the wild is just as often a continuation, a critique, or a different question as it is a direct answer. Instruction tuning takes many different tasks, all reformatted into the same instruction-then-response shape, and trains directly on that shape. Wei et al. (2022) show that fine-tuning on a large, diverse set of such reformatted tasks generalizes to unseen task clusters, evidence the model is learning the general pattern "map instruction to compliant response" rather than memorizing per-task answers, because held-out clusters could not have been memorized.

2. Per matrix, LoRA trainable params = r·d + d·r = 2rd = 2 × 16 × 4096 = 131,072. Across 40 layers, applied to one matrix per layer: 131,072 × 40 = 5,242,880 (about 5.24 million). Full fine-tuning of the same 40 matrices: each is d×d = 4096×4096 = 16,777,216; across 40 layers, 16,777,216 × 40 = 671,088,640 (about 671 million). Ratio: 671,088,640 / 5,242,880 = 128, matching d/(2r) = 4096/32 = 128 exactly, since the per-layer/per-matrix reduction factor is scale-invariant to how many layers you sum over.

3. This is catastrophic forgetting (McCloskey and Cohen, 1989): narrow, aggressive fine-tuning overwrote weights that were load-bearing for a skill (arithmetic) the narrow ticketing data never exercised or rewarded. Mitigations: lower the learning rate and/or reduce epochs so each update perturbs the pretrained weights less; mix general-purpose instruction data back into the fine-tuning set so the training objective itself keeps crediting retained general behavior; or switch to LoRA-style parameter-efficient fine-tuning, which structurally leaves the base weights (and hence the arithmetic-relevant circuitry sitting in them) untouched.

4. Shapes: A goes from 1×4 to 2×4 (8 params), B goes from 4×1 to 4×2 (8 params); trainable total becomes 8+8 = 16, equal to |W| = 16, so the reduction factor collapses to d/(2r) = 4/4 = 1, meaning this rank choice saves nothing over full fine-tuning of this matrix. The scale factor changes too, and it is easy to miss: scale = alpha/r = 2/2 = 1, not 2 anymore. Computing A@x with the new rows: row 1 gives 1·1+(-1)·0+1·1+(-1)·0 = 2 as before; row 2 gives 0·1+1·0+0·1+1·0 = 0. So A@x = [2, 0]. Then B@(A@x), with B's rows [1,1],[0,1],[-1,0],[2,0]: row 1: 1·2+1·0=2; row 2: 0·2+1·0=0; row 3: -1·2+0·0=-2; row 4: 2·2+0·0=4. So B@(A@x) = [2,0,-2,4], the same vector as the r=1 case, because the new second column of B never gets exercised (Ax's second component is 0). Applying the new scale of 1 (not 2): scaled = [2,0,-2,4]. Final h = Wx + scaled = [1,1,1,1] + [2,0,-2,4] = [3, 1, -1, 5], different from the r=1 answer of [5,1,-3,9] even though the raw B@(A@x) vector happened to coincide, purely because the scale factor changed when r changed.

5. Fine-tuning mostly reshapes behavior and elicits latent capability; it is not a reliable way to store individually checkable, frequently changing facts like a weekly timetable, because the fact is smeared across a diffuse weight update rather than kept as a retrievable record, and it can be contradicted by rephrasing or degraded by further training. The better fix is retrieval-augmented generation: keep the timetable in a database or document store and insert the relevant rows into the model's context at query time, so the model reads the current fact rather than trying to recall a fine-tuned approximation of it.

6. Mixed-precision Adam full fine-tuning keeps five tensors per parameter (fp16 weight, fp16 gradient, fp32 master weight, fp32 momentum, fp32 variance) totalling 16 bytes/parameter, so 7e9 × 16 = 112 GB. Inference needs only the fp16 weights to run the forward pass, 7e9 × 2 bytes = 14 GB; there is no gradient or optimizer state to keep because no weights are being updated. The 8× gap (112/14 = 8) is exactly the ratio of "state kept for training" to "state kept for serving," which is why fine-tuning infrastructure for large models is a materially bigger lift than serving infrastructure for the same model.

Think About It

Think about this: How would you explain fine-tuning and instruction tuning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind fine-tuning and instruction tuning, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.