Instruction Tuning: Making Models Follow Directives

When a language model won't do what you ask

Picture a Bengaluru ed-tech startup fine-tuning an open base checkpoint — a raw, freshly pretrained transformer, nothing else done to it — to become a JEE doubt-solving bot. A student types: "Solve: If f(x) = x² + 3x, find f′(2)." A properly instruction-tuned assistant answers directly: differentiate to get f′(x) = 2x + 3, substitute x = 2, so f′(2) = 7. But hand that exact same prompt to the base model — before any instruction tuning — and something stranger often happens: it continues with "Solve: If g(x) = 2x³ − x, find g′(1). Solve: If h(x) = 5x², find h′(3)." It generates more practice problems instead of solving the one it was given.

This is not a bug in the sampling code. It is the base model doing exactly what it was trained to do: predict the statistically most likely continuation of the text it has seen so far. On the open web, and in the scraped textbooks and question banks that feed pretraining corpora, a line that looks like "Solve: ..." is very often followed by another line that looks like "Solve: ..." — because problem sets, exam papers, and practice sheets are exactly that shape. The base model has learned the surface statistics of "what tends to follow an instruction-shaped sentence in text," and frequently that answer is "another instruction," not "an answer." OpenAI's own account of building InstructGPT (Ouyang et al., 2022) describes precisely this failure mode as the motivation for instruction tuning: language modeling objectives reward predicting the next token of internet text, which is a different goal from being asked something and complying with it.

Instruction tuning is the training stage that closes this gap — it takes a model that is excellent at modelling what text looks like and teaches it, specifically, that an instruction-shaped input should be followed by a compliant, complete response, not by more instruction-shaped text.

What pretraining actually optimizes

To see why the gap exists, look at the objective itself. Pretraining maximizes, over an enormous corpus of raw text, the log-probability the model assigns to each next token given everything before it:

maximize  ∑_t log P(x_t | x_1, x_2, …, x_{t-1})

Nothing in that sum distinguishes "this span is a command directed at the model" from "this span is a question in a textbook" from "this span is a Reddit comment." The model is a distribution matcher over web-scale text. It picks up an enormous amount of world knowledge, syntax, and reasoning patterns as a side effect of getting good at that prediction task — but it never sees a training signal that says "when a human directs you to do X, the correct behavior is to do X, completely, and then stop." Whatever conversational or obedient behavior falls out of pretraining alone is incidental, inconsistent, and unreliable, because the training objective never targeted it.

Supervised fine-tuning: the same loss, a different mask

Instruction tuning — more precisely called supervised fine-tuning (SFT) at this stage — does not invent a new loss function. It reuses the exact next-token cross-entropy loss from pretraining. What changes is the data and, critically, which token positions are allowed to contribute to the gradient.

The training data is now a curated set of (instruction, response) pairs, each written or filtered so the response actually satisfies the instruction. Each pair is concatenated into one sequence — instruction tokens followed by response tokens, usually separated by a template marker. The model is trained with the ordinary next-token objective across the whole sequence, but a loss mask zeroes out the contribution of every position that falls inside the instruction span. Only the response tokens generate gradient. The instruction tokens still pass through the model as context — the model must read and understand them — but predicting them is not a task the model is scored on, because there is nothing to teach there: the model already models "how instructions are phrased" perfectly well from pretraining. What it does not yet reliably do is treat an instruction as something to be obeyed by the tokens that follow it. Masking the prompt concentrates every unit of gradient on exactly that missing behavior.

The diagram below shows this mechanism on one training sequence: four instruction tokens (masked, no gradient) followed by two response tokens (unmasked, gradient flows).

Worked example: computing the masked loss by hand

Take a toy vocabulary of four tokens, {A, B, C, D}, and a six-token training sequence whose first four tokens are the instruction and last two are the response — matching the diagram above. At each of the five positions, the model (at some point during training) outputs logits over the four-token vocabulary. Softmax converts logits to probabilities, and the cross-entropy loss at that position is −log(p_target), where target is the token that actually comes next in the training example.

Sum the five per-position losses and divide by 5: the ordinary, unmasked mean cross-entropy over the whole sequence is (0.3880 + 0.6318 + 0.5642 + 0.1681 + 0.2018) / 5 = 1.9539 / 5 = 0.3908. Now apply the mask: keep only positions 4 and 5, whose target tokens are the response. The masked mean loss is (0.1681 + 0.2018) / 2 = 0.3699 / 2 = 0.1849 — the exact figure shown inside the loss box in the diagram. This is the number that actually determines the size and direction of the SFT gradient update; positions 1–3 never enter the average at all.

Two things are worth noticing in the table itself. First, the model already assigns fairly high probability to the correct instruction-continuation tokens (0.68, 0.53, 0.57) even though this is a fine-tuning checkpoint and instruction phrasing was never explicitly trained for — that is pretraining knowledge showing through, exactly as the theory predicts. Second, masking does not mean those three positions are skipped in the forward pass; the model still reads "Summarize this email :" as context when predicting position 5. Masking only strips positions 1–3 out of the loss average, not out of the sequence.

Here is the same computation as runnable Python, using only the standard library, with no helper functions — every function used is defined in the snippet:

import math

def softmax(z):
    m = max(z)
    exps = [math.exp(x - m) for x in z]
    s = sum(exps)
    return [e / s for e in exps]

logits = [
    [1.2, 0.3, 2.5, 0.1],
    [0.5, 1.8, 0.2, 0.9],
    [2.0, 0.1, 0.4, 1.1],
    [0.2, 0.1, 3.0, 0.3],
    [0.1, 0.2, 0.3, 2.8],
]
targets = [2, 1, 0, 2, 3]     # index of A/B/C/D: C,B,A,C,D
mask    = [0, 0, 0, 1, 1]     # 0 = instruction token, 1 = response token

losses = []
for z, t in zip(logits, targets):
    p = softmax(z)
    losses.append(-math.log(p[t]))

unmasked_loss = sum(losses) / len(losses)
masked_loss   = sum(l for l, m in zip(losses, mask) if m) / sum(mask)

print(round(unmasked_loss, 4))   # 0.3908
print(round(masked_loss, 4))     # 0.1849

Running this prints 0.3908 then 0.1849, matching the table exactly. In a real SFT run this pattern repeats over millions of tokens: every training sequence contributes gradient only from its response span, and the accumulated effect over many such sequences is what reshapes the model's behavior on instruction-shaped input, without touching the vast bulk of what it already knows.

Why the prompt/response template matters

The mask has to know where the instruction ends and the response begins, so every SFT example is wrapped in a fixed template with explicit role markers, for instance:

<|system|>
You are a helpful assistant.
<|user|>
Summarize this email: "Server maintenance is scheduled for 2 AM IST tonight."
<|assistant|>
Sure! One-line summary: scheduled server maintenance tonight at 2 AM IST.

The loss mask is applied mechanically from these markers: everything from the start of the sequence up to and including <|assistant|> is masked out; everything from the first response token onward is scored. This is exactly what utilities like Hugging Face's DataCollatorForCompletionOnlyLM implement — they search for the assistant marker in each tokenized example and build the mask automatically. It also explains a practical failure mode: if the template used at inference time (in the chat app or API) does not exactly match the template used during SFT, the model receives input in a distribution it never learned the "obey" behavior for, and instruction-following quality degrades even though the underlying weights are unchanged. The template is not decoration; it is the addressing scheme the mask depends on.

How much data, and why so little is enough

Pretraining corpora run into trillions of tokens. Instruction-tuning datasets are dramatically smaller — the original InstructGPT SFT set used on the order of ten thousand or so human-written demonstrations; Stanford's Alpaca used 52,000 generated instruction-response pairs; Meta's LIMA experiment (Zhou et al., 2023) showed a base LLaMA model fine-tuned on just 1,000 carefully curated instruction-response pairs already producing competitive, well-formatted, helpful responses. That is roughly six orders of magnitude less data than pretraining, yet the behavioral change is large and consistent. The masked-loss mechanism explains why this is not surprising: SFT is not asking the model to learn language or facts from scratch on this small dataset — that job was already done during pretraining. It is only asking the model to learn a much narrower mapping: "instruction-shaped input → complete on-topic response," using representations that already exist. A narrow mapping needs far fewer examples to pin down than the full breadth of natural language.

Misconception: "instruction tuning teaches the model new facts"

The most common misreading of this stage is to treat SFT as a place where new knowledge gets loaded into the model — as if writing enough example question-answer pairs about, say, ISRO's Aditya-L1 mission would make the model reliably knowledgeable about it. That is not what the masked cross-entropy update is doing. Look again at the loss box in the diagram: it is scoring how well the model predicts response tokens, i.e. how well it reproduces fluent, on-topic, correctly formatted continuations. If the correct factual content was never present anywhere in the pretraining corpus, a handful of SFT examples mentioning it will typically be memorized narrowly — the model may parrot that exact sentence if asked in nearly the same words, but it will not have generalized the fact the way it generalizes facts absorbed from billions of pretraining tokens describing the same entity from many angles. This is precisely why LIMA's authors call their finding the "superficial alignment hypothesis": a model's knowledge and core capabilities are learned almost entirely during pretraining, and SFT teaches which subdistribution of already-learned outputs to surface, and in what format, when given an instruction. Practically, this means fixing a model's factual gaps is a pretraining-data or retrieval-augmentation problem, not something a few hundred instruction-tuning examples can patch — and it is why instruction-tuned models can sound authoritative on a topic while still being wrong about its specifics: fluent formatting was successfully taught; the missing fact was not.

Instruction tuning is not RLHF

A second, adjacent confusion worth separating cleanly: instruction tuning (SFT) and RLHF are two different stages with two different objectives, not two names for the same thing. SFT is supervised learning — the masked cross-entropy loss above, trained directly on human- or model-written target responses, with no reward model and no sampling from the policy during training. RLHF (or its more recent stand-in, direct preference optimization, DPO) comes after SFT: it trains on pairs of candidate responses ranked by human preference, fits a reward model (or, in DPO, an implicit one) to that preference data, and then further adjusts the SFT model to increase the relative likelihood of preferred-style outputs. SFT teaches the model the basic shape of "answer the instruction, then stop." RLHF/DPO refines which of several valid answers the model should prefer — more helpful over less helpful, more concise over rambling, safer over unsafe — using comparative rather than absolute supervision. A model can be instruction-tuned without ever going through an RLHF stage and will still follow directives reasonably; it typically will not be as consistently polished or preference-aligned as one that has been through both stages.

What can go wrong: forgetting and format-chasing

Because SFT updates the same weights that pretraining produced, an aggressive fine-tuning run — too high a learning rate, too many epochs on a narrow dataset — can overwrite pretrained capabilities faster than it installs the new instruction-following behavior, a failure called catastrophic forgetting. The model gets very good at the SFT dataset's exact style and starts losing breadth elsewhere. Standard mitigations follow directly from the diagnosis: use a small learning rate relative to pretraining, train for only one to three epochs over the SFT set, and where possible use a parameter-efficient method like LoRA, which restricts the update to a small set of low-rank adapter matrices instead of the full weight matrices, physically limiting how much any one fine-tuning run can move the model away from its pretrained state. A related failure is format-chasing: if the SFT dataset is narrow or stylistically repetitive (every example happens to start with "Certainly!" or always uses bullet points), the masked loss will happily drive the model to reproduce that surface tic on unrelated prompts, because from the loss function's point of view that tic is just as much "correctly predicting the response tokens" as the substantive part of the answer is. Diverse, high-quality instruction data — not just more of it — is what keeps the masked gradient pointed at genuine task competence rather than stylistic mimicry.

Active recall

Attempt each question before reading its answer.

1. Why does SFT use loss masking on the prompt tokens instead of computing cross-entropy over the full instruction + response sequence?
2. A training position has vocabulary {A, B, C, D} with logits [0.6, 1.4, 0.2, 0.0], and the target token is B. Compute the cross-entropy loss at that position.
3. A base model, given "List the prime numbers below 20," continues with "List the prime numbers below 30." instead of listing 2, 3, 5, 7, 11, 13, 17, 19. Explain this in terms of the pretraining objective.
4. Why can 1,000–50,000 instruction-response pairs meaningfully change a model's behavior when pretraining used trillions of tokens?
5. A team fine-tunes a model on 300 Q&A pairs about a newly launched product so the model can answer customer questions about it accurately. Based on the superficial alignment hypothesis, what should they expect, and what would actually fix factual accuracy about the new product?

Answers

1. The instruction tokens are context the model must read, but predicting them is not a skill the model is missing — pretraining already made it fluent at modelling instruction-shaped text. What is missing is the behavior of treating an instruction as something to be obeyed by the tokens that follow. Masking the prompt means every unit of gradient goes toward learning that missing "instruction → compliant response" mapping instead of being diluted by re-teaching the model to predict text it can already predict well.

2. softmax([0.6, 1.4, 0.2, 0.0]): exponentials are e^0.6=1.8221, e^1.4=4.0552, e^0.2=1.2214, e⁰=1.0, summing to 8.0987. Dividing gives probabilities [0.2250, 0.5007, 0.1508, 0.1235]. The target is B (index 1), so loss = −log(0.5007) = 0.6917.

3. Pretraining's objective is to maximize the likelihood of the next token given the preceding text, estimated from the statistics of the training corpus, not to satisfy a request. On the web, "List the primes below N" lines very often appear inside problem sets, where they are followed by more similarly-shaped instruction lines rather than by the worked answer. The base model, having never had prompt tokens masked out of a loss so that only "answer, then stop" was rewarded, samples the continuation its pretraining distribution says is likely — which can easily be another instruction rather than compliance with the one given.

4. Because SFT is not teaching language or world knowledge from scratch on that small dataset — pretraining already did that job using trillions of tokens. SFT is only pinning down a comparatively narrow mapping ("instruction-shaped input → complete, on-topic response using existing knowledge"), and a narrow mapping requires far fewer examples to specify than the full breadth of a language, the way a handful of labelled examples can fix the orientation of a mostly-known function without re-deriving it from scratch.

5. The superficial alignment hypothesis predicts the fine-tune will make the model sound fluent and well-formatted when discussing the new product, and it may correctly reproduce facts that appear near-verbatim in the 300 examples, but it should not be expected to generalize reliably to product questions phrased differently or covering details outside those 300 pairs — the underlying factual knowledge was never in pretraining, so SFT cannot manufacture broad, generalized knowledge of it. What would actually fix accuracy is either putting the product documentation into the model's context at inference time (retrieval-augmented generation) or including the product information densely and repeatedly, from many angles, in further pretraining/continued pretraining — not by adding more instruction-formatted pairs, which mainly teach format and behavior, not facts.

Think About It

Think about this: How would you explain instruction tuning: making models follow directives to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.