Prompt Engineering: From Zero-Shot to Chain-of-Thought

In 2022, researchers at Google Research ran an experiment that looks almost like a magic trick. Jason Wei, Xuezhi Wang, Dale Schuurmans, and colleagues fed a 540-billion-parameter PaLM model thousands of grade-school math word problems from the GSM8K benchmark — the kind of multi-step numerical problem where an answer depends on three or four dependent steps — using a standard few-shot prompt whose worked examples mapped each question straight to a bare final number. The model got roughly one in five of these multi-step problems right: 17.9% accuracy. It wasn't that the model didn't know the relevant arithmetic; ask it to compute 23 − 20, or 3 + 6, in isolation, and it would answer correctly every time. The failure was structural: shown only exemplars that jumped from question to final number, the model produced an answer — a single token sequence pattern-matched from similar-looking problems — without ever working through the intermediate steps. Change nothing about the model's weights, change only what the worked exemplars demonstrated — full intermediate reasoning instead of a bare number — and accuracy on the identical problem set rose to 56.9%, more than tripling. No retraining, no new data, no change to the neural network at all. Just different words arranged differently in the input. This is chain-of-thought prompting; the paper behind it, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022), is examined in full below.

This is the entire premise of prompt engineering, and it is worth taking seriously as a technical discipline rather than a bag of tricks: a large language model's behavior is a function of its parameters and its input context, and for a fixed set of parameters, the input context is the only lever a developer has. Chain-of-thought prompting — the technique behind that tripling — is the single most consequential discovery in this space, and understanding exactly why it works, and exactly where it stops working, is what separates someone who can coax good answers out of an LLM from someone who is guessing.

First principles: what a prompt actually does

You covered in Grade 11 how a transformer decoder generates text: at each step it computes a probability distribution over the next token, conditioned on every token that came before it in the context window, via stacked self-attention layers. Nothing about inference-time prompting changes any weight in those layers. What changes is the conditioning set — the sequence of tokens the model attends over before it emits the next one. This is the whole mechanism. A prompt is not an instruction in the way a function call is an instruction to a program; it is evidence that shifts a probability distribution.

Zero-shot prompting supplies only a task description — no worked examples — and relies entirely on the model's parametric knowledge and instruction-tuning to interpret it correctly: "Q: [problem]. A:" and nothing more.

Few-shot prompting, introduced at scale by Tom Brown and colleagues in "Language Models are Few-Shot Learners" (NeurIPS 2020, the GPT-3 paper), places k worked input–output exemplars directly in the context before the real question. Brown et al. showed that a sufficiently large model, given a handful of exemplars in-context, could match or approach the performance of a model that had been fine-tuned on thousands of labelled examples for that specific task — without a single gradient update. This is called in-context learning, and it is genuinely strange when you first meet it: the exemplars never touch backpropagation. They only ever exist as extra tokens in the attention window. Attention lets every token the model generates attend back over the exemplars, and because the exemplars establish a pattern — a format, a style, a level of granularity — the model's next-token distribution shifts toward continuing that pattern rather than falling back on its prior.

One-shot is the special case k = 1. What chain-of-thought prompting adds on top of this taxonomy is not a fourth category of "how many examples" — it is a claim about what those examples should contain.

Worked example: where zero-shot breaks

Take the canonical illustration from Jason Wei and colleagues' chain-of-thought paper (more on it below): "The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?" Trace the arithmetic yourself before reading on: they start with 23, use 20 (23 − 20 = 3 remaining), then buy 6 more (3 + 6 = 9). The correct answer is 9.

A zero-shot standard prompt — "Q: [problem]. A: [number]" — asks the model to jump directly from the problem statement to a single number. Under next-token prediction, the model has no scratch space in which to perform 23 − 20 and then 3 + 6 as two separate, checkable operations; it must predict a plausible-looking numeral directly, conditioned only on the surface pattern of the question. On problems this short the model often gets lucky. On problems requiring three or more dependent operations, this pattern-matching approach degrades sharply and reliably: a common zero-shot failure mode is to combine the numbers in the order they appear (23 + 6 = 29, ignoring that 20 were used rather than added) rather than tracking which operations are additions and which are subtractions. The problem is not missing knowledge — the model knows what "used" and "bought" mean — it is the absence of any mechanism forcing it to apply that knowledge serially, one dependent step at a time, before committing to a final token.

Chain-of-thought prompting (Wei et al., 2022)

Jason Wei, Xuezhi Wang, Dale Schuurmans, and colleagues at Google Research published "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" at NeurIPS 2022. Their fix keeps the few-shot format from Brown et al. but changes what the exemplars demonstrate: instead of exemplars mapping question directly to final answer, each exemplar shows the full intermediate reasoning path.

Q: Roger has 5 tennis balls. He buys 2 more cans of
   tennis balls. Each can has 3 tennis balls. How many
   tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls
   each is 2 × 3 = 6 tennis balls. 5 + 6 = 11. The
   answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make
   lunch and bought 6 more, how many apples do they have?
A:

Conditioned on an exemplar whose answer field is itself a chain of small, checkable arithmetic statements, the model's next-token distribution now favours continuing in that same format for the test question — emitting "They used 20, so 23 − 20 = 3. They bought 6 more, so 3 + 6 = 9. The answer is 9." — rather than jumping straight to a bare number. Each intermediate statement narrows the space of plausible continuations for the next one, because the model is now conditioning each step not just on the original question but on its own prior, already-committed reasoning tokens.

The scale of the effect, as reported in Wei et al.'s paper, is large: on the GSM8K grade-school math benchmark, standard few-shot prompting with PaLM 540B scored 17.9% accuracy; chain-of-thought few-shot prompting on the identical model scored 56.9% — enough to surpass the prior fine-tuned state of the art on that benchmark, without any fine-tuning at all. The paper also reports that this gain is an emergent property of scale: chain-of-thought prompting barely helps, and can even hurt, on small models (under roughly 10 billion parameters), and only produces large gains once models cross a scale threshold — smaller models tend to generate fluent-looking but logically incoherent reasoning chains.

Zero-shot chain-of-thought (Kojima et al., 2022)

A natural question follows: does chain-of-thought require hand-written exemplars at all, or can you get the same benefit from an instruction alone? Takeshi Kojima, Shixiang Shane Gu, and colleagues answered this in "Large Language Models are Zero-Shot Reasoners" (NeurIPS 2022). Their method needs no exemplars whatsoever — it inserts a single fixed sentence, "Let's think step by step," between the question and the answer field, then (in a second pass) extracts the final numeral from the resulting reasoning text.

Q: The cafeteria had 23 apples. If they used 20 to make
   lunch and bought 6 more, how many apples do they have?
A: Let's think step by step.

As reported in Kojima et al.'s Table 2, this single trigger phrase, applied to text-davinci-002 with no exemplars at all, raised GSM8K accuracy from 10.4% (standard zero-shot) to 40.7% (zero-shot chain-of-thought) — and raised MultiArith accuracy from 17.7% to 78.7%. This is the more surprising result of the two papers: the model already "knows" how to decompose a multi-step problem — that capability was latent in its pretraining — but a bare "A:" prompt never elicits it. A single sentence nudging the output distribution toward step-by-step text is enough to surface a capability that was already there.

Why intermediate tokens are the mechanism, not "trying harder"

It helps to be precise about what generating a chain of thought actually buys the model computationally. A transformer performs a bounded amount of computation per generated token — a fixed number of layers, each doing a fixed amount of matrix arithmetic, regardless of how "hard" the underlying step is. A single "A: [number]" token position cannot encode an unbounded number of sequential sub-computations; it has exactly one forward pass through the network to produce that token. When the model is instead allowed to emit "23 − 20 = 3" as its own tokens, and then those tokens re-enter the context for the next forward pass, the model effectively gets a second full forward pass to compute "3 + 6", conditioned on an already-correct intermediate result rather than having to hold it all in one shot. Chain-of-thought text functions as an external memory — a scratchpad, in the terminology used by Maxwell Nye and colleagues in a closely related 2021 paper ("Show Your Work: Scratchpads for Intermediate Computation with Language Models") — that trades output length for computational depth. More reasoning tokens is, in a very literal sense, more forward passes, hence more serial computation available to the problem, at the cost of latency and API spend proportional to the chain's length.

Self-consistency: sampling multiple chains and voting

Because a single chain-of-thought sample can still go wrong — the model might make one arithmetic slip mid-chain and propagate it — Xuezhi Wang, Jason Wei, and colleagues proposed self-consistency in "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (ICLR 2023): sample k independent chain-of-thought completions at nonzero temperature, extract each one's final answer, and take a majority vote. Independent reasoning errors rarely agree with each other, so the correct answer — which every valid chain converges on regardless of the exact path — tends to dominate the vote even when individual chains disagree.

from collections import Counter

def self_consistency_vote(cot_completions):
    """cot_completions: list of full CoT completion strings, one per
    independently sampled generation. Assumes each ends with a line of
    the form 'The answer is <integer>.' Returns (winning_answer,
    vote_count, full_tally)."""
    answers = []
    for completion in cot_completions:
        last_line = completion.strip().split("\n")[-1]
        digits = "".join(ch for ch in last_line if ch.isdigit())
        answers.append(int(digits))
    tally = Counter(answers)
    winner, count = tally.most_common(1)[0]
    return winner, count, tally

# k = 5 independent samples for the cafeteria-apples problem
# (each completion is two lines: working, then a final "The answer is N." line)
samples = [
    "23 apples, used 20 leaves 23-20=3. Bought 6 more: 3+6=9.\nThe answer is 9.",
    "They used 20 of the 23, leaving 3. Then 3+6=9.\nThe answer is 9.",
    "23-20=3, then 3+6=9.\nThe answer is 9.",
    "Start 23, minus 20 used is 3, plus 6 bought is 9.\nThe answer is 9.",
    "23 apples, plus 6 more bought is 29.\nThe answer is 29.",  # forgot to subtract the 20 used
]

winner, count, tally = self_consistency_vote(samples)
print(winner, count, tally)

Trace it: each of the first four strings ends with the line "The answer is 9.", so digits extracts "9" four times. The fifth chain skipped the subtraction step entirely and ends with "The answer is 29.", contributing one vote for 29. tally is therefore Counter({9: 4, 29: 1}), and tally.most_common(1)[0] returns (9, 4). The printed output is exactly:

9 4 Counter({9: 4, 29: 1})

The flawed chain is outvoted, not because the system detected the arithmetic error, but because four independently-sampled correct reasoning paths outnumbered one independently-sampled flawed one. This only works because sampling is stochastic (temperature > 0) — five greedy, deterministic samples of the same prompt would produce five identical chains and the vote would be meaningless.

The misconception to unlearn: a chain-of-thought is not a debug log

The most common misreading of this technique, among students and practitioners alike, is to treat the visible reasoning text as a faithful, causal trace of what the network actually computed internally — as if printing "23 − 20 = 3" proves that some internal circuit literally executed subtraction and that step is why the final answer came out correct. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman tested this directly in "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting" (NeurIPS 2023). They showed that a model's stated chain of reasoning can be systematically decoupled from the actual cause of its answer: when they biased the order or content of multiple-choice options in ways known to sway model outputs, the model's final answer shifted accordingly — but its chain-of-thought text confidently narrated a reasoning path that never mentioned the biasing factor at all, constructing a plausible-sounding justification for whatever answer the bias had already pushed it toward.

The correct mental model is this: chain-of-thought text is generated by the same next-token prediction process as everything else the model outputs — it is not a separate, privileged introspection channel. It happens to help arithmetic and symbolic reasoning tasks because forcing token-by-token decomposition genuinely gives the model more computation to work with, as the GSM8K numbers above demonstrate. But "the model wrote down a plausible-looking justification" and "that justification is why the model reached that answer" are separate claims, and the second does not follow automatically from the first. Treat a chain of thought as a strong hint worth checking, especially on tasks with an independently verifiable answer — never as a certified proof.

Where chain-of-thought does not help

Prompting overhead is not free — each reasoning token costs latency and, on a metered API, money — so it is worth knowing where the technique earns its cost. Wei et al. explicitly report that chain-of-thought gains are concentrated in tasks requiring multiple dependent arithmetic, logical, or symbolic steps; on single-hop factual lookups ("What is the capital of Rajasthan?") or straightforward classification (sentiment of a short review), there is no multi-step computation to decompose, so a step-by-step scaffold adds tokens without adding accuracy, and can occasionally hurt by giving the model more surface area to talk itself into a wrong answer. Modern instruction-tuned and reasoning-specialized models increasingly generate internal step-by-step deliberation by default even under a bare zero-shot prompt, which is narrowing — but has not eliminated — the gap that an explicit "let's think step by step" used to open up.

Active recall

Attempt each question before reading its answer.

1. In formal terms, what distinguishes zero-shot, one-shot, and few-shot prompting, and why does none of them involve a gradient update to the model's weights?
2. A classmate says: "Chain-of-thought just means giving the model more examples." What is actually different between a standard few-shot prompt (Brown et al., 2020) and a few-shot chain-of-thought prompt (Wei et al., 2022) that use the identical set of questions?
3. Using the same exemplar pattern as the Roger tennis-ball example, hand-write a chain-of-thought answer for: "A shop had 40 pencils. It sold 3 boxes of pencils, each box holding 8 pencils, and then received a fresh delivery of 25 pencils. How many pencils does the shop have now?" Show every intermediate step and the final numeric answer.
4. Why would prepending "Let's think step by step" to the zero-shot prompt "What is the capital of India?" be unlikely to change the model's accuracy, while it substantially raises accuracy on GSM8K-style problems?
5. A classmate says: "If the model's chain-of-thought explanation lists steps A, B, C, then those are literally the internal steps its network carried out — I can read it like a debug log." Using Turpin et al.'s finding, explain what is wrong with this claim and what a chain-of-thought explanation should be treated as instead.
6. In the self-consistency code above, suppose the underlying problem changes: the cafeteria still starts with 23 apples and still uses 20, but now buys 15 more instead of 6. Recompute the correct final answer, rewrite all five sample completions' final lines accordingly (keeping the same reasoning pattern, including the one flawed chain that forgets to subtract), and state the new output of self_consistency_vote.

Answers

1. The distinction is purely the count k of worked input–output exemplars placed in the prompt before the real question: zero-shot has k = 0 (instruction only), one-shot has k = 1, few-shot has k > 1 (typically 2–8 for context-window reasons). None involves a gradient update because the exemplars are consumed only as extra tokens in the attention context during a single forward pass; the network's parameters are exactly what they were before the prompt was written. The "learning" in in-context learning is a shift in the conditional output distribution for this one query, not a persistent change to the model.

2. A standard few-shot prompt's exemplars map question directly to final answer ("Q: ... A: 11"), teaching the model the output format (a bare number) but nothing about how to derive it. A few-shot chain-of-thought prompt's exemplars map question to a full intermediate reasoning trace ending in the answer ("A: Roger started with 5... 5+6=11. The answer is 11."), teaching the model to emit and condition on its own intermediate computation before committing to a final token. Same questions, same k, structurally different exemplar content — and that content difference is the entire source of the accuracy gap (17.9% → 56.9% on GSM8K with PaLM 540B, per Wei et al.).

3. Correct trace: 3 boxes of 8 pencils sold is 3 × 8 = 24 pencils sold. 40 − 24 = 16 pencils remain. A delivery of 25 more arrives: 16 + 25 = 41. Full chain-of-thought answer: "The shop started with 40 pencils. It sold 3 boxes of 8 pencils each, which is 3 × 8 = 24 pencils. 40 − 24 = 16 pencils remain. It then received 25 more: 16 + 25 = 41. The answer is 41."

4. "What is the capital of India?" is a single-hop factual retrieval — the answer is a direct lookup in the model's parametric memory with no dependent intermediate arithmetic or logical steps to decompose, so there is no serial computation for extra reasoning tokens to provide room for; the trigger phrase adds latency without adding useful computational depth. GSM8K-style problems require chaining two or more dependent operations, where each additional reasoning token gives the model an extra forward pass to correctly compute the next dependent step — which is exactly the computational bottleneck that a bare "A:" prompt cannot get past.

5. Turpin et al. showed that biasing the model's answer (for example, by manipulating which multiple-choice option appears in a suggestive position) shifted the model's final answer while its chain-of-thought text never mentioned the biasing factor, instead narrating an unrelated, confident-sounding justification for whatever answer the bias produced. This shows the visible reasoning text is generated by the same token-prediction process as the answer itself, not a separate introspective readout of the network's true internal computation — so an explanation being fluent and step-by-step is no guarantee it is the actual cause of the answer. Treat a chain of thought as a checkable hint, verify it against independently known correct steps when it matters, and never accept it as proof of the model's true internal process.

6. New correct answer: 23 − 20 = 3, then 3 + 15 = 18. Every correctly-reasoned chain's final line changes from "The answer is 9." to "The answer is 18." The flawed chain, which forgets to subtract the 20 used and just adds the new apples to the starting count, now computes 23 + 15 = 38 instead of 23 + 6 = 29, so its final line becomes "The answer is 38." tally becomes Counter({18: 4, 38: 1}), and the function returns (18, 4, Counter({18: 4, 38: 1})) — printed as 18 4 Counter({18: 4, 38: 1}). The majority vote still correctly resolves to 18 despite the changed numbers, because changing the shared parameter (15 instead of 6) shifted every chain's arithmetic consistently but did not change which chains were structurally correct versus flawed.

Think About It

Think about this: How would you explain prompt engineering: from zero-shot to chain-of-thought to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where prompt engineering: from zero-shot to chain-of-thought is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting prompt engineering: from zero-shot to chain-of-thought to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind prompt engineering: from zero-shot to chain-of-thought, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.