A shopkeeper's stock count and a language model's blind spot
Picture a wholesale grocer in Devaraja Market, Mysuru, closing the evening ledger. The day began with 23 crates of mangoes at 12 mangoes per crate. During the day 8 crates were sold off whole. In the evening a fresh delivery of 15 crates arrived, each holding 9 mangoes. How many mangoes sit in the godown now? A Class 8 student solves this in four lines: 23×12=276 to start, minus 8×12=96 sold leaves 180, plus the new 15×9=135 gives 315. Nothing about the problem is conceptually hard — it is four multiplication and addition facts, applied in the right order. Yet when researchers at Google Research posed exactly this style of grade-school arithmetic word problem — the GSM8K benchmark — to a 540-billion-parameter PaLM model and asked it to output only the final number, with no room to work anything out, the model answered correctly on just 17.9% of the problems. Give that same frozen model, on the same questions, a handful of worked examples that show the intermediate arithmetic before the final answer, and accuracy jumps to 56.9% — more than triple, without changing a single weight (Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le and Zhou, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," NeurIPS 2022). The only thing that changed between the two runs is whether the model was allowed to write its steps down before committing to a number. That one lever — writing the intermediate steps instead of jumping straight to the answer — is chain-of-thought (CoT) reasoning, and understanding exactly why it produces a threefold swing in accuracy on identical questions, with an identical model, is this chapter's task.
What chain-of-thought prompting actually is
Chain-of-thought prompting is a decoding-time technique, not a training-time one: nothing about the model's parameters changes. In its original form, a prompt is built from a handful of few-shot exemplars — question-and-answer pairs where the answer is written not as a bare number but as a short natural-language derivation ending in the final result — followed by the new question the model must answer. Contrast this with standard (or "direct-answer") prompting, where the exemplars pair each question directly with its final answer and nothing else. Both prompts hand the model the same underlying arithmetic; the only difference is whether the exemplars model the *behavior* of reasoning aloud. Because a transformer language model has no built-in notion of "think before you answer" — it simply predicts the next token conditioned on everything generated so far — showing it exemplars where the next tokens after a question are a derivation, not a number, is enough to make it continue the new question the same way: producing its own derivation before committing to an answer token. Nothing is injected into the model architecture; the entire effect comes from conditioning the autoregressive generation process on a different kind of context.
Why it works: a transformer's compute budget is spent per token, not per problem
A transformer decoder has a fixed number of layers, call it L. Every time it generates one token, that token's representation is built by pushing the current context through exactly L layers of attention and feed-forward computation — a bounded, constant amount of serial computation, regardless of how hard the underlying problem is. If a model is forced to emit the final answer as the very next token after the question, then the entire multi-step composition — multiply, subtract, multiply, add — has to be completed within that single pass through L layers. Composing four sequentially dependent operations inside a fixed-depth circuit is a real computational constraint, not a metaphor: complexity-theoretic arguments about bounded-depth circuits show that some functions requiring many sequential steps simply cannot be computed by a circuit that is too shallow, no matter how wide it is. Chain-of-thought reasoning sidesteps the constraint without adding a single layer to the network. Every generated token — including a token that is part of the model's own reasoning chain — gets appended to the context and triggers a brand-new forward pass through all L layers when the next token is produced. Writing "23×12=276" as an intermediate token is not commentary for the reader; it is additional serial computation, purchased one autoregressive step at a time. A four-step chain of thought does not use L layers of compute — it uses on the order of N×L, where N is the total number of tokens generated across the chain (each arithmetic expression like "23×12=276" is itself several tokens, not one, so N is typically many times the number of reasoning steps). This idea was anticipated by Nye, Andreassen, Gur-Ari, Michalewski, Austin, Bieber, Dohan, Lewkowycz, Bosma, Luan, Sutton and Odena in "Show Your Work: Scratchpads for Intermediate Computation with Language Models" (arXiv, 2021), which trained models to use an explicit scratchpad for exactly this reason before Wei et al. showed the effect could be elicited purely through prompting, with no retraining at all.
Worked example: tracing a chain of thought step by step
Return to the Mysuru grocer. A chain-of-thought prompt for this question is built from one or two worked exemplars of similarly structured problems, each showing the derivation before the answer, followed by the new question. A model conditioned on such a prompt continues in the same style, producing something like the following completion:
Q: A grocer starts with 23 crates of mangoes at 12 mangoes per
crate. She sells 8 crates whole. That evening a new delivery of
15 crates arrives, each holding 9 mangoes. How many mangoes are
in stock now?
A: The grocer starts with 23 crates x 12 mangoes = 276 mangoes.
She sells 8 crates, which is 8 x 12 = 96 mangoes, leaving
276 - 96 = 180 mangoes. The new delivery adds 15 x 9 = 135
mangoes, giving 180 + 135 = 315 mangoes in stock.
The answer is 315.Verify each line independently before trusting the final number: 23×12=276 (23×10=230, plus 23×2=46, total 276 — correct). 8×12=96 (8×10=80, plus 8×2=16, total 96 — correct). 276−96=180 — correct. 15×9=135 (15×10=150, minus 15=135 — correct). 180+135=315 — correct. Every intermediate quantity checks out, and because each was written as a separate token span, the model that generated this text spent many separate full-depth forward passes getting there — one for every token in the derivation, not one for each of the four arithmetic lines, and certainly not one overall. Contrast this with a direct-answer prompt, where the model must emit "315" as its very first output token with no derivation preceding it. A characteristic failure mode of forced direct answering on multi-step word problems is that the model pattern-matches to a quantity that appeared partway through the question — outputting 276 (the starting count, ignoring both the sale and the delivery) or 219 (an arithmetically plausible-looking combination that skips a step) — because the single forward pass available to it was enough to retrieve or lightly combine numbers from the prompt, but not enough to correctly compose all four operations in the right order. This is exactly the gap Wei et al.'s 17.9%-versus-56.9% result is measuring at scale, across thousands of GSM8K problems rather than one.
Zero-shot chain-of-thought and self-consistency
Building hand-crafted worked exemplars for every task is labour-intensive, so Kojima, Gu, Reid, Matsuo and Iwasawa asked a simpler question in "Large Language Models are Zero-Shot Reasoners" (NeurIPS 2022): what if you skip the exemplars entirely and just append the words "Let's think step by step" after the question, before letting the model generate? On the MultiArith arithmetic benchmark, this single fixed phrase raised GPT-3 (text-davinci-002) accuracy from 17.7% with a bare direct-answer prompt to 78.7% — zero-shot, with no worked examples of any kind in the prompt. The phrase itself carries no problem-specific information; its only function is to bias the very next generated tokens toward a derivation rather than a number, after which the same per-token compute argument from the previous section takes over. This is strong evidence that the effect is not about the model "learning from examples" in the prompt so much as about unlocking extra serial computation that the model already had the capacity to use, given the chance.
A second refinement addresses the fact that a single chain of thought, sampled once, can go wrong exactly the way a human's mental arithmetic can — a single dropped digit anywhere in the chain corrupts everything after it. Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery and Zhou's "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (ICLR 2023) proposes sampling several independent chains for the same question at nonzero temperature, extracting the final answer from each, and taking a majority vote instead of trusting any single chain. Errors introduced by an unlucky sampling path tend not to repeat identically across independent samples, so they get outvoted, while the correct derivation — reachable by more than one valid path through the arithmetic — tends to win. The code below traces this on five sampled completions for the Mysuru problem, one of which contains a genuine arithmetic slip:
import re
from collections import Counter
# Five independently sampled chain-of-thought completions for the
# same prompt (temperature > 0). Shown as fixed strings here since
# this trace does not call a live model.
chains = [
"23 x 12 = 276. Sold 8 x 12 = 96. 276 - 96 = 180. "
"New: 15 x 9 = 135. 180 + 135 = 315. Answer: 315",
"23 x 12 = 276 mangoes to start. 8 x 12 = 96 removed, "
"leaving 180. Adding 15 x 9 = 135 gives 315. Answer: 315",
"Total before sale: 23 x 12 = 276. After selling 96: "
"276 - 96 = 180. New stock: 15 x 9 = 135. Final: "
"180 + 135 = 315. Answer: 315",
"23 x 12 = 276. 8 x 12 = 96. 276 - 96 = 190. 15 x 9 = 135. "
"190 + 135 = 325. Answer: 325",
"276 mangoes originally. Removing 96 sold leaves 180. "
"Adding the new delivery of 135 gives 315. Answer: 315",
]
def extract_final_answer(chain: str) -> int:
match = re.search(r"Answer:\s*(\d+)", chain)
return int(match.group(1))
votes = Counter(extract_final_answer(c) for c in chains)
majority_answer, count = votes.most_common(1)[0]
print(votes)
print(f"Self-consistency answer: {majority_answer} "
f"({count}/{len(chains)} chains agree)")Trace it by hand: the fourth chain makes a genuine subtraction error (276−96=190 instead of 180, then 190+135=325 instead of 315), while the other four independently reach 315. votes becomes Counter({315: 4, 325: 1}), most_common(1)[0] returns (315, 4), and the two print statements output Counter({315: 4, 325: 1}) followed by Self-consistency answer: 315 (4/5 chains agree). The single flawed chain is outvoted without anyone having to detect the error directly — majority vote over independently sampled reasoning paths is doing the error correction.
Beyond a single chain: decomposition and search
Chain-of-thought as described so far is still one linear sequence of tokens, generated left to right with no backtracking. Two follow-on techniques relax that constraint. Zhou, Schärli, Hou, Wei, Scales, Wang, Schuurmans, Cui, Bousquet, Le and Chi's "Least-to-Most Prompting Enables Complex Reasoning in Large Language Models" (ICLR 2023) splits a hard problem into an explicit list of easier sub-problems first, then solves them in order, feeding each solved sub-problem back in as context for the next — useful precisely when a single chain of thought is long enough that the model tends to drift or lose track partway through, because it shrinks each individual reasoning span the model has to get right in one pass. Yao, Yu, Zhao, Shafran, Griffiths, Cao and Narasimhan's "Tree of Thoughts: Deliberate Problem Solving with Large Language Models" (NeurIPS 2023) goes further, treating each partial chain of thought as a node in a search tree: at each step, several candidate next-thoughts are generated, a scoring step evaluates which look promising, and search (breadth-first or depth-first) explores the more promising branches while abandoning weak ones — turning a single autoregressive guess into deliberate, backtracking search over reasoning paths, at the cost of many more forward passes per question answered. Several current large reasoning-focused models push this idea further still, training the model with reinforcement learning to produce very long internal chains of thought before answering, rather than relying only on a prompt to elicit the behavior — the underlying per-token compute mechanism from the previous section is the same, but the length and structure of the chain becomes something the model is optimized to control rather than something a prompt author writes by hand.
A common misconception: the chain is not a report on reasoning that already happened
It is tempting to picture chain-of-thought the way a student writes up a maths solution: work the problem out mentally first, then write the steps down afterward, for the reader's benefit, in a form separate from the "real" thinking. This is exactly backwards for a transformer generating a chain of thought, and the mistake matters because it hides the actual mechanism. The model has no private scratch memory where it silently solves the problem before generating any text; the token stream it produces autoregressively is not a report on computation that already finished elsewhere — it is the only extra computation the model gets. Section 3 established this concretely: each generated token triggers one full-depth forward pass, conditioned on the tokens generated so far, and that is the entire extent of the model's additional "thinking." If a model is truncated after "23×12=276" and forced to answer immediately, it has not silently already computed 315 and merely been cut off from reporting it — it genuinely has not done the remaining three operations yet, because doing them was never anything other than generating the tokens that describe them. This is precisely why forcing an immediate final-token answer, with no chain allowed, measurably degrades performance on multi-step problems: there is no hidden computation being skipped over, only computation that never happened.
Where chain-of-thought reasoning breaks down
Three limits are worth knowing precisely rather than treating CoT as a strictly-better free upgrade. First, the benefit is scale-dependent: Wei et al. report that chain-of-thought prompting provides little or no benefit, and can even hurt, on models below roughly 100 billion parameters — smaller models often generate fluent-looking reasoning chains that are internally inconsistent or arithmetically wrong, so the extra tokens add noise rather than correct computation. The technique is better described as unlocking a latent capability that emerges at sufficient scale than as a universal trick. Second, a chain of thought that reads as a valid, step-by-step derivation is not guaranteed to be the actual cause of the final answer. Turpin, Michael, Perez and Bowman, in "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting" (NeurIPS 2023), show that a model's stated reasoning can be systematically biased by features of the prompt the model never mentions in its chain — the visible steps can be a plausible-sounding post-hoc rationalization of an answer reached for other reasons, rather than a faithful trace of what actually determined the output. A fluent, correct-looking derivation is evidence the answer is right, not proof. Third, chain-of-thought is not free: every extra reasoning token is an extra forward pass, so a technique that triples accuracy on hard multi-step problems can multiply inference latency and cost several-fold, and gains nothing on tasks that were already single-hop lookups rather than multi-step compositions — asking a model to "think step by step" before naming the capital of a state does not help, because there was never a serial-depth bottleneck to relieve in the first place.
Active recall
1. A transformer decoder has 80 layers. Explain, in terms of per-token compute, why a single forward pass struggles to reliably solve a problem whose correct solution requires composing 12 sequentially dependent operations — and explain why chain-of-thought prompting resolves this without adding a single layer to the network.
2. A classmate says: "Chain-of-thought works because the model thinks the problem through silently first, then writes the answer, then writes out an explanation of its thinking afterward for the reader's benefit." Identify precisely what is wrong with this claim, and state what actually happens instead.
3. Return to the Mysuru grocer, but now suppose 2 of the original 23 crates were found spoiled and discarded before any sales took place, with everything else in the problem unchanged. Recompute the final stock, and state explicitly which of the four downstream numbers from the original worked example change and which stay identical.
4. In the self-consistency code trace, suppose a sixth sampled chain is added whose text ends "...Answer: 180" because generation was cut off mid-derivation. Give the new value of votes and state whether majority_answer changes.
5. Turpin et al. (2023) show that a model's chain of thought can be unfaithful. If a model produces a fluent, step-by-step derivation and reaches the correct final answer, is it valid to conclude the derivation shown is what actually caused that answer? Justify your answer.
6. Wei et al. report that chain-of-thought prompting provides little benefit — and can even hurt — below roughly 100 billion parameters. Propose a mechanistic reason (not just "small models are worse at everything") for why an intervention that helps a 540B model can fail to help, or actively hurt, a much smaller one.
Answers.
1. Each of the 80 layers performs one bounded step of computation per forward pass; a single generated token is the output of exactly one pass through all 80 layers, so the entire chain of 12 sequential dependencies must be resolved within that one fixed-depth computation. If the true dependency structure of the problem requires more sequential steps than the layer stack can compose in one pass, no width or parameter count fixes that — the depth budget is spent. Chain-of-thought prompting does not add layers; it adds forward passes. By writing each intermediate result as its own generated token, the model triggers a fresh 80-layer pass for every step, so a 12-step dependency chain can be resolved using roughly N separate 80-layer passes, where N is the number of tokens needed to write out all 12 steps — considerably more than 12, since each step is itself several tokens — instead of one.
2. The claim assumes the model has already computed the answer in some hidden form before it starts writing, and that the visible chain is a report written afterward. A transformer has no private scratch memory of this kind: the only extra computation available to it is the sequence of forward passes triggered by the tokens it actually generates. The chain of thought is the computation happening, in real time, one token at a time — not a post-hoc write-up of computation that finished elsewhere. This is why truncating a model's chain and forcing an immediate answer measurably lowers accuracy: the remaining steps were never silently done, they simply never happened.
3. Discarding 2 of the 23 crates before any sales changes the starting count to 21 crates, so the first number changes: 21×12=252 (not 276). The 8 crates sold are unaffected in count (still 8, still valid since 21>8), but the mangoes-sold figure is unchanged at 8×12=96, and the subtraction changes because the starting figure changed: 252−96=156 (not 180). The new delivery term, 15×9=135, is completely untouched by the spoilage — it is an independent event later in the day, so it stays 135. The final total changes because it inherits the changed intermediate: 156+135=291 (not 315). So three of the four numbers ripple (276→252, 180→156, 315→291), the sold-mangoes figure (96) stays identical because it depends on crates sold and mangoes per crate, neither of which changed, and the new-delivery figure (135) stays identical because it is causally unconnected to the spoilage.
4. The five original chains give four votes for 315 and one for 325. Adding a sixth chain reporting 180 makes votes equal to Counter({315: 4, 325: 1, 180: 1}). most_common(1)[0] still returns (315, 4), because 315 still has strictly more votes than any other single value — one additional, differently-wrong outlier does not change the majority answer. This illustrates why self-consistency is robust to a single bad sample: it takes correlated errors, not just one more independent wrong answer, to flip a clear majority.
5. No. A correct final answer alongside a fluent, plausible-looking derivation is consistent with the derivation being faithful, but it does not prove it. Turpin et al. specifically show cases where a model's stated reasoning is shaped by features of the prompt it never mentions, meaning the same correct answer could have been reached (or merely rationalized after the fact) independently of the steps shown. Correctness of the final answer and faithfulness of the stated reasoning are two separate properties, and verifying one does not verify the other; treating a nice-looking chain as proof of how the answer was reached is exactly the failure mode this result warns against.
6. Chain-of-thought only helps if the model's own intermediate generations are more likely to be correct than an immediate guess would be — the technique buys extra compute, but that compute is only useful if each additional pass reliably produces a correct next step. Smaller models have less capacity to reliably execute even single-step arithmetic or logical operations, so their self-generated intermediate steps are more likely to already contain an error; feeding a wrong or malformed intermediate result back into the model as context for the next step can then compound the mistake across the following passes rather than correcting it. The extra forward passes are real, but "more compute spent per problem" only helps when the thing being computed at each step is trustworthy, and that reliability is exactly the property that emerges with scale.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where chain-of-thought reasoning: enabling complex step-by-step problem solving is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting chain-of-thought reasoning: enabling complex step-by-step problem solving to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind chain-of-thought reasoning: enabling complex step-by-step problem solving, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.