A transformer-based LLM decoder has L = 24 layers, model dimension d_model = 1024 (16 attention heads × 64-dimensional head vectors), batch size B = 8, and a maximum cached context length T = 2048 tokens. The KV cache stores one key vector and one value vector per head per layer per token, all in FP16 (2 bytes per value, with 1 GiB = 2^30 bytes). Given this setup, what is the total KV-cache memory footprint, and which of batching, speculative decoding, and grouped-query attention actually reduces that footprint?
The KV cache holds 2 × 24 layers × 8 sequences × 2048 tokens × 1024 dimensions × 2 bytes = 1,610,612,736 bytes ≈ 1.5 GiB total, and grouped-query attention reduces this footprint by letting several query heads share one set of key/value heads, cutting the number of stored KV vectors per token independent of batch size or sequence length.
Counting only the value vectors and forgetting the separate key vectors gives 805,306,368 bytes ≈ 0.75 GiB, and increasing the batch size shrinks this footprint because parallel sequences would share one cache allocation across the whole batch.
This configuration also totals 1,610,612,736 bytes ≈ 1.5 GiB, but speculative decoding is what shrinks the footprint, since verifying several draft tokens per forward pass means the target model ends up storing key/value entries for fewer tokens overall.
Because self-attention's compute cost scales as O(T²), the KV cache itself grows quadratically with sequence length, yielding roughly 2 × 24 × 8 × 2048² × 1024 × 2 bytes ≈ 3 TiB, so shortening the maximum context length is the only lever that controls cache growth.
Answer: A. The KV cache holds 2 × 24 layers × 8 sequences × 2048 tokens × 1024 dimensions × 2 bytes = 1,610,612,736 bytes ≈ 1.5 GiB total, and grouped-query attention reduces this footprint by letting several query heads share one set of key/value heads, cutting the number of stored KV vectors per token independent of batch size or sequence length.
ExplanationKV-cache memory scales linearly, not quadratically, in every one of its dimensions: it must hold one key vector and one value vector — hence the factor of 2 — for every layer, every sequence in the batch, and every cached token, each of length d_model = 1024 in FP16 (2 bytes each). Multiplying step by step: 2 (K and V) × 24 layers × 8 sequences × 2048 tokens × 1024 dimensions × 2 bytes/value = 1,610,612,736 bytes, and dividing by 2^30 bytes/GiB gives exactly 1.5 GiB. Batching does not shrink this number — it multiplies it, since B appears as a direct linear factor, which is precisely why batching trades memory for throughput rather than reducing memory; sequences in a batch each need their own KV entries, they cannot share one allocation. Speculative decoding changes how many forward passes are needed to produce a token, not how many tokens end up in the final generated sequence, so it leaves the number of cached KV entries — and hence the cache size — completely unchanged; its benefit is latency, not memory footprint. Self-attention's O(T²) cost applies to the attention score matrix computed on the fly during a single forward pass, not to the KV cache itself, which stores only one K/V pair per token and therefore scales linearly in T; treating it as quadratic overstates the true 1.5 GiB cache by a factor of T (2048×), landing at 3 TiB instead. What actually reduces the per-token KV footprint without touching L, B, or T is cutting the number of distinct key/value heads that must be stored, which is exactly what grouped-query attention achieves by letting multiple query heads share a single key/value head.
Question 22 · Prompt Engineering · hard
A benchmark evaluation applies three prompt-engineering techniques to a model with baseline accuracy 75% (so baseline error rate = 25%) in sequence: chain-of-thought prompting first, self-consistency sampling second, and few-shot prompting third. Each technique produces an independent relative reduction in the *current* error rate — chain-of-thought reduces error by 20%, self-consistency by 25%, and few-shot by 10% — so each reduction is applied multiplicatively to whatever error rate remains after the previous step. What is the final accuracy after all three techniques are applied in this sequence?
Sequential compounding turns the 25% baseline error into 20% after chain-of-thought, 15% after self-consistency, and 13.5% after few-shot, for a final accuracy of 86.5%.
Summing all three relative error reductions into one combined 55% cut and applying it in a single step drops the error to 11.25%, for a final accuracy of 88.75%.
Adding each relative-reduction percentage directly onto baseline accuracy gives 75 + 20 + 25 + 10 = 130%, which the model reports as a capped final accuracy of 100%.
Because multiplicative error reduction is order-dependent, reversing the sequence to self-consistency, then chain-of-thought, then few-shot changes the final accuracy to 87%.
Answer: A. Sequential compounding turns the 25% baseline error into 20% after chain-of-thought, 15% after self-consistency, and 13.5% after few-shot, for a final accuracy of 86.5%.
ExplanationCompounding relative error reductions means each percentage is applied to the error rate remaining after the previous step, not to the original baseline. Starting error = 100% − 75% = 25%. Chain-of-thought cuts this by 20%, leaving 25% × (1 − 0.20) = 20% error (80% accuracy). Self-consistency then cuts the remaining 20% error by 25%, leaving 20% × (1 − 0.25) = 15% error (85% accuracy). Few-shot then cuts the remaining 15% error by 10%, leaving 15% × (1 − 0.10) = 13.5% error, so final accuracy = 100% − 13.5% = 86.5%. Combining the three relative reductions into a single 55% cut applied once (20+25+10) is a common but incorrect shortcut — it overstates the benefit because it ignores that each later reduction acts on an already-shrunk error rate, producing 88.75% instead of the correct 86.5%. Treating the reduction percentages as direct accuracy percentage-point gains (75+20+25+10) is a different, more severe error — it isn't dimensionally valid, since the reductions are defined relative to the error rate, not as additive points on the accuracy scale. Because multiplication is commutative, the order in which the three multiplicative factors (0.80, 0.75, 0.90) are applied does not change the product, so reordering the techniques still yields 13.5% error and 86.5% accuracy, not a different result.
Question 23 · Model Compression · hard
Analyze post-training quantization where a 7-billion parameter model trained in float32 is quantized to int8 (1 byte per parameter) using min-max scaling with clipping range [−127, 127]. Calculate the memory reduction, estimate the accuracy drop given empirical quantization noise, and determine whether per-channel vs per-tensor quantization provides better accuracy preservation at the cost of increased memory overhead?
Float32 weights require 28 GB and int8 weights require 7 GB (4× reduction), but per-tensor quantization preserves accuracy just as well as per-channel quantization, since a single scale factor is mathematically equivalent for any weight distribution.
Quantization to int8 reduces memory from 28 GB to 7 GB (4×), but quantization error is catastrophic, causing 20+ percent accuracy drops on most large language models, making int8 impractical without extra calibration techniques.
Memory drops from 28 GB (float32) to 7 GB (int8), a 4× reduction; well-calibrated int8 quantization typically costs only 1 to 2 percentage points of accuracy, and per-channel quantization preserves accuracy better than per-tensor quantization by using a separate scale per output channel, at the cost of a negligible (under 0.1 percent) memory overhead for the extra scale values.
Post-training quantization provides no meaningful compression since int8 and float32 require similar memory footprints once scale and zero-point storage is counted, making quantization-aware training the only way to shrink model size.
Answer: C. Memory drops from 28 GB (float32) to 7 GB (int8), a 4× reduction; well-calibrated int8 quantization typically costs only 1 to 2 percentage points of accuracy, and per-channel quantization preserves accuracy better than per-tensor quantization by using a separate scale per output channel, at the cost of a negligible (under 0.1 percent) memory overhead for the extra scale values.
ExplanationFirst, float32→int8: 7B×4 bytes = 28 GB → 7B×1 byte = 7 GB, a 4× compression. Per-tensor quantization uses one scale/zero-point for the whole matrix, so outliers in any channel force a wide clipping range that hurts every other channel's precision. Per-channel quantization instead keeps a separate scale for each output channel (~768 per layer in a typical transformer), so one channel's outliers no longer degrade the rest, meaningfully reducing quantization error. The extra scale values cost roughly 768×4 bytes ≈ 3 KB per layer, well under 0.1% of the 28 GB original footprint — a negligible price for the accuracy gained. A is wrong because per-tensor and per-channel scaling are not equivalent: a single scale cannot fit varying per-channel distributions as well. B is wrong because well-calibrated int8 post-training quantization typically costs only 1-2 percentage points of accuracy, not 20+. D is wrong because int8 clearly compresses 4× and the small scale-storage overhead does not erase that saving. C correctly gives the 4× memory reduction, the realistic accuracy cost, and per-channel's accuracy advantage over per-tensor at negligible memory overhead.
Question 24 · Model Compression · hard
A teacher model reaches 90 percent accuracy on CIFAR-100, while a student model trained from scratch reaches 70 percent. After knowledge distillation at temperature T=3, the student's accuracy rises to 82 percent, so what is the student's knowledge transfer rate (actual improvement divided by the maximum possible improvement toward the teacher), and what does that rate reveal about the distillation?
Dividing the student's actual 12-point gain (70→82) by the maximum possible 20-point gain (70→90) gives a transfer rate of 60 percent, meaning the student recovered most of the achievable improvement but retained a real capacity gap despite the soft-label training via temperature scaling.
Comparing the student's final 82 percent directly against the teacher's 90 percent yields a transfer rate of roughly 91 percent, suggesting the student essentially matched the teacher's performance after distillation.
The 12-point accuracy gain expressed as a fraction of the teacher's absolute 90 percent accuracy works out to about 13 percent, indicating that distillation transferred only a minor sliver of the teacher's overall capability.
Inverting the ratio so the 20-point maximum gap sits over the 12-point actual gain produces a value near 167 percent, which would mean the student surpassed the theoretical ceiling for improvement.
Answer: A. Dividing the student's actual 12-point gain (70→82) by the maximum possible 20-point gain (70→90) gives a transfer rate of 60 percent, meaning the student recovered most of the achievable improvement but retained a real capacity gap despite the soft-label training via temperature scaling.
ExplanationKnowledge distillation compares the student's baseline accuracy (70%) to its post-distillation accuracy (82%) and to the theoretical ceiling set by the teacher (90%). The actual gain is 82−70 = 12 percentage points, and the maximum gain available before matching the teacher is 90−70 = 20 percentage points, so the transfer rate is 12/20 = 60 percent. This means the student closed 60% of the gap between its own scratch-trained performance and the teacher's, a solid but incomplete recovery — consistent with a smaller student network only partially absorbing the teacher's "dark knowledge" (the soft, temperature-scaled probability distribution over all 100 classes, per Hinton et al. 2015) rather than merely memorizing hard labels. Comparing the student's raw post-distillation score to the teacher's raw score (82/90 ≈ 91%) conflates absolute performance with relative improvement and ignores the student's starting point entirely, so it overstates how much was actually transferred. Dividing the 12-point gain by the teacher's absolute accuracy (12/90 ≈ 13%) mixes a percentage-point difference with an unrelated percentage base, understating how much of the achievable gap was actually closed. Inverting the numerator and denominator (20/12 ≈ 167%) produces a transfer rate exceeding 100 percent, which is logically impossible since a student cannot recover more improvement than the maximum available before reaching teacher-level accuracy.
Question 25 · Scaling Laws · hard
A language model's validation loss follows the power-law scaling relation L(C) = L₀ · (C / C₀)^(−α), where C is the training compute in FLOPs. A team measures a baseline compute C₀ = 1×10^19 FLOPs giving a loss L₀ = 4.0 nats, with scaling exponent α = 0.5. If they increase compute fourfold to C₁ = 4×10^19 FLOPs, what is the resulting loss L₁, computed correctly from the power-law relation?
Since 4^(−0.5) equals 0.5, the loss falls to L₁ = 2.0 nats — exactly half of L₀.
Applying a positive exponent instead of the specified negative one gives L₁ = 8.0 nats, implying loss worsens as compute grows.
Treating the exponent as −1 rather than −0.5 yields L₁ = 1.0 nat, a quartering of the original loss.
Subtracting α·ln(4) from L₀ in a log-linear model gives L₁ ≈ 3.31 nats, describing an additive rather than multiplicative decline.
Answer: A. Since 4^(−0.5) equals 0.5, the loss falls to L₁ = 2.0 nats — exactly half of L₀.
ExplanationThe scaling law is multiplicative, not additive: L(C) = L₀ · (C/C₀)^(−α). With C₁/C₀ = 4 and α = 0.5, the compute ratio raised to the power −0.5 is 4^(−0.5) = 1/√4 = 1/2 = 0.5, so L₁ = 4.0 nats × 0.5 = 2.0 nats — the loss is exactly halved. Flipping the sign of the exponent (treating it as +0.5) would incorrectly predict loss rising to 8.0 nats as compute grows, which contradicts the entire premise of scaling laws — more compute should reduce loss, not increase it. Using −1 instead of −0.5 as the exponent conflates a much steeper scaling regime, giving 4.0 × 0.25 = 1.0 nat rather than the correct 2.0 nats. Subtracting α·ln(C₁/C₀) = 0.5 × ln(4) ≈ 0.693 from L₀ to get ≈3.31 nats mistakes the power law for a log-linear (additive) decay model — real scaling laws are log-log linear (linear in log(L) vs log(C)), not linear in L vs log(C).
Question 26 · LoRA and Parameter-Efficient Fine-Tuning · hard
Evaluate the trade-offs in fine-tuning strategy where full fine-tuning updates all 7 billion parameters, LoRA (rank-8) updates only 131K parameters, and adapter modules update 1.3 million parameters. Calculate the gradient computation and storage overhead for each method (assuming standard backprop), measure the reduction in optimizer states (Adam requires 2× parameters for momentum and variance), and determine the practical memory bottleneck for each approach?
Full fine-tuning: gradients 28 GB, optimizer states 56 GB (7B×2 for Adam), total ≈ 84 GB.
Full fine-tuning: model params 28 GB + gradients 28 GB + optimizer states 56 GB = 112 GB excluding activations.
LoRA and adapters both fit in GPU memory, but full fine-tuning requires CPU offloading, making gradient checkpointing necessary for all approaches.
Answer: C. Full fine-tuning: model params 28 GB + gradients 28 GB + optimizer states 56 GB = 112 GB excluding activations.
ExplanationMemory accounting for training a 7B-parameter model, assuming float32 (4 bytes/param): 7B × 4 bytes = 28 GB for the base model weights, and gradients require another 28 GB since one gradient exists per trainable parameter. For full fine-tuning, all 7B parameters are trainable, so Adam's optimizer states (momentum + variance, each sized to the parameter count) add 2 × 28 GB = 56 GB. Total memory = 28 GB (weights) + 28 GB (gradients) + 56 GB (optimizer states) = 112 GB, excluding activations. LoRA trains only 131K parameters while the same 28 GB base model stays frozen in memory; the trainable slice needs roughly 131K × 4 bytes ≈ 0.5 MB each for parameters, gradients, and optimizer states, for about 2 MB of trainable-parameter memory — a reduction of roughly four orders of magnitude versus full fine-tuning's 112 GB. Full fine-tuning's total comes out wrong if gradients are left out entirely (84 GB) or if Adam's per-parameter overhead is miscounted as 4× instead of 2× (168 GB); the 112 GB figure is the one that correctly sums every term — weights, gradients, and optimizer states — without double-counting or omission.
Question 27 · LLM Inference Optimization · hard
An LLM inference server uses continuous batching, where each decoding step advances every sequence in the current batch by exactly one token, so the step latency equals the per-token latency delivered to each individual sequence in that batch. Measured step latencies are 10 milliseconds at batch size 1 (aggregate throughput 100 tokens/second), 25 milliseconds at batch size 8 (aggregate throughput 320 tokens/second), and 64 milliseconds at batch size 32 (aggregate throughput 500 tokens/second). A chat application requires each user to receive successive tokens no more than 30 milliseconds apart while the operator wants to maximize aggregate server throughput; which batch size should be selected, and why?
Batch size 8 is the right choice because its 25-millisecond per-token latency stays under the 30-millisecond requirement while its 320 tokens/second aggregate throughput is more than triple what batch size 1 delivers, and batch size 32 must be excluded despite its higher raw throughput because its 64-millisecond latency breaks the per-user SLA.
Batch size 32 is actually optimal once per-token latency is correctly computed as step latency divided by batch size, which gives an effective latency of exactly 2 milliseconds (64 ÷ 32) for each sequence, so it comfortably meets the SLA while also producing the highest throughput of the three options.
Batch size 1 should be selected because it has the smallest step latency at 10 milliseconds, and whenever any option satisfies a hard latency requirement, an operator should always choose the configuration with the lowest possible latency rather than optimizing throughput any further.
Maximizing throughput alone favors batch size 32, since 500 tokens per second exceeds both other configurations, and prioritizing total server throughput over per-user latency would make it the correct pick once a request enters the batch.
Answer: A. Batch size 8 is the right choice because its 25-millisecond per-token latency stays under the 30-millisecond requirement while its 320 tokens/second aggregate throughput is more than triple what batch size 1 delivers, and batch size 32 must be excluded despite its higher raw throughput because its 64-millisecond latency breaks the per-user SLA.
ExplanationIn continuous batching, every sequence active in a decoding step receives exactly one new token when that step completes, so the measured step latency is the per-token latency each user experiences — it is not further divided by batch size, because the step's cost is already shared in parallel across all sequences rather than serialized per sequence. Applying the 30-millisecond SLA: batch size 1 (10 ms/token, 100 tokens/second) and batch size 8 (25 ms/token, 320 tokens/second) both satisfy it, but batch size 32 (64 ms/token, 500 tokens/second) does not, since 64 ms exceeds the 30-millisecond ceiling. Among the two SLA-compliant configurations, batch size 8 delivers 320 tokens/second versus only 100 tokens/second at batch size 1, so it maximizes throughput without breaching the latency requirement. Dividing the 64-millisecond step latency by 32 to get 2 milliseconds is a category error: that division would only be valid if the 64 milliseconds were spent serving one sequence at a time, but continuous batching processes all 32 sequences concurrently within that same 64-millisecond window, so each sequence still waits the full 64 milliseconds for its next token.
Question 28 · Transformer Architecture · hard
A Transformer encoder layer uses d_model = 512, 8 attention heads (each head dimension = 64), and a feed-forward sublayer with inner dimension d_ff = 2048; all projection matrices are linear layers without bias terms. The multi-head self-attention sublayer uses one combined d_model×d_model matrix each for queries, keys, values, and the output projection (the 8 heads split the 512 dimensions into groups of 64 rather than each head owning a separate full-sized matrix), while the feed-forward sublayer has two linear layers, d_model→d_ff and d_ff→d_model. Calculate the total trainable parameters in each sublayer, determine which sublayer contains more parameters, and explain what this implies about parameter count versus computational scaling with sequence length n?
The self-attention sublayer has 4×512² = 1,048,576 parameters, while the feed-forward sublayer has only 2×512² = 524,288 parameters, so self-attention contains more parameters than the feed-forward sublayer at this configuration.
The feed-forward sublayer has 2×512×2048 = 2,097,152 parameters, roughly twice the self-attention sublayer's 4×512² = 1,048,576 parameters, so feed-forward parameters dominate even though attention's O(n²·d_model) cost scales quadratically with sequence length while feed-forward's O(n·d_model²) cost does not.
Each of the 8 attention heads requires its own full 512×512 projection matrices, giving self-attention 8×4×512² = 8,388,608 parameters, far exceeding the feed-forward sublayer's 2,097,152 parameters.
Since self-attention's complexity is O(n²·d_model) and feed-forward's is O(n·d_model²), the two sublayers must contain equal parameter counts by design, each totaling 2,097,152 parameters regardless of sequence length.
Answer: B. The feed-forward sublayer has 2×512×2048 = 2,097,152 parameters, roughly twice the self-attention sublayer's 4×512² = 1,048,576 parameters, so feed-forward parameters dominate even though attention's O(n²·d_model) cost scales quadratically with sequence length while feed-forward's O(n·d_model²) cost does not.
ExplanationSelf-attention's Q, K, V, and output projections each use one combined d_model×d_model matrix — the 8 heads split the 512 dimensions into groups of 64, they do not each get a separate full-sized matrix — giving 4×512×512 = 1,048,576 parameters total. The feed-forward sublayer's two layers contribute 512×2048 = 1,048,576 and 2048×512 = 1,048,576 parameters, for a combined 2×512×2048 = 2,097,152 — almost exactly double the attention sublayer's count. So the feed-forward sublayer holds more trainable weights. Computationally the relationship flips for long sequences: self-attention costs O(n²·d_model) because every token attends to every other token, while the feed-forward sublayer costs O(n·d_model²) since the same weights are applied independently at each position. As n grows large, attention's compute eventually outpaces feed-forward's even though attention has fewer fixed parameters — parameter count and computational cost scale independently and can favor different sublayers depending on sequence length.
Question 29 · Transfer Learning · hard
Analyze a zero-shot transfer scenario where a model trained on ImageNet (1000 classes) is applied to novel classes (e.g., bird species) without fine-tuning, using text embeddings from a language model to describe target classes. Assess the expected performance gap compared to fine-tuned models, evaluate why semantic text embeddings enable zero-shot transfer, and determine whether CLIP-style vision-language models improve zero-shot accuracy compared to visual features alone?
Visual-only zero-shot transfer underperforms fine-tuned models by a clear margin, but adding text embeddings provides no real benefit—class names carry no information beyond what image similarity already captures—so CLIP-style joint training performs about the same as vision-only zero-shot.
Zero-shot transfer cannot work at all, since a model that has never seen a single labeled example of the target class cannot produce any usable prediction, and neither text embeddings nor CLIP-style vision-language training can narrow that gap with fine-tuned models.
Text-embedding zero-shot transfer matches fully fine-tuned accuracy, because semantic class descriptions substitute completely for labeled training examples, which means CLIP-style vision-language pretraining offers no additional advantage once text embeddings are already in use.
Fine-tuned models retain a clear but not overwhelming accuracy advantage over zero-shot transfer, because text embeddings let the model relate unseen classes to concepts it already knows (a sparrow is a kind of bird) instead of requiring labeled examples, and CLIP-style vision-language models — jointly trained on paired image-text data — consistently outperform zero-shot transfer that relies on visual features alone, since their embedding space was optimized specifically for cross-modal matching.
Answer: D. Fine-tuned models retain a clear but not overwhelming accuracy advantage over zero-shot transfer, because text embeddings let the model relate unseen classes to concepts it already knows (a sparrow is a kind of bird) instead of requiring labeled examples, and CLIP-style vision-language models — jointly trained on paired image-text data — consistently outperform zero-shot transfer that relies on visual features alone, since their embedding space was optimized specifically for cross-modal matching.
ExplanationZero-shot transfer applies a model to classes it has never been trained on, without any fine-tuning on labeled examples from those classes. This inherently caps accuracy below what fine-tuning could achieve, since fine-tuning has direct supervision from the target distribution while zero-shot transfer does not — but that gap is not so large as to make the approach useless, especially once text embeddings are involved. Text embeddings help because a language model encodes semantic relationships between concepts (a sparrow is a kind of bird, birds have wings and beaks), so the vision model can map an unfamiliar class onto a description built from concepts it effectively already understands, rather than needing to see a single labeled example of that class. CLIP-style vision-language models take this further: because they are pretrained jointly on paired image-text data, their shared embedding space is explicitly optimized for matching images to text descriptions, which consistently gives them an edge over zero-shot approaches that rely on visual features alone (e.g., nearest-neighbor matching to ImageNet class prototypes) and have no comparable cross-modal grounding. The claim that text embeddings add nothing beyond visual features, the claim that zero-shot transfer is impossible outright, and the claim that zero-shot can match fully fine-tuned accuracy are each contradicted by this reasoning: text embeddings measurably help, some transfer is possible without labeled examples, and fine-tuning's direct supervision still holds a real, if bounded, advantage.
Question 30 · Mixture of Experts · hard
In a Mixture of Experts (MoE) transformer with 8 experts per layer, a top-2 gating mechanism, and 64 layers, calculate the total number of expert activations per forward pass for a single token, and evaluate why sparse activation enables scaling model capacity without proportionally increasing compute cost?
8×64 = 512 expert calls per token because every expert in every layer must process the input to maintain model quality and consistency across the expert pool
64×64 = 4096 because MoE requires all layers to communicate with all experts in adjacent layers through cross-expert attention mechanisms
1×64 = 64 because only the highest-scoring expert processes each token, and the second expert serves as a backup that doesn't contribute to the final output
Total activations = 2×64 = 128 expert forward passes per token because top-2 gating selects only 2 of 8 experts per layer, meaning each token uses 25% of total parameters while the model stores 8× more knowledge than a dense equivalent of equal compute
Answer: D. Total activations = 2×64 = 128 expert forward passes per token because top-2 gating selects only 2 of 8 experts per layer, meaning each token uses 25% of total parameters while the model stores 8× more knowledge than a dense equivalent of equal compute
ExplanationWith top-2 gating, each token is routed to exactly 2 of the 8 experts in every layer, so a single layer contributes 2 expert activations. Across all 64 layers, the total is 2 × 64 = 128 expert forward passes per token. Because only 2 of 8 experts (25%) fire per layer, per-token compute scales with the number of active experts, not the total expert count — the model can store roughly 8× the parameters of a dense model with equal per-token compute, since the unused experts contribute zero FLOPs for that token. This decoupling of total parameter count from per-token compute is what lets MoE scale capacity without a proportional rise in inference cost.
Question 31 · RLHF and LLM Alignment · hard
Consider a DPO (Direct Preference Optimization) training setup where the policy model π_θ is trained on preference pairs (y_w, y_l). The DPO loss is L = -log σ(β(log π_θ(y_w)/π_ref(y_w) - log π_θ(y_l)/π_ref(y_l))). If β=0.1 and the log-ratio difference equals 5.0, what is the approximate loss value, and analyze how DPO eliminates the need for a separate reward model compared to standard RLHF?
Loss ≈ 2.5 because DPO divides the log-ratio by 2 before applying sigmoid, and the loss equals the scaled ratio directly without the logarithmic transformation
Loss ≈ 0.007 because the argument to sigmoid is β×5.0 = 0.5, giving σ(0.5) ≈ 0.622, but DPO inverts this probability before taking the log yielding -log(1-0.622) ≈ 0.007
Loss ≈ 5.0 because the sigmoid function saturates at this input range and the loss equals the raw log-ratio difference without any transformation or scaling
Loss ≈ 0.474 because σ(0.5) ≈ 0.622 and -log(0.622) ≈ 0.474, since DPO reparameterizes the reward as r(y) = β×log(π_θ(y)/π_ref(y)), collapsing the reward model into the policy itself and optimizing the closed-form preference probability directly
Answer: D. Loss ≈ 0.474 because σ(0.5) ≈ 0.622 and -log(0.622) ≈ 0.474, since DPO reparameterizes the reward as r(y) = β×log(π_θ(y)/π_ref(y)), collapsing the reward model into the policy itself and optimizing the closed-form preference probability directly
ExplanationFirst, loss ≈ 0.474. The computation: β × (log-ratio difference) = 0.1 × 5.0 = 0.5. Then, σ(0.5) = 1/(1+exp(-0.5)) = 1/(1+0.6065) ≈ 0.622. Loss = -log(0.622) ≈ 0.474. DPO's key insight is that the optimal RLHF policy has a closed-form relationship to the reward: r*(y) = β×log(π*(y)/π_ref(y)) + C. Therefore, instead of training a separate reward model then running PPO, DPO directly optimizes the policy using the Bradley-Terry preference model with the policy's own log-ratios as implicit rewards. Finally, this eliminates reward model training, reward hacking, and PPO instability — making alignment simpler and more stable.
Question 32 · Federated Learning · hard
A hospital consortium trains a diagnostic model using federated learning across 500 hospital edge servers. Each local model has 20 million parameters stored in float32 precision. The central server runs FedAvg, aggregating once after every device completes 8 local training epochs. What is the total communication cost (upload plus download) for a single federated round, and how does the number of local epochs per round affect that cost?
The total per-round cost is 80 GB, since 500 devices each upload an 80 MB model (20M parameters times 4 bytes) and download an 80 MB aggregated model, giving 500 times 80 MB times 2 equals 80 GB, and running more local epochs before that single synchronization step adds no communication because only one upload and one download happen per round regardless of local epoch count.
Because float32 values require 8 bytes each, the round costs 500 times 20M times 8 bytes times 2, which equals 160 GB, and doubling the local epoch count to 16 would double this figure to 320 GB since each additional epoch adds its own communication round.
The relevant figure is only the upload direction, giving 500 times 20M times 4 bytes equals 40 GB for the round, and increasing local epochs raises the per-round communication cost proportionally because more epochs mean more gradient updates must be transmitted to the server.
Since FedAvg only ever transmits one aggregated global model rather than 500 separate copies, the round costs just 20M times 4 bytes times 2, which is 160 MB, and the number of local epochs has no bearing on this because aggregation always compresses all client updates into a single parameter set before any transfer occurs.
Answer: A. The total per-round cost is 80 GB, since 500 devices each upload an 80 MB model (20M parameters times 4 bytes) and download an 80 MB aggregated model, giving 500 times 80 MB times 2 equals 80 GB, and running more local epochs before that single synchronization step adds no communication because only one upload and one download happen per round regardless of local epoch count.
ExplanationEach device's local model has 20 million parameters stored in float32, and float32 uses 4 bytes per value, so one model copy is 20,000,000 x 4 bytes = 80 MB. With 500 devices, uploading their locally-trained models costs 500 x 80 MB = 40 GB, and the server then broadcasting the newly aggregated global model back to all 500 devices costs another 40 GB. Combining both directions gives 500 x 80 MB x 2 = 80 GB for the round. This total depends only on the number of participating devices, the parameter count, and the byte-width of the numeric format, because FedAvg synchronizes exactly once per round -- so whether each device runs 8 local epochs or 80 before that synchronization, the bytes actually transferred stay at 80 GB; only the wall-clock compute time per round changes. This is precisely why communication-efficient FL techniques (gradient sparsification, quantization, and deliberately running more local epochs to amortize a fixed communication cost over more local computation) are active research areas: the per-round transfer volume is invariant to local epoch count, but the number of rounds needed to reach convergence is not.
Question 33 · Diffusion Models · hard
In a diffusion model's forward process, x_t = √ᾱ_t · x_0 + √(1-ᾱ_t) · ε, where ε ~ N(0, I) is standard Gaussian noise and the training data x_0 has been normalized to unit variance. At a given timestep t, ᾱ_t = 0.64. What is the signal-to-noise ratio (SNR), defined as Var(signal component)/Var(noise component), of x_t at this timestep?
The SNR equals approximately 1.78, since the signal component √ᾱ_t·x_0 contributes variance ᾱ_t = 0.64 while the noise component √(1-ᾱ_t)·ε contributes variance (1-ᾱ_t) = 0.36, giving SNR = 0.64/0.36 ≈ 1.78
Squaring is unnecessary here, so the SNR is found directly from the coefficients themselves as √ᾱ_t divided by √(1-ᾱ_t), which equals 0.8/0.6 ≈ 1.33
Since ᾱ_t already represents the fraction of variance retained from the original signal, the SNR is simply ᾱ_t itself, so SNR ≈ 0.64
Because noise-to-signal comparisons are conventionally built the opposite way, the SNR should be computed as (1-ᾱ_t)/ᾱ_t = 0.36/0.64 ≈ 0.56
Answer: A. The SNR equals approximately 1.78, since the signal component √ᾱ_t·x_0 contributes variance ᾱ_t = 0.64 while the noise component √(1-ᾱ_t)·ε contributes variance (1-ᾱ_t) = 0.36, giving SNR = 0.64/0.36 ≈ 1.78
ExplanationThe forward diffusion process defines x_t as a weighted mix of the original data and Gaussian noise, with weights √ᾱ_t and √(1-ᾱ_t) respectively. Variance scales with the square of a coefficient, not the coefficient itself, so the signal variance is (√ᾱ_t)² · Var(x_0) = ᾱ_t · 1 = 0.64, and the noise variance is (√(1-ᾱ_t))² · Var(ε) = (1-ᾱ_t) · 1 = 0.36. SNR = Var(signal)/Var(noise) = 0.64/0.36 = 16/9 ≈ 1.78. This is exactly why SNR is written as ᾱ_t/(1-ᾱ_t) in diffusion model literature: as t increases and ᾱ_t shrinks toward 0, the SNR collapses toward 0, meaning the signal is progressively overwhelmed by noise — precisely the mechanism the reverse (denoising) process must learn to undo at every step. A common error is working with the standard-deviation coefficients (√ᾱ_t and √(1-ᾱ_t)) directly instead of squaring them to get variances, which understates how quickly the process becomes noise-dominated as t approaches T.
Question 34 · Explainable AI (SHAP, LIME, attention viz) · hard
A gradient boosting ensemble for a credit-approval model has 50 trees, 8 input features, and is evaluated for explainability on 500 test samples. Exact Shapley value computation requires evaluating the model on all 2^n possible feature coalitions independently for every tree, with results combined across all trees and samples. How many total model evaluations does exact Shapley computation require, and why does this make TreeSHAP's polynomial-time algorithm necessary for production deployment?
Total evaluations equal 2^8 × 500 = 128,000, since coalition enumeration happens once per sample regardless of tree count, meaning the number of trees in the ensemble has no effect on the exponential term.
The exact computation requires 2^8 × 50 × 500 = 6,400,000 model evaluations, because all 256 feature coalitions must be evaluated separately for each of the 50 trees across every one of the 500 samples, and TreeSHAP replaces this exponential enumeration with a single polynomial-time traversal of each tree's leaves.
Using 8! × 500 = 20,160,000 evaluations is correct, because the Shapley value formula is defined as a sum over all orderings (permutations) of the features rather than over feature subsets, so factorial growth is the true exact-computation cost.
A cost of 8 × 50 × 500 = 200,000 evaluations applies, because tree-based ensembles let each feature's marginal contribution be computed independently of every other feature, avoiding the need to enumerate coalitions at all.
Answer: B. The exact computation requires 2^8 × 50 × 500 = 6,400,000 model evaluations, because all 256 feature coalitions must be evaluated separately for each of the 50 trees across every one of the 500 samples, and TreeSHAP replaces this exponential enumeration with a single polynomial-time traversal of each tree's leaves.
ExplanationExact Shapley value computation must evaluate the model on every one of the 2^8 = 256 possible feature coalitions, and since these coalition evaluations are carried out with respect to each tree in the ensemble before the results are combined, the enumeration is repeated across all 50 trees and all 500 test samples: 256 × 50 × 500 = 6,400,000 model evaluations. The wrong answers arise from three common errors: forgetting to multiply by the number of trees (giving 128,000), confusing subset enumeration with permutation enumeration by using 8! instead of 2^8 (giving 20,160,000), and assuming tree ensembles let feature contributions be computed independently and additively rather than through coalitions (giving 200,000). TreeSHAP avoids the exponential blowup entirely: instead of enumerating coalitions explicitly, it propagates weighted contributions through each tree's internal nodes in a single traversal per tree, so its cost is polynomial in the number of trees, leaves, and tree depth rather than exponential in the number of features. This structural shortcut — not any loss of accuracy — is why TreeSHAP, rather than brute-force Shapley enumeration, is the algorithm used in production explainability tools such as the `shap` Python library for tree ensembles.
Question 35 · LoRA and Parameter-Efficient Fine-Tuning · hard
In LoRA (Low-Rank Adaptation) applied to a transformer with weight matrix W ∈ ℝ^{4096×4096}, if the LoRA rank r=16 and adapters are applied to Q, K, V, and output projections in all 32 layers, calculate the total trainable parameters and compare to full fine-tuning parameter count?
LoRA params = 32 × 4 × 16 = 2048 because only the rank value is stored per projection per layer, and full fine-tuning trains the same 2048 parameters but with higher precision
LoRA params = 16 × 16 × 32 = 8192 because the rank-16 decomposition creates a 16×16 bottleneck matrix per layer, and the 4096-dimensional projections are computed from this compressed representation
LoRA params = 32 × 4 × 4096 × 4096 = 2.15B because LoRA still modifies the full weight matrix but uses a different optimizer, not actually reducing parameter count
LoRA params = 32 × 4 × (4096×16 + 16×4096) = 16,777,216 (≈16.8M) vs full fine-tuning's 32 × 4 × 4096² ≈ 2.15B, achieving a 128× parameter reduction because each W is decomposed as W + BA where B∈ℝ^{4096×16} and A∈ℝ^{16×4096}, training only the low-rank factors while freezing the original weights
Answer: D. LoRA params = 32 × 4 × (4096×16 + 16×4096) = 16,777,216 (≈16.8M) vs full fine-tuning's 32 × 4 × 4096² ≈ 2.15B, achieving a 128× parameter reduction because each W is decomposed as W + BA where B∈ℝ^{4096×16} and A∈ℝ^{16×4096}, training only the low-rank factors while freezing the original weights
ExplanationFirst, LoRA decomposes the weight update ΔW into two low-rank matrices: B∈ℝ^{4096×16} and A∈ℝ^{16×4096}. Parameters per projection = 4096×16 + 16×4096 = 131,072. Then, with 4 projections (Q,K,V,O) × 32 layers: 131,072 × 4 × 32 = 16,777,216 ≈ 16.8M trainable parameters. Full fine-tuning: 4096² × 4 × 32 = 2,147,483,648 ≈ 2.15B. Ratio: 2.15B / 16.8M ≈ 128×. Finally, this is why LoRA is revolutionary — it achieves 97-100% of full fine-tuning quality with only 0.78% of the parameters, dramatically reducing GPU memory (since optimizer states scale with trainable params) and enabling fine-tuning of 7B+ models on a single GPU.
Question 36 · RAG · hard
In a retrieval-augmented generation (RAG) pipeline, a vector database holds M = 1,000,000 document embeddings of dimension d = 768, indexed so that a query can be answered in O(d·log M) operations. A query retrieves k = 5 documents (250 tokens each) plus a 250-token user question, so the LLM's context totals N = 1,500 tokens; the LLM has P = 7 billion parameters, and a forward pass over N tokens costs approximately 2×N×P FLOPs. Using log2(1,000,000) ≈ 19.93, how does the computational cost of retrieval compare to the computational cost of the LLM forward pass?
Searching all 1,000,000 stored embeddings for the top-k=5 matches is the true bottleneck: even with an efficient index, comparing against a million-entry database costs far more than a single 1,500-token LLM forward pass, making retrieval the dominant cost in the pipeline.
Each of the 5 retrieved documents must pass through the LLM in its own separate forward pass alongside the query, so the real cost is 5 × 2 × 500 × 7×10^9 ≈ 3.5×10^13 FLOPs — noticeably more than processing all the context in one pass, meaning retrieval effectively multiplies the LLM's workload by roughly k.
Retrieval needs only about 768 × 19.93 ≈ 15,306 operations under the O(d·log M) index, while the LLM's forward pass over the full 1,500-token context costs about 2 × 1,500 × 7×10^9 = 2.1×10^13 FLOPs — a gap of roughly nine orders of magnitude, so total pipeline cost is governed almost entirely by the LLM, not the search.
Because vector databases like FAISS return results in constant time regardless of how many documents are indexed, retrieval cost stays fixed near zero even as M grows into the billions, making the search step cost-free compared to any LLM call.
Answer: C. Retrieval needs only about 768 × 19.93 ≈ 15,306 operations under the O(d·log M) index, while the LLM's forward pass over the full 1,500-token context costs about 2 × 1,500 × 7×10^9 = 2.1×10^13 FLOPs — a gap of roughly nine orders of magnitude, so total pipeline cost is governed almost entirely by the LLM, not the search.
ExplanationUnder the stated O(d·log M) index (e.g., an HNSW or IVF-based structure), a single retrieval query costs about 768 × 19.93 ≈ 15,306 operations — this is why approximate-nearest-neighbor libraries avoid brute-force comparison against all 1,000,000 stored vectors, and instead only touch a number of candidates proportional to log M. The LLM, by contrast, must run a full forward pass over the concatenated context of all five retrieved passages plus the query, totaling N = 1,500 tokens; at 2×N×P FLOPs per pass, that's 2 × 1,500 × 7×10^9 = 2.1×10^13 FLOPs. Dividing the two costs, 2.1×10^13 ÷ 15,306 ≈ 1.4×10^9, so the LLM pass consumes roughly nine orders of magnitude more computation than the retrieval step — retrieval is effectively free by comparison, and pipeline latency is governed almost entirely by the LLM. The claim that retrieval dominates confuses the size of the database (1,000,000 entries) with the cost of searching it, which scales with log M rather than M once indexed. The claim that each retrieved document triggers its own separate LLM forward pass misunderstands how RAG prompting works: the retrieved passages are concatenated into a single context window and processed together in one pass, not run through the model five separate times, which would overstate the true cost. And while approximate-nearest-neighbor search is fast, it is not literally constant-time — its cost still grows, slowly and logarithmically, with the database size M, so treating it as free regardless of scale overstates what the index actually guarantees.
Question 37 · LLM Optimization · hard
A decoder-only transformer generates a sequence one token at a time (greedy autoregressive decoding). At generation step t, attention at that step must look back over all t tokens produced so far, so the model needs Key (K) and Value (V) vectors ready for all t of them. Each token's K vector and V vector are each produced by a separate linear projection costing 2·d_model² FLOPs, so computing both K and V for one token costs c = 4·d_model² FLOPs. Without a KV-cache, nothing from earlier steps is stored, so the model recomputes K and V for every one of the t tokens seen so far at every single step. With a KV-cache, K and V for all earlier tokens are already stored from previous steps, so the model computes K and V only for the newly generated token at each step. For generating a full sequence of n = 8 tokens, summed across all 8 generation steps, by what factor does the KV-cache reduce the total FLOPs spent on K,V projections?
Summing K,V-projection work across all 8 steps gives a 4.5x reduction: 36 units of work without caching versus 8 units with caching, since without caching step t recomputes K,V for all t tokens seen so far (1+2+...+8 = 36 units of c) while caching computes only the 1 new token's K,V at each of the 8 steps.
Comparing an uncached total of 64 units (treating every one of the 8 steps as reprocessing the full 8-token context) against a cached total of 8 units yields an 8x reduction in K,V-projection FLOPs.
Approximating the uncached workload as n^2/2 = 32 units against a cached total of 8 units puts the K,V-projection FLOPs reduction at 4x.
No reduction in K,V-projection FLOPs actually occurs, because attention must still scan over every previously generated token's K,V at each step whether or not those vectors were cached.
Answer: A. Summing K,V-projection work across all 8 steps gives a 4.5x reduction: 36 units of work without caching versus 8 units with caching, since without caching step t recomputes K,V for all t tokens seen so far (1+2+...+8 = 36 units of c) while caching computes only the 1 new token's K,V at each of the 8 steps.
ExplanationAt step t (t = 1, 2, ..., 8), attention needs K,V for all t tokens generated so far. Without a cache, this work is redone from scratch at every step: step 1 needs K,V for 1 token, step 2 for 2 tokens, and so on up to step 8 needing K,V for 8 tokens, so the total projection work, measured in units of c = 4·d_model² FLOPs per token, is 1+2+3+4+5+6+7+8 = 36 units. With a cache, each step computes K,V for only the single newly generated token, so the total is 8 units (one per step, across 8 steps). The reduction factor is 36/8 = 4.5x. In general this ratio equals (n+1)/2 for a sequence of length n, and it does not depend on d_model at all: d_model² appears identically in both the cached and uncached totals through the constant c, so it cancels out of the ratio. It is worth noting what this speedup does and does not cover: it applies specifically to the linear-projection cost of producing K and V. The attention score computation itself, the dot products between the current query and all previously cached keys, still requires work proportional to t at step t regardless of caching, because caching eliminates redundant recomputation, not the unavoidable need to attend over every position in the growing context.
Question 38 · Transformers · hard
Standard causal self-attention trains in O(N^2) time on a length-N sequence and costs O(N) time per generated token at inference, since each new token must attend over the entire growing cache of past keys and values. Linear attention replaces the softmax kernel in softmax(QK^T)V with a feature map φ so the output can be reordered by matrix associativity as φ(Q)(φ(K)^T V) instead of (φ(Q)φ(K)^T)V, and this reordering lets it train in O(N) time while costing only O(1) time per generated token. Why does this reordering deliver both speedups?
It discards every token beyond a fixed sliding window of the w most recent positions, so training and generation costs are both bounded by O(w), independent of the sequence length N.
It lets the term φ(K)^T V be computed as a running d×d state matrix: training accumulates this state with one O(N) pass over the sequence, and generation reuses the same fixed-size state, updated as S_t = S_{t-1} + φ(k_t)v_t^T, so each new token costs O(1) work regardless of how many tokens came before.
It keeps the full N×N similarity matrix but factorizes it into two low-rank matrices via φ, which lowers training to O(N) while generation still requires scanning the entire cached low-rank matrix, so per-token inference stays at O(N).
It keeps the O(N^2) softmax computation unchanged during training but quantizes the key and value vectors to 8-bit integers before caching them, which shrinks the constant factor in inference without changing its asymptotic complexity class.
Answer: B. It lets the term φ(K)^T V be computed as a running d×d state matrix: training accumulates this state with one O(N) pass over the sequence, and generation reuses the same fixed-size state, updated as S_t = S_{t-1} + φ(k_t)v_t^T, so each new token costs O(1) work regardless of how many tokens came before.
ExplanationBecause φ(Q)(φ(K)^T V) reorders the multiplication, φ(K)^T V need never be expanded into an N×N attention matrix — it is a single d×d matrix S. During training, S is built by summing φ(k_i)v_i^T over all N positions, an O(N) pass (causal masking is handled with a running prefix sum, still O(N) total, not O(N^2) like softmax attention). During autoregressive generation, that same state is carried forward token by token as S_t = S_{t-1} + φ(k_t)v_t^T, a fixed-size d×d update that does not grow with N, so producing the t-th token costs O(1) work instead of the O(N) work standard attention spends re-scanning its growing key/value cache. The sliding-window option describes a different technique (local attention) that genuinely discards context; the low-rank-factorization option is self-contradictory with the O(1) claim since it still leaves per-token inference at O(N); and quantization only shrinks constants, it never changes the complexity class.
A lab originally trained a GPT-3-style model with N = 175 billion parameters on D = 300 billion tokens (D/N ≈ 1.7), using compute C = 6ND FLOPs. Chinchilla scaling analysis shows the compute-optimal ratio is D/N ≈ 20. Holding the compute budget C fixed at its original value, what are the approximate compute-optimal parameter count and token count, and why does this reallocation reduce loss compared with the original allocation?
Total loss under the scaling law depends only on the total compute C = 6ND, so redistributing the same 3.15×10^23 FLOPs between parameters and tokens — for instance keeping N = 175B, D = 300B versus any other split — produces identical loss as long as C stays unchanged.
Applying the 20:1 ratio the other way around gives N_opt ≈ 1.02 trillion parameters and D_opt ≈ 51 billion tokens, which is compute-optimal because larger parameter counts extract more capacity from the fixed FLOPs budget than additional training tokens can.
Solving C = 6ND with D = 20N fixed gives N_opt ≈ 51 billion parameters and D_opt ≈ 1.02 trillion tokens for the same 3.15×10^23 FLOPs; loss falls because the original 175B-parameter model was undertrained relative to its data, so shifting compute from excess parameters into additional tokens yields a larger marginal loss reduction per FLOP.
Keeping the parameter count fixed at N = 175 billion and simply doubling the token count to roughly 600 billion is the compute-optimal rebalancing, since Chinchilla scaling only requires increasing data volume without ever reducing an already-trained model's parameter count.
Answer: C. Solving C = 6ND with D = 20N fixed gives N_opt ≈ 51 billion parameters and D_opt ≈ 1.02 trillion tokens for the same 3.15×10^23 FLOPs; loss falls because the original 175B-parameter model was undertrained relative to its data, so shifting compute from excess parameters into additional tokens yields a larger marginal loss reduction per FLOP.
ExplanationThe original allocation has N = 175×10^9 and D = 300×10^9, giving C = 6ND = 6 × 175×10^9 × 300×10^9 = 3.15×10^23 FLOPs. Chinchilla's compute-optimal ratio requires D = 20N. Substituting into the compute equation: C = 6N(20N) = 120N², so N_opt = √(C/120) = √(3.15×10^23 / 120) = √(2.625×10^21) ≈ 5.12×10^10 ≈ 51 billion parameters. Then D_opt = 20 × N_opt ≈ 1.02×10^12 ≈ 1.02 trillion tokens, using the same 3.15×10^23 FLOPs (check: 6 × 51.2×10^9 × 1.02×10^12 ≈ 3.15×10^23, consistent). The original 175B model, trained on only 300B tokens, has an actual ratio of about 1.7 — far below the 20:1 optimum — meaning it was oversized relative to the data it saw and therefore undertrained. Near that point the loss curve falls much faster along the undertrained token axis than along the already-large parameter axis, so spending the same fixed FLOPs on a smaller, more thoroughly trained model reduces loss more than spending them on excess parameters. This mirrors the qualitative result Hoffmann et al. (2022) reported: for a fixed GPT-3-scale compute budget, a substantially smaller but far more heavily trained model outperformed the original, undertrained large model.
Question 40 · Transformer Architecture: Attention Score Matrices · hard
A transformer encoder layer has d_model = 768, n_heads = 12, and processes a sequence of length seq_len = 512 tokens. Each attention head computes a score matrix S = QK^T ∈ R^{seq_len × seq_len} before the softmax, stored in 32-bit floating point. If a training framework materializes the full score matrix for every head simultaneously without any tiling or memory-efficient tricks, how many bytes of memory are required to store all 12 heads' score matrices for this one layer?
Each head's score matrix has 512 × 512 = 262,144 entries; at 4 bytes each that's 1,048,576 bytes per head, and across 12 heads the total is 12,582,912 bytes (12 MiB).
Since attention operates in the d_model-dimensional space, each score matrix is 768 × 768 entries, giving 2,359,296 bytes per head and 28,311,552 bytes total across 12 heads.
All 12 heads share a single seq_len × seq_len score matrix before splitting into head-specific projections, so only 1,048,576 bytes are needed regardless of head count.
Because each head's queries and keys have dimension d_head = 64, the score matrix is 512 × 64 entries per head, requiring 131,072 bytes per head and 1,572,864 bytes total across 12 heads.
Answer: A. Each head's score matrix has 512 × 512 = 262,144 entries; at 4 bytes each that's 1,048,576 bytes per head, and across 12 heads the total is 12,582,912 bytes (12 MiB).
ExplanationThe attention score matrix S = QK^T/√d_head is computed independently per head and has shape seq_len × seq_len — not d_model × d_model, and not seq_len × d_head. During the QK^T matrix multiplication the d_head dimension is contracted away (summed over), so it never appears in S's shape; d_head only resurfaces later, in the P·V product, where the output regains shape seq_len × d_head. Multi-head attention also gives every head its own W_Q and W_K projections, so each head produces a fully independent S rather than sharing one across heads. With seq_len = 512, each head's S has 512 × 512 = 262,144 entries. Stored in FP32 (4 bytes per entry), that's 262,144 × 4 = 1,048,576 bytes (exactly 1 MiB) per head. Across all 12 heads, the total is 12 × 1,048,576 = 12,582,912 bytes, i.e., 12 MiB. This quadratic-in-seq_len, linear-in-heads memory cost is precisely the bottleneck that memory-efficient attention algorithms like FlashAttention avoid, by never materializing the full S matrix and instead computing the softmax normalization incrementally over tiled blocks.