Grade 12 AI & Computer Science Practice Questions — Set 3

ExplanationThe exponent is 2i/d = 2×128/512 = 256/512 = 0.5, so the wavelength term is 10000^0.5 = 100 (the square root of 10000). Since i ranges from 0 to d/2, this exponent scales from 0 (wavelength 1) to 1 (wavelength 10000) as i increases, giving the sinusoidal encoding a geometric progression of wavelengths across dimensions — small wavelengths at low-index dimensions capture fine-grained differences between nearby positions, while large wavelengths at high-index dimensions capture coarse-grained differences between distant positions, letting the model represent both absolute and relative position information.

ExplanationKV-cache size is determined by the number of key/value heads, not query heads, since only K and V are cached during autoregressive decoding: cache_size = 2 (for K and V) × seq_len × batch × num_kv_heads × d_k × bytes_per_value, summed across layers. With standard multi-head attention, num_kv_heads equals n_heads = 8, giving 2×2048×16×8×64×4 = 134,217,728 bytes = 128MB per layer, and 128MB × 80 layers = 10GB total. GQA groups the 8 query heads to share just 2 KV heads, so num_kv_heads drops to 2: 2×2048×16×2×64×4 = 33,554,432 bytes = 32MB per layer, and 32MB × 80 layers = 2.5GB total. The reduction is 10GB / 2.5GB = 4x, exactly matching the KV-head reduction factor of 8/2 = 4. This 4x smaller cache directly speeds up autoregressive decoding, since the entire KV-cache must be streamed from HBM at every generation step.

ExplanationThe roofline model gives achieved throughput as min(peak_compute, bandwidth × arithmetic_intensity). Arithmetic intensity here is 26 GFLOPs ÷ 13 GB = 2 FLOP per byte moved. The GPU's balance point — the intensity at which compute time and memory time are equal — is peak_compute ÷ bandwidth = 500 TFLOP/s ÷ 2000 GB/s = 250 FLOP/byte. Because 2 FLOP/byte is far below the 250 FLOP/byte balance point, the workload sits deep in the memory-bound region of the roofline curve: achieved throughput is capped at bandwidth × intensity = 2000 GB/s × 2 = 4000 GFLOP/s, only 0.8% of the 500,000 GFLOP/s peak. Per-token latency is therefore 26 GFLOPs ÷ 4000 GFLOP/s = 6.5 milliseconds (about 154 tokens/sec) — identical to simply computing weight-load time (13 GB ÷ 2000 GB/s = 6.5 ms), since the compute itself finishes in a mere 52 microseconds and is fully hidden behind the memory transfer, not added to it. The practical fix for this bottleneck is the same as for any memory-bound decode step: shrink the bytes moved per token (more aggressive quantization, smaller KV cache) or amortize the weight load across more tokens per pass (speculative decoding, larger batch sizes).

ExplanationThe per-token contribution to β·KL(π_θ‖π_ref) is β · p · ln(p/q), where p = π_θ(token) is the probability under the policy being trained and q = π_ref(token) is the probability under the fixed reference policy — the current policy's probability must weight the log-ratio, since KL(π_θ‖π_ref) is defined as an expectation taken over π_θ, not π_ref. Here p = 0.6 and q = 0.3, so ln(p/q) = ln(2) ≈ 0.6931, giving p·ln(p/q) = 0.6 × 0.6931 ≈ 0.4159 nats. Multiplying by the KL coefficient β = 0.1 scales this raw divergence term down to its actual weight inside the training objective: 0.1 × 0.4159 ≈ 0.0416. Weighting the log-ratio by q instead of p computes the reverse-KL term instead, which has the wrong sign and magnitude here; reporting 0.4159 skips the β scaling entirely, giving the unweighted divergence rather than its contribution to the penalty; and using log base 2 conflates the natural-log (nats) convention required for KL divergence with the bits convention used in entropy coding, which is why that path lands on 0.06 instead of 0.0416.

ExplanationWith 256 levels spanning [-1, 1], the quantization step is Δ = 2/256 = 0.0078125. Modeling the rounding error as uniform over [-Δ/2, Δ/2] gives a mean squared error of Δ²/12 ≈ 5.09×10⁻⁶, so the RMS error is √(Δ²/12) ≈ 0.00226. Dividing by the typical weight magnitude of 0.5 gives a relative RMS error of about 0.45%. Using the step size itself as the error, which skips the divide-by-12 that converts a uniform distribution's range into its RMS, overstates the noise by a factor of √12 ≈ 3.46, landing near 1.6%. Using half the step size as if it were the RMS, rather than computing the RMS of a uniform distribution over that half-range, doubles the true figure to about 0.78%. Assuming a 4-bit, 16-level scheme instead of 8-bit, 256-level quantization inflates the step size by 16×, which inflates the relative error by the same factor, to roughly 7.2%.

ExplanationThe outer loop runs over Tr = N/B = 8192/256 = 32 row-blocks, and for each row-block the inner loop runs over Tc = N/B = 32 column-blocks, so the double loop performs Tr × Tc = 32 × 32 = 1024 block-pair score computations in total. The key IO-saving trick, though, is that the running accumulator O_i, along with the running softmax statistics m_i and l_i, live entirely in on-chip SRAM for the full duration of the inner loop over j = 1...Tc — every rescaling and update to O_i during those 32 inner steps costs no HBM traffic at all. Only once the inner loop finishes for a given row-block i does the algorithm normalize O_i and write it back to HBM, so there are exactly Tr = 32 HBM writes of output blocks, not 1024. Persisting statistics after every inner update (as one wrong option claims) would erase the entire IO advantage, restricting computation to diagonal blocks only (as another claims) describes local attention, not Flash Attention, and assuming the full N×N output fits in SRAM at once (as another claims) ignores why block size B is needed in the first place — SRAM cannot hold the whole matrix. This gap between 1024 compute steps and 32 HBM writes is precisely why IO complexity drops from O(N^2·d) in standard attention to O(N^2·d/B): B determines how many block-pair computations are amortized against a single HBM write.

ExplanationEach LoRA matrix pair contributes 4096×16 + 16×4096 = 65,536 + 65,536 = 131,072 trainable parameters per projection. Across the four attention projections (Q, K, V, O) that totals 4 × 131,072 = 524,288 trainable parameters, versus 4 × (4096×4096) = 67,108,864 parameters in the original full-rank matrices — a ratio of 524,288 / 67,108,864 ≈ 0.78%. At inference, B and A are multiplied and added directly into W: the product BA has shape (4096×16)·(16×4096) = 4096×4096, exactly matching W, so W_merged = W + BA can replace W in place with no shape change and no added inference latency. The claim that only 16 parameters are trained per matrix confuses the rank r — the shared inner dimension of B and A — with the actual parameter count, which depends on both the rank and the full 4096 dimension, not the rank alone. The claim that the merged matrix has shape 4096×16 misreads the matrix product BA, which takes on the shape of W (4096×4096), not the shape of a single factor like B. The claim that LoRA trains roughly 7.8% of the original parameters is off by a factor of ten from the correct 0.78% figure.

ExplanationApplying softmax to z = [1.2, 0.3, 2.1, -0.5] gives probabilities of approximately 0.247, 0.100, 0.607, and 0.045 for Experts 1 through 4 (the exponentials are about 3.32, 1.35, 8.17, and 0.61, summing to roughly 13.44). The two largest probabilities belong to Expert 3 (0.607) and Expert 1 (0.247), so these are the experts the token gets routed to. Because the combination weights of a top-k MoE layer must sum to 1, the selected probabilities are renormalized by dividing each by their sum (0.607 + 0.247 = 0.854): Expert 3 gets 0.607/0.854 ≈ 0.71 and Expert 1 gets 0.247/0.854 ≈ 0.29. Skipping this renormalization step and using the raw softmax outputs directly would leave weights of about 0.61 and 0.25, which do not sum to 1 and therefore cannot correctly scale the combined output — that is the flaw in using pre-renormalization values as the final weights. Normalizing the raw logits (2.1 and 1.2) instead of their softmax values confuses linear scaling with the exponential weighting softmax actually performs, producing the incorrect pair of about 0.64 and 0.36. Treating all four experts as equally weighted ignores the entire point of top-k routing, which is to activate only a sparse subset of experts per token — with top-k=2 out of 4 experts, only half the expert parameters are active for this token, not all of them at equal weight.

ExplanationNucleus sampling keeps the smallest prefix of sorted tokens whose cumulative probability is at least p, then samples from that truncated, renormalized distribution — it never picks the argmax deterministically. Walking the cumulative sums [0.42, 0.70, 0.85, 0.93, ...], the running total first meets p = 0.85 at the third token, so the nucleus is {0.42, 0.28, 0.15}, three tokens whose probabilities sum to 0.85. Dividing each by that sum renormalizes them to roughly [0.494, 0.329, 0.176], so the top token's sampling probability rises from 0.42 to about 0.494 rather than staying fixed. The claim that reaching p exactly should exclude the crossing token misapplies the threshold rule: the standard "cumulative >= p" condition includes the token that first satisfies it, so trimming to a 2-token, 0.70 nucleus is wrong. The claim of deterministic single-token output confuses nucleus sampling with greedy decoding; nucleus sampling still draws stochastically from several candidates whenever more than one survives truncation. And the retained probabilities 0.42 + 0.28 + 0.15 sum to 0.85, not 1.0 — the full vocabulary's probabilities sum to 1.0 before truncation, but once the tail is discarded, renormalization is required so the kept tokens form a valid probability distribution again.

ExplanationFlash Attention's online-softmax recurrence keeps a running max m, running denominator l, and running unnormalized numerator, updating them one tile at a time so the full score row is never stored. After tile 1 ([2,4], values [10,20]): m=4, l = e^(2-4)+e^(4-4) = 0.1353+1 = 1.1353, numerator = 10(0.1353)+20(1) = 21.3534. Tile 2 ([1,5], values [1,2]) has local max 5, so the running max updates to 5. Before adding tile 2, the tile-1 accumulators must be rescaled by exp(m_old − m_new) = exp(4−5) = exp(−1) ≈ 0.3679, giving a rescaled numerator ≈ 7.8555 and rescaled denominator ≈ 0.4177. Tile 2's own contribution relative to m=5 is numerator = 1·e^(1-5) + 2·e^(5-5) = 0.0183+2 = 2.0183, denominator = 0.0183+1 = 1.0183. Totals: numerator ≈ 9.8738, denominator ≈ 1.4360, so output = 9.8738/1.4360 ≈ 6.876, rounding to 6.88 — the same value a direct one-shot softmax over all four scores [2,4,1,5] produces. Stopping at the raw weighted-value sum of 9.87 skips the mandatory division by the running denominator, so that path reports an unnormalized numerator rather than an attention output. Flipping the rescale exponent to exp(new_max − old_max) = exp(1) instead of exp(old_max − new_max) inflates the stale tile-1 term rather than discounting it, landing near 14.63 — a genuine bug pattern seen in naive implementations. Finally, the exact equivalence to global softmax does not depend on tile sizes dividing the sequence length evenly: the same max-rescaling algebra holds for a ragged final tile of any size, so tying correctness to even divisibility misstates the algorithm's actual guarantee.

ExplanationTop-k=2 routing means exactly two of the eight experts run a full forward pass for each token, not all eight and not just one — so active parameters equal 2 x 200 million = 400 million, while the other six experts (and their 1.2 billion parameters) are skipped entirely for this token. This is the core efficiency property of MoE: total model capacity (8 x 200M = 1.6 billion parameters) is far larger than the compute actually spent per token. Separately, after the router picks the top-2 experts by raw softmax score, those two scores are renormalized (divided by their own sum) so the token's output is a proper convex combination of only the chosen experts' outputs — the two renormalized weights must add to exactly 1. Since expert 5's renormalized weight is given as 0.65, the other selected expert must take the remainder, 1 - 0.65 = 0.35; it cannot also be 0.65, since that would sum to 1.30 and violate the renormalization constraint.

ExplanationWith P=8 stages and M=32 micro-batches, the bubble fraction is (P−1)/M = 7/32 = 0.21875 = 21.875%. GPipe's all-forward-then-all-backward schedule forces a GPU to hold activations for every micro-batch that has completed its forward pass but not yet its backward pass — in the worst case all 32 of them — giving peak memory of order O(M). 1F1B interleaves forward and backward passes so that each pipeline stage never has more than P=8 micro-batches' activations in flight simultaneously, giving O(P) peak memory. The reduction factor is therefore M/P = 32/8 = 4. One incorrect approach drops the (P−1) term and computes P/M = 25%, then wrongly claims memory falls to exactly P−1=7 sets by reasoning that the last stage "never buffers" anything — but every stage, including the last, still holds up to P in-flight activation sets under 1F1B, and dropping the −1 term also produces the wrong bubble value. Another approach inverts the ratio to (M−1)/P = 31/8 = 387.5%, an impossible idle-time value since a fraction of time cannot exceed 100%, paired with the false claim that 1F1B needs zero activation memory — in reality 1F1B still buffers each stage's own P in-flight micro-batches, it just avoids GPipe's much larger backlog. A third approach lands on the correct bubble (21.875%) and the correct factor of 4, but for the wrong reason: 1F1B does not skip 24 of the 32 micro-batches — it still processes all 32, just staggered so that no more than P are in flight at once, which is what actually bounds memory at O(P) rather than O(M).

ExplanationWorking through the loop by hand: the cumulative sums are 0.32, 0.60, 0.78, 0.90, 0.96, 1.00, and the break condition c >= 0.75 first triggers at the third entry (0.78), so the nucleus consists of exactly the three highest-probability tokens {0.32, 0.28, 0.18}, whose combined mass is 0.78 — not the six-token total of 1.00. Renormalizing means every probability that survived truncation is divided by that nucleus total of 0.78 so the retained distribution sums to 1 again; nucleus membership does not exempt a token from this step, so 0.18 must still be rescaled rather than left unchanged. Carrying out the division: 0.18 / 0.78 ≈ 0.231. Using 0.90 as the denominator instead — the cumulative sum one step past the actual cutoff — would incorrectly give 0.200, mistaking where the loop's break condition fires. Normalizing 0.18 against only the two higher-ranked tokens' combined 0.60, as if a token's renormalization reference set excluded itself, would incorrectly give 0.300. The correctly computed renormalized probability is approximately 0.231.

ExplanationFlash Attention never materializes the full N×N score matrix; instead it processes key/value blocks one at a time and keeps a running (unnormalized) output accumulator O, running sum l, and running max m in fast SRAM. Because every accumulated exponential term is only meaningful relative to the max value it was computed against, whenever a later block introduces a larger row-max, all previously accumulated terms must be re-expressed relative to that new max before anything more is added — otherwise the running sum and output would mix exponentials taken relative to two different reference points and the result would no longer equal a valid softmax. Concretely, since exp(score − m1) = exp(score − m2) · exp(m2 − m1), multiplying the old accumulators by exp(m1 − m2) converts every term from being relative to m1 into being relative to m2. Here m1 = 4 and m2 = 7, so the correction factor is exp(4 − 7) = exp(−3) ≈ 0.0498: both O_1 and l_1 are scaled down by this factor before block 2's exp(score − m2) contributions are added in. This online rescaling — the "running softmax" trick — is exactly what lets Flash Attention produce results mathematically identical to standard attention while touching HBM only O(N) times instead of O(N^2), since it never needs to revisit block 1's raw scores once they have been folded into the accumulators.

ExplanationFor token x, the raw gating logits are z = [2.0, 0.5, -1.0, 3.0, 0.0, 1.0], so the full 6-way softmax gives expert 3 (logit 3.0) the highest probability and expert 0 (logit 2.0) the second-highest, since softmax is monotonic in its inputs. Top-k=2 routing keeps only these two experts and discards the other four; because the original softmax probabilities were normalized against a denominator that included those discarded experts, the two kept weights no longer sum to 1 and must be renormalized against each other. Renormalizing means dividing each selected expert's exponentiated logit by the sum of just the exponentiated logits of the two selected experts: exp(3.0) ≈ 20.086 and exp(2.0) ≈ 7.389, which sum to ≈ 27.475. This gives w3 = 20.086 / 27.475 ≈ 0.73 and w0 = 7.389 / 27.475 ≈ 0.27, values that correctly sum to 1 and preserve the ranking that the expert with the larger logit (expert 3) receives the larger share of the combination. Skipping renormalization leaves the weights at their raw full-softmax values (≈ 0.60 and ≈ 0.22), which understate each expert's true contribution and fail to sum to 1. Treating top-k selection as an equal split discards the router's learned confidence entirely, collapsing MoE's weighted combination into plain averaging. Swapping which expert gets the larger share would invert the router's signal, handing more influence to the expert it was actually less confident about.

ExplanationCumulative sums of the sorted distribution are 0.40, 0.60, 0.75, 0.87, 0.95, 1.00. Because the third partial sum already equals 0.75, the nucleus rule (retain the smallest prefix whose cumulative probability reaches or exceeds p) stops right there and keeps exactly the top 3 tokens, whose combined probability is 0.75. Renormalizing divides each retained probability by this 0.75: 0.40/0.75 ≈ 0.533, 0.20/0.75 ≈ 0.267, and 0.15/0.75 = 0.20 for the third-ranked token. Requiring the cumulative sum to strictly exceed p instead of merely reaching it is a common implementation slip that would incorrectly pull in a fourth token, changing the renormalized value to about 0.172 instead. Picking whichever prefix's cumulative sum lies numerically closest to p, or truncating the distribution without renormalizing the survivors afterward, are both misreadings of the algorithm — nucleus sampling filters by a fixed threshold rule, not by proximity, and always renormalizes the surviving probability mass so it remains a valid distribution.

ExplanationWith N = 4096 and B = 256, Q and K each split into N/B = 16 blocks, so the nested loop visits 16 × 16 = 256 block pairs in total — every pair of blocks is still compared, so the dot-product and weighted-sum work performed is the same O(N²d) as standard attention; Flash Attention does not skip any query-key interaction. What changes is memory: instead of allocating the full 4096×4096 score matrix, the kernel keeps only the current B×B tile in SRAM together with a running maximum and running sum of exponentials per query row, updating the output accumulator incrementally as each new block arrives. This running-statistics trick (the "online softmax") is mathematically exact — rescaling previously accumulated sums whenever a new, larger max is found reproduces precisely the same softmax weights as computing over the whole row at once, so no approximation error is introduced regardless of how many blocks are processed. The claim of O(N²d/B) floating-point operations confuses the IO/data-movement formula — which genuinely does shrink with block size because fewer bytes move between HBM and SRAM — with the compute formula, which does not shrink. And because every one of the 256 block pairs is visited exactly once, no query ever loses visibility into keys sitting in other blocks; the full sequence is still attended to.

ExplanationWorking through the routing computation step by step: the softmax gate vector is g(x) = [0.05, 0.10, 0.35, 0.05, 0.30, 0.15], indexed 0 through 5. Top-2 routing keeps only the two largest gate values, which belong to expert 2 (0.35) and expert 4 (0.30); every other expert is dropped entirely for this token. The router then renormalizes just these two surviving weights so they sum to 1: it divides each by their combined total, 0.35 + 0.30 = 0.65. Expert 4's renormalized weight is therefore 0.30 / 0.65 ≈ 0.462, and expert 2's is 0.35 / 0.65 ≈ 0.538 — together they still sum to 1, confirming the renormalization is consistent. The claim that expert 4 keeps its raw softmax value of 0.300 unchanged ignores that top-k gating always renormalizes over the surviving subset, since the original 6-way softmax was calibrated to sum to 1 across all six experts, not just the two chosen ones. The claim of 0.538 mistakenly assigns expert 4 the weight that actually belongs to expert 2, confusing which of the two selected experts gets the larger share. The claim of 0.375 comes from mistakenly including a third expert (expert 5, gate value 0.15) in the renormalization pool, which violates the top-2 constraint specified by the algorithm — only the two highest-scoring experts should ever be renormalized and combined.

ExplanationWith P = 4 pipeline stages and the batch split into M = 8 micro-batches, the bubble fraction is (P−1)/M = 3/8 = 37.5%: three of the four stages sit idle while the pipeline fills (waiting for the first micro-batch to arrive) and drains (waiting for the last micro-batch to finish), and that fixed idle cost is amortized over all M micro-batches processed. On the memory side, GPipe runs every micro-batch's forward pass before starting any backward pass, so it must hold activations for all M micro-batches simultaneously — O(M) memory. 1F1B instead starts the backward pass for an early micro-batch as soon as its gradient is needed rather than waiting for every micro-batch to finish forward propagation, which caps the number of micro-batches with live, un-backpropagated activations at roughly P, giving O(P) memory — a real reduction versus GPipe. A claim that substitutes P/M for (P−1)/M overstates the idle fraction as 50% instead of 37.5%, since it double-counts stage 4's own processing time as idle time. A claim that 1F1B raises memory to O(M) has the comparison backwards — O(M) buffering is what GPipe does, not what 1F1B avoids. And a claim that activations can shrink to O(1) right after the forward pass ignores that the backward pass still needs those stored activations to compute gradients; eliminating them entirely requires a separate technique, activation checkpointing with recomputation, not 1F1B scheduling by itself.

ExplanationNucleus (top-p) sampling keeps the smallest prefix of the sorted probability list whose cumulative sum is at least p, then renormalizes only the kept probabilities so they sum to 1. Here the prefix sums are 0.40, 0.65, 0.80, 0.90, 0.96, and 1.00. The first prefix sum that reaches or exceeds p = 0.75 is 0.80, reached once the third token is included, so the nucleus is the set {0.40, 0.25, 0.15} — not just the first two tokens, and not the first four. Dividing each of these three probabilities by the nucleus's total mass (0.80) renormalizes them to 0.5, 0.3125, and 0.1875, so the third-ranked token's renormalized probability is 0.15 / 0.80 = 0.1875. Stopping at the top two tokens is invalid because their cumulative sum of 0.65 is still below p = 0.75 — nucleus sampling requires the cumulative mass to reach or exceed p, not merely come close to it, so a partial prefix that hasn't accumulated enough mass is never a valid cutoff. Treating p as a fraction of the vocabulary's token count confuses top-p with top-k: top-p thresholds cumulative probability mass, while top-k thresholds a fixed number of tokens, and the number of tokens top-p retains varies with how peaked or flat the distribution is rather than tracking a fixed percentage of the vocabulary. Nucleus sampling is also not deterministic like greedy decoding — it samples from the renormalized distribution over all retained tokens, so the third-ranked token stays eligible for selection with probability 0.1875 rather than being excluded.