In a CNN architecture, if you apply a 3x3 convolutional filter with stride=2 and padding=1 to a 28x28 input image, what are the spatial dimensions of the output feature map, and how would you mathematically derive this using the formula (W - F + 2P) / S + 1?
The resulting feature map is 7x7 because stride=2 is applied twice internally in the convolution operation.
The output would be 28x28 since padding preserves input size regardless of stride, incorrectly assuming stride has no effect on the output dimensions.
The output dimensions would be 14x14 using the formula (W - F + 2P) / S + 1 = (28 - 3 + 2*1) / 2 + 1 = 27/2 + 1 = 14.5, which floors to 14.
Output dimensions are 56x56 because the formula multiplies stride by input size.
Answer: C. The output dimensions would be 14x14 using the formula (W - F + 2P) / S + 1 = (28 - 3 + 2*1) / 2 + 1 = 27/2 + 1 = 14.5, which floors to 14.
ExplanationThe output dimension formula (W - F + 2P) / S + 1 relates the input width W, filter size F, padding P, and stride S to the resulting output size. Substituting the given values: (28 - 3 + 2*1) / 2 + 1 = (25 + 2) / 2 + 1 = 27 / 2 + 1 = 13.5 + 1 = 14.5. Since spatial output dimensions must be whole numbers, this value is floored to 14. So the 28x28 input, convolved with a 3x3 filter using stride 2 and padding 1, produces a 14x14 output feature map.
Question 2 · Pooling and Stride · hard
A convolutional layer outputs a feature map of shape 32×32×64 (height × width × channels). This is fed into a max pooling layer with pool_size=2×2 and stride=2, using the standard pooling formula output = floor((W − F)/S) + 1 on each spatial dimension. What is the resulting output shape, and why does adding this pooling layer typically reduce overfitting compared to a network that skips it?
Pooling produces a 16×16×128 feature map, since halving the spatial dimensions is paired with doubling the channel depth to compensate for lost resolution, and this depth increase is what curbs overfitting by adding parameter redundancy.
A 15×15×64 output results, because the 2×2 window is centered on each pixel and therefore cannot be placed over the first or last row and column, and overfitting falls because these discarded boundary pixels were most prone to noise.
This layer yields a 16×16×64 map, but the drop in overfitting mainly comes from the pooling layer's own trainable weights, which let it learn to average out validation-set noise more precisely than a fixed convolution stride would.
The correct output is 16×16×64, since pooling halves each spatial dimension independently while leaving channel depth untouched, and overfitting eases because the layer contributes zero trainable parameters while its max operation grants small-translation invariance in the input.
Answer: D. The correct output is 16×16×64, since pooling halves each spatial dimension independently while leaving channel depth untouched, and overfitting eases because the layer contributes zero trainable parameters while its max operation grants small-translation invariance in the input.
ExplanationApplying the pooling formula (W − F)/S + 1 to each spatial dimension gives (32 − 2)/2 + 1 = 16, so the output is 16×16×64 — channel depth is unchanged because each of the 64 channels is pooled independently and pooling never alters depth. Max pooling introduces no trainable parameters at all: it is a fixed, deterministic operation that simply keeps the largest activation in each non-overlapping 2×2 window. This shrinks the volume of information passed to the next layer and grants approximate translation invariance, since small shifts in the input often still land the strongest activation in the same window. Both effects work against memorizing training-specific noise, which is why pooling is a standard tool for controlling overfitting in CNN architectures.
Question 3 · Image Classification Architectures (ResNet/VGG/EfficientNet) · hard
Consider a 3-block residual network where each block computes y_i = F_i(x_i) + x_i (a skip connection around branch F_i), and every branch has an identical local Jacobian ∂F_i/∂x_i = 0.2. Given that the gradient of the loss with respect to the final block's output is ∂L/∂y_3 = 1, what is ∂L/∂x_1 after backpropagating through all three blocks, and how does this compare to a plain (non-residual) stack of the same three branches?
Backpropagation collapses the gradient to about 0.008 in the residual network too, because the identity paths do not carry gradient signal once dimension-matching convolutions are involved, leaving only the branch Jacobians to multiply: 0.2 cubed.
Only one overall '+1' term applies across the whole stack rather than one per block, giving a gradient of about 1.6, computed as 1 plus the sum of the three branch Jacobians: 1 + (3 × 0.2).
Each residual block multiplies the incoming gradient by (1 + 0.2) = 1.2 from its identity path, so three blocks compound multiplicatively to about 1.728, while the plain stack without skip connections would shrink the same gradient to 0.2 cubed = 0.008, roughly 216 times smaller.
Skip connections leave backpropagation completely unchanged, producing the same gradient of about 0.2 in both the residual and plain networks, since skip connections only affect the forward pass by adding activations.
Answer: C. Each residual block multiplies the incoming gradient by (1 + 0.2) = 1.2 from its identity path, so three blocks compound multiplicatively to about 1.728, while the plain stack without skip connections would shrink the same gradient to 0.2 cubed = 0.008, roughly 216 times smaller.
ExplanationFor a residual block y_i = F_i(x_i) + x_i, the chain rule gives ∂L/∂x_i = ∂L/∂y_i · (1 + ∂F_i/∂x_i). Since every branch has ∂F_i/∂x_i = 0.2, each block multiplies the incoming gradient by (1 + 0.2) = 1.2. Starting from ∂L/∂y_3 = 1 and backpropagating through three blocks gives ∂L/∂x_1 = 1 × 1.2 × 1.2 × 1.2 = 1.2³ ≈ 1.728. The "+1" inside each block's factor guarantees the gradient never falls below what arrived from the layer above — even a weak branch Jacobian cannot push the multiplier below 1. A plain (non-residual) stack of the same three branches has no such floor: each layer multiplies the gradient by ∂F_i/∂x_i = 0.2 alone, so after three layers the gradient shrinks to 0.2³ = 0.008 — about 216 times smaller than the residual network's 1.728 (since 1.728 / 0.008 = 216). This per-block "+1" floor is exactly the mechanism that lets ResNet train networks tens of layers deep: the identity shortcut guarantees a lower bound of 1 on each block's gradient multiplier, whereas a plain stack compounds sub-1 Jacobians into a vanishingly small gradient.
Question 4 · Regularization (Dropout/BatchNorm/Weight Decay) · hard
A neuron has a pre-dropout activation value of z = 100 and, during training, is zeroed out with dropout probability p = 0.25 (kept unchanged the rest of the time). Using the classical ("vanilla") dropout rule — where activations are left unscaled during training but rescaled once at inference — what activation value should this neuron pass forward at test time so that it matches the expected activation seen during training?
Scaling the raw activation by the keep probability gives 100 × (1 − 0.25) = 75, matching the training-time expected value E[activation] = 0.75×100 + 0.25×0 = 75 — because multiplying by (1−p) reproduces the same expectation deterministically instead of relying on the random dropout mask.
Multiplying the activation by the dropout probability itself gives 100 × 0.25 = 25, since the network is thought to need its output shrunk by the fraction of neurons that were sometimes dropped during training.
Dividing the raw activation by the keep probability gives 100 / (1 − 0.25) ≈ 133.3, on the reasoning that inference must amplify every surviving activation to compensate for reduced network capacity during training.
Leaving the activation unchanged at 100 is correct here, because dropout masks apply only during training and should be entirely absent — including any compensating scale factor — at inference time.
Answer: A. Scaling the raw activation by the keep probability gives 100 × (1 − 0.25) = 75, matching the training-time expected value E[activation] = 0.75×100 + 0.25×0 = 75 — because multiplying by (1−p) reproduces the same expectation deterministically instead of relying on the random dropout mask.
ExplanationDuring training with dropout probability p = 0.25, this neuron's activation z = 100 is kept unchanged with probability 0.75 and zeroed with probability 0.25, so its expected contribution to the next layer is E[activation] = 0.75×100 + 0.25×0 = 75. Downstream weights learn throughout training to expect inputs of roughly this reduced magnitude. At test time no neurons are randomly dropped, so passing the full activation of 100 forward would systematically inflate every downstream sum relative to what the network was trained on. Multiplying the test-time activation by the keep probability, 100 × (1 − 0.25) = 75, restores exactly the expected value the downstream weights were calibrated against, with no randomness needed at inference. This is the classical ("vanilla") dropout scaling rule; the mathematically equivalent alternative, inverted dropout, instead divides by the keep probability during training so that nothing needs to be rescaled at test time — but that is a different implementation choice, not simply omitting the scale factor.
Question 5 · Convolutional Neural Networks · hard
In a semantic segmentation network, a 3×3 convolutional filter is applied with dilation_rate = 3 (an atrous/dilated convolution), and the effective receptive field is computed using RF = k + (k − 1)(d − 1), where k is the kernel size and d is the dilation rate. What is the effective receptive field of this filter, and why does using dilation instead of stride-based downsampling let the network expand this context while keeping the parameter count and output resolution unchanged?
Applying the formula RF = k + (k − 1)(d − 1) gives 3 + (3 − 1)(3 − 1) = 3 + 4 = 7×7, and because dilation only spaces out the existing kernel weights with zero-gaps rather than adding new ones, this larger context window is gained without increasing the parameter count or altering the output resolution.
The effective receptive field works out to 9×9, since RF = k + (k − 1)×d = 3 + 2×3, treating each of the two gaps created by dilation as spanning a full d = 3 pixels rather than d − 1 pixels.
Setting dilation_rate=3 downsamples the feature map by a factor of 3 the same way a stride-3 convolution would, shrinking a 224×224 input to roughly 75×75 while leaving the receptive field at its original 3×3 size.
The receptive field simply becomes 3 + 3 = 6×6, because each unit increase in the dilation rate adds exactly one row and one column to the kernel's footprint regardless of the kernel's own size.
Answer: A. Applying the formula RF = k + (k − 1)(d − 1) gives 3 + (3 − 1)(3 − 1) = 3 + 4 = 7×7, and because dilation only spaces out the existing kernel weights with zero-gaps rather than adding new ones, this larger context window is gained without increasing the parameter count or altering the output resolution.
ExplanationThe formula for the effective receptive field of a dilated (atrous) convolution is RF = k + (k − 1)(d − 1), where k is the kernel size and d is the dilation rate. Substituting k = 3 and d = 3 gives RF = 3 + (3 − 1)(3 − 1) = 3 + 2×2 = 3 + 4 = 7, so the filter now covers a 7×7 region of the input instead of the original 3×3. This works because dilation inserts d − 1 = 2 zero-valued gaps between each pair of adjacent kernel weights, spreading the same 9 learnable parameters over a wider area instead of adding new ones — the parameter count stays fixed at 3×3 = 9 weights no matter how large d is. Because dilation widens the filter's view by inserting gaps rather than by using a stride or pooling operation to subsample, it does not shrink the feature map: an input of size H×W still produces an H×W output (with matching padding), preserving full spatial resolution. This combination — a receptive field that grows quickly as d increases, a constant parameter count, and no loss of resolution — is exactly what architectures such as DeepLab rely on to gather multi-scale context for segmentation while still producing precise, pixel-accurate boundaries.
Question 6 · Image Classification Architectures (GoogLeNet/Inception) · hard
Inception modules in GoogLeNet use parallel convolutional branches: 1×1 conv (1M params) + 3×3 conv (9M) + 5×5 conv (25M) + max pool (0 params) concatenated. Calculate total Inception module parameters and explain why parallel branches improve feature richness without exponential parameter growth?
Total params: 1M + 9M + 25M = 35M per Inception module.
Total params: 35M is correct, but parallel branches hurt performance by splitting network capacity across multiple paths, causing underfitting.
Inception modules have zero parameters since 1×1, 3×3, 5×5 convolutions are merely feature extractions that don't require trainable weights.
Parallel branches increase parameters exponentially: 1M × 9M × 25M = 225 quintillion params, making Inception modules intractable for large networks.
Answer: A. Total params: 1M + 9M + 25M = 35M per Inception module.
ExplanationThe four branches act on the same input in parallel, so their parameter counts are added, not multiplied: 1M (1×1 branch) + 9M (3×3 branch) + 25M (5×5 branch) + 0 (pooling branch has no learnable weights) = 35M total parameters for the module. That additive relationship is also the answer to why parallel branches boost feature richness without exponential growth: each branch independently extracts features at its own receptive-field scale — the 1×1 branch captures fine-grained per-pixel patterns, the 3×3 and 5×5 branches capture progressively wider spatial context, and the pooling branch preserves the strongest activations — and their outputs are then concatenated along the channel dimension rather than combined multiplicatively. Concatenation stacks channels instead of creating combinatorial interactions between branches, so adding more parallel paths scales the parameter count linearly (35M is a simple sum, not a product) even though the module now represents multiple spatial scales at once. That is precisely why the multiplicative framing (1M × 9M × 25M) is wrong: it misrepresents concatenation as a combinatorial interaction and produces a number many orders of magnitude too large to describe a real Inception module.
Question 7 · GAN Training Dynamics · hard
In the original GAN formulation, the generator minimizes L_G = log(1 − D(G(z))), where D(G(z)) = σ(a) and a is the discriminator's pre-sigmoid logit on the generated sample. Differentiating gives dL_G/da = −D(G(z)), so this gradient shrinks toward 0 whenever the discriminator confidently rejects the generator's output (D(G(z)) → 0) — precisely the situation early in training. In practice this saturating objective is replaced with L_G' = −log D(G(z)), whose gradient is dL_G'/da = −(1 − D(G(z))). Given that both objectives are minimized at D(G(z)) = 1, why does this substitution improve generator training in the early, saturating regime?
Switching to -log D(G(z)) moves the generator's optimum to D(G(z)) = 0.5 rather than D(G(z)) = 1, so the generator only needs to make the discriminator uncertain instead of fully deceiving it, which is an easier target for gradient descent to reach.
The original loss's gradient with respect to the discriminator's logit equals −D(G(z)), which is close to 0 early in training when the discriminator confidently rejects poor samples, starving the generator of a learning signal exactly when it needs one most; the alternative loss's gradient equals −(1 − D(G(z))), which is close to −1 in that same regime, and both objectives reach their minimum at the same point, D(G(z)) = 1.
The expression log(1 − D(G(z))) is numerically undefined whenever D(G(z)) approaches 0, producing NaN values during backpropagation, so the reformulation exists to avoid this numerical failure rather than to change how large the gradient is.
As a function of the discriminator's logit, −log D(G(z)) is convex, which guarantees that gradient descent on the generator converges to the global minimum of the full minimax game regardless of how the discriminator is trained.
Answer: B. The original loss's gradient with respect to the discriminator's logit equals −D(G(z)), which is close to 0 early in training when the discriminator confidently rejects poor samples, starving the generator of a learning signal exactly when it needs one most; the alternative loss's gradient equals −(1 − D(G(z))), which is close to −1 in that same regime, and both objectives reach their minimum at the same point, D(G(z)) = 1.
ExplanationIn the original minimax GAN objective, the generator minimizes L_G = log(1 − D(G(z))), where D(G(z)) = σ(a) is the discriminator's sigmoid output on the generator's logit a. Differentiating log(1 − σ(a)) with respect to a gives dL_G/da = −σ(a) = −D(G(z)). Early in training, when the generator's samples are weak, the discriminator confidently assigns D(G(z)) ≈ 0, so this gradient is also ≈ 0 — the generator receives almost no corrective signal exactly when its output is worst. Replacing the objective with L_G' = −log D(G(z)) gives dL_G'/da = −(1 − σ(a)) = −(1 − D(G(z))), which is ≈ −1 in that same regime — a strong, useful gradient. Both objectives are minimized at the same point, D(G(z)) = 1 (the generator perfectly fools the discriminator), so the substitution changes only the gradient's magnitude away from that point, not the training target itself. The claim that switching objectives shifts the optimum to D(G(z)) = 0.5 is incorrect: −log D(G(z)) keeps decreasing as D(G(z)) approaches 1, so 1 remains the minimum, not 0.5. The numerical-instability claim has the failure direction backwards: log(1 − D(G(z))) is perfectly well-defined when D(G(z)) ≈ 0 (it simply equals log 1 = 0) and only diverges as D(G(z)) → 1; it is −log D(G(z)) whose value grows large as D(G(z)) → 0, even though its gradient stays bounded there. Convexity of −log D(G(z)) in the logit a says nothing about the overall minimax game, because a itself depends non-linearly on the generator's own parameters through the network that produces it — the full objective as a function of those parameters remains non-convex, so gradient descent is not guaranteed to reach any global optimum.
Question 8 · CNN Architecture · hard
In a Depthwise Separable Convolution, you first apply depthwise Conv(3x3) with 32 input channels (32 filters, one per channel), then pointwise Conv(1x1, 128). Compare FLOPs to a standard Conv(3x3, 32→128)?
Separable uses 3x fewer FLOPs because splitting into two operations instead of one yields a 9x reduction from the 3x3 depthwise step offset by a 3x increase from the 1x1 pointwise step, for a net 3x savings.
FLOPs are equal because both paths ultimately process the same 32 input channels and 128 output channels, so the total multiply-accumulate count comes out the same regardless of how the convolution is factored.
Standard convolution is 4x more efficient because processing all 32 channels together in a single 3x3 operation avoids the overhead of a separate pointwise step, cutting total FLOPs by a factor of four relative to the separable approach.
ExplanationFirst, depthwise Separable FLOPs: Depthwise Conv(3x3): 32 filters * 3*3 kernel * H*W locations = 288*H*W FLOPs. Pointwise Conv(1x1): 32*128 * H*W = 4096*H*W FLOPs. Then, total ≈ 4384*H*W. Standard Conv(3x3, 32→128): 32*128*3*3 * H*W = 36864*H*W FLOPs. Ratio: 36864/4384 ≈ 8.4x fewer FLOPs with separable convolution. Common mistake: thinking FLOPs are equal, or assuming splitting into two operations must cost more than one combined operation — depthwise convolution scales linearly with the channel count while a standard 3x3 convolution scales with the product of input and output channels, which is exactly why separable convolutions are so much cheaper here.
Question 9 · CNN Architecture · hard
You implement a Group Convolution with groups=4 on 32 input channels and 64 output channels, kernel=3x3. How many distinct filter groups are created, and what is the shape of each group's weight tensor?
4 groups, each with weight shape [16, 8, 3, 3].
4 groups, but each weight tensor is [64, 32, 3, 3], as if groups were not applied at all.
8 groups, each with weight shape [8, 4, 3, 3], from miscomputing channels-per-group as total/groups².
4 groups, with input and output swapped: weight shape [8, 16, 3, 3] instead of [16, 8, 3, 3].
Answer: A. 4 groups, each with weight shape [16, 8, 3, 3].
ExplanationGroup convolution partitions both input and output channels. With groups=4, input_channels=32, output_channels=64: each group processes 32/4=8 input channels and produces 64/4=16 output channels. The weight shape per group is therefore [16 (output_per_group), 8 (input_per_group), 3, 3]; stacked across all groups this is [64, 8, 3, 3], or conceptually [4, 16, 8, 3, 3]. The distractors arise from forgetting to divide by groups at all (giving [64, 32, 3, 3]), swapping which dimension is input versus output ([8, 16, 3, 3]), or miscomputing the number of groups and channels-per-group entirely (8 groups of [8, 4, 3, 3]).
Question 10 · Object Detection · hard
In YOLOv3, an input image is resized to 416×416 and divided into a 13×13 grid, so each cell spans 32×32 pixels. For a bounding box's center, the network predicts raw values t_x and t_y, which are passed through a sigmoid function and added to the grid cell's own row and column index to produce the final center location. Why does YOLOv3 use this sigmoid-bounded, per-cell offset instead of having the network output the box center as absolute pixel coordinates directly?
Bounding the offset to the [0, 1] range with a sigmoid guarantees the predicted center lands inside the cell responsible for that object, giving every object a single, well-defined detecting cell while keeping the regression target a small, consistently scaled value instead of an absolute pixel count that grows with image resolution.
Sigmoid-bounded offsets let every grid cell that overlaps a given object register it as a detection, so the same object can be reported by several neighbouring cells before non-max suppression trims the extras.
Restricting the center to an offset within the cell removes the need for anchor box priors on width and height, since YOLOv3 derives box scale directly from how far the offset value sits from 0.5.
Working with a cell-relative offset lets the objectness term alone decide which cells contribute to the localization loss, so the coordinate parameterization itself is what keeps background cells from producing any gradient.
Answer: A. Bounding the offset to the [0, 1] range with a sigmoid guarantees the predicted center lands inside the cell responsible for that object, giving every object a single, well-defined detecting cell while keeping the regression target a small, consistently scaled value instead of an absolute pixel count that grows with image resolution.
ExplanationYOLOv3 computes the final box center as bx = cx + σ(tx) and by = cy + σ(ty), where (cx, cy) is the grid cell's own row/column index and σ is the sigmoid. Because σ(tx), σ(ty) ∈ [0, 1], the predicted center can never leave the cell that produced it, so exactly one cell — the one whose region contains the object's true center — is trained to be responsible for that object; no other cell's loss includes localization error for it. This also keeps the regression target itself small and dimension-independent: the network always learns to output a fraction of one 32×32 cell rather than a raw pixel count that would range up to 416 and would change if the input resolution changed, which is a far easier quantity to fit with stable gradients from a random initialization. Suggesting that several neighbouring cells could all claim the same object confuses YOLOv3's single-cell assignment rule with two-stage detectors that later merge overlapping proposals. Claiming the offset removes the need for anchor priors on width and height is also incorrect: YOLOv3 still predicts bw = pw·e^(tw) and bh = ph·e^(th) against predefined anchor dimensions pw, ph — the sigmoid offset governs only the center location, not the box scale. And while it is true that only cells assigned an object contribute to the localization loss, that masking comes from the objectness/ground-truth assignment step during training, not from the sigmoid parameterization of the offset itself.
Question 11 · Object Detection · hard
Consider the following scenario and evaluate: Intersection over Union (IoU) for two bounding boxes is: IoU = (area of intersection) / (area of union). If box A is [x1=10, y1=10, x2=50, y2=50] and box B is [x1=30, y1=30, x2=70, y2=70], what is IoU(A, B)?
IoU = 0.25 because the intersection area is (50-30) × (50-30) = 400, which is 1/4 of one box's area
IoU = 0.5 because intersection 20×20=400 and each box is 40×40=1600, so average overlap is 400/(1600+1600) = 0.125, but this incorrectly uses average instead of union
IoU ≈ 0.14: Area A = 40×40=1600. Area B = 40×40=1600. Intersection (overlap) is [30,30] to [50,50], giving (50-30)×(50-30) = 20×20 = 400. Union = 1600+1600-400 = 2800. So IoU = 400/2800 ≈ 0.143
IoU ≈ 0.33 because the intersection height is mistakenly taken as the full 40-unit side of box A instead of the actual 20-unit overlap, giving intersection = 20×40 = 800 and IoU = 800/(1600+1600-800) = 0.33
Answer: C. IoU ≈ 0.14: Area A = 40×40=1600. Area B = 40×40=1600. Intersection (overlap) is [30,30] to [50,50], giving (50-30)×(50-30) = 20×20 = 400. Union = 1600+1600-400 = 2800. So IoU = 400/2800 ≈ 0.143
ExplanationIoU measures bounding box overlap. Box A: (10,10) to (50,50) is 40×40 = 1600 sq units. Box B: (30,30) to (70,70) is 40×40 = 1600 sq units. Intersection region: x1_inter = max(10,30) = 30, y1_inter = max(10,30) = 30, x2_inter = min(50,70) = 50, y2_inter = min(50,70) = 50. Intersection area = (50-30)×(50-30) = 20×20 = 400. Union = area_A + area_B − intersection = 1600 + 1600 − 400 = 2800. IoU = 400/2800 ≈ 0.143, the value used in practice for evaluating detection accuracy and setting NMS overlap thresholds.
Question 12 · Batch/Layer Normalization · hard
A transformer feeds an input tensor of shape (batch=6, seq_len=12, embedding_dim=128) through a normalization layer. Layer Normalization (LN) computes a separate mean and standard deviation for each token, taken across that token's embedding_dim=128 features. Batch Normalization (BN), applied here as it would be to a channel-wise 1D signal, computes a separate mean and standard deviation for each embedding channel, taken across all batch and sequence positions for that channel. How many scalar mean values does each method compute in total, and over how many values is each individual mean averaged?
Layer Norm produces 72 scalar means (one per token, each averaged over 128 features), while Batch Norm produces 128 scalar means (one per channel, each averaged over 72 batch-position pairs)
Batch Norm here yields 72 scalar means, one per token computed over the 128 embedding features, while Layer Norm yields 128 scalar means, one per channel computed over 72 batch-position pairs
Treating the sequence dimension as part of the sample, Layer Norm produces only 6 scalar means — one per batch element averaged jointly over sequence length and embedding — whereas Batch Norm still produces 128 channel means
Because normalization statistics in Batch Norm are computed per training example, this model would produce 6 scalar means for Batch Norm — one per batch item averaged over sequence and embedding jointly — while Layer Norm correctly gives 72 token-level means
Answer: A. Layer Norm produces 72 scalar means (one per token, each averaged over 128 features), while Batch Norm produces 128 scalar means (one per channel, each averaged over 72 batch-position pairs)
ExplanationLN computes one mean per token, i.e. per (batch, seq_len) pair. With batch=6 and seq_len=12 there are 6×12=72 tokens, so LN produces 72 scalar means, and each one is the average of that token's 128 embedding features (72 means, each over 128 values). BN computes one mean per channel, i.e. per embedding_dim index. With embedding_dim=128, BN produces 128 scalar means, and each one is the average taken over all batch×seq_len = 6×12=72 (batch, position) combinations for that channel (128 means, each over 72 values). Swapping these totals — BN yielding 72 means and LN yielding 128 — reverses which axis each method actually reduces over: LN reduces over the feature axis per token, BN reduces over the batch/position axes per channel. Claiming LN pools jointly over both sequence and embedding to give only 6 means describes per-example (instance-level) pooling, not how LN is defined, since LN keeps every token's statistics separate. Claiming BN produces only 6 means, one per batch example, describes per-sample normalization like LN or Instance Norm, not BN, which is defined per channel and must aggregate across the batch and sequence positions, not across channels within one example.
Question 13 · Optimization Algorithms · hard
Learning rate scheduling linearly decreases the learning rate from α_0 = 0.1 to α_f = 0.001 over T=1000 iterations: α_t = α_0 - (α_0 - α_f) * t / T. At t=500 (halfway), α_500 = ? And why might linear decay be suboptimal compared to cosine annealing for deep learning?
α_500 = 0.0505. Linear and cosine annealing schedules are mathematically equivalent at every step, not just the midpoint — the choice between them only affects wall-clock computation time, never the trajectory of the learning rate or the final convergence quality
α_500 = 0.05, treating the schedule as if it simply halves at the midpoint rather than following the exact linear formula. Linear decay is actually the theoretically optimal schedule, and cosine annealing is merely a cosmetic variation with no effect on training dynamics
α_500 = 0.1 - (0.1 - 0.001) * 500 / 1000 = 0.1 - 0.0495 = 0.0505. Linear decay decreases learning rate uniformly. Cosine annealing (α_t = α_f + (α_0 - α_f) * (1 + cos(πt/T)) / 2) is often better because it decays slower early (preserving large steps when loss landscape is steep) and faster late (fine-tuning when near local minima), matching the typical loss curve curvature
α_500 = 0.1 * (1 - 0.5^2) = 0.075, using a quadratic decay formula instead of the linear one specified in the problem. Quadratic schedules are claimed to reduce learning-rate error accumulation, but this conflates the shape of the decay curve with numerical stability, which uniform linear decay does not actually compromise
Answer: C. α_500 = 0.1 - (0.1 - 0.001) * 500 / 1000 = 0.1 - 0.0495 = 0.0505. Linear decay decreases learning rate uniformly. Cosine annealing (α_t = α_f + (α_0 - α_f) * (1 + cos(πt/T)) / 2) is often better because it decays slower early (preserving large steps when loss landscape is steep) and faster late (fine-tuning when near local minima), matching the typical loss curve curvature
ExplanationFirst, linear decay: α_500 = 0.1 - (0.099) * 0.5 = 0.0505. The learning rate drops uniformly from 0.1 toward 0.001, passing through 0.0505 at the halfway point. Then, cosine annealing decays as: α_t = 0.001 + (0.1 - 0.001) * (1 + cos(π * t/1000)) / 2. At t=500: α_500 = 0.001 + 0.099 * (1 + cos(π/2)) / 2 = 0.001 + 0.099 * 1/2 ≈ 0.0505 — the two schedules happen to agree exactly at the midpoint, since cos(π/2) = 0. They diverge everywhere else: cosine annealing keeps the learning rate closer to α_0 during early training (preserving large steps while the loss landscape is steep) and drops it off faster near the end (allowing fine-tuning close to a minimum), which is why it often converges better than uniform linear decay even though the two schedules cross at t=T/2.
Question 14 · Optimization Algorithms · hard
A transformer is trained with linear warmup followed by cosine decay: the learning rate ramps linearly from 0 up to α₀ = 0.02 over warmup_steps = 800, then cosine decay takes over. What is the learning rate at step 200, and why does warmup particularly matter for adaptive optimizers such as Adam?
The linear ramp yields α(200) = 0.02 × (200/800) = 0.005; this schedule is favored because it forces the loss landscape into a convex shape near initialization, which is what guarantees convergence to the global minimum.
Reading the warmup fraction as steps remaining instead of steps completed gives α(200) = 0.02 × (600/800) = 0.015, matching how most optimizer implementations track warmup progress internally.
Treating the ramp as quadratic instead of linear produces α(200) = 0.02 × (200/800)² = 0.00125, a steeper and more conservative initial ramp than the linear schedule the problem describes.
Computing the linear ramp directly, α(200) = 0.02 × (200/800) = 0.005; warmup matters most for adaptive optimizers like Adam because their early gradient moment estimates are based on only a handful of updates and are therefore noisy, so a large learning rate applied to those unreliable estimates can produce destructively large parameter updates before the moving averages settle.
Answer: D. Computing the linear ramp directly, α(200) = 0.02 × (200/800) = 0.005; warmup matters most for adaptive optimizers like Adam because their early gradient moment estimates are based on only a handful of updates and are therefore noisy, so a large learning rate applied to those unreliable estimates can produce destructively large parameter updates before the moving averages settle.
ExplanationLinear warmup follows α(t) = α₀ · (t / warmup_steps) for t ≤ warmup_steps. At t = 200 with α₀ = 0.02 and warmup_steps = 800: α = 0.02 × (200/800) = 0.02 × 0.25 = 0.005. Warmup is especially important for adaptive optimizers such as Adam because the exponential moving averages of the gradient (first moment) and its square (second moment) are computed from very few samples in the earliest steps, making the effective per-parameter update size have high variance. Applying a full-size learning rate to those unreliable early estimates can push weights into a poor region before the averages stabilize; keeping the learning rate small during this window, then handing off to cosine decay once the moment estimates settle, is precisely what warmup accomplishes. Reversing the warmup fraction (steps remaining rather than completed) or substituting a quadratic ramp for the specified linear one are common bookkeeping and schedule-confusion errors, not properties of this problem, and neither convexity nor guaranteed global-minimum convergence has anything to do with why warmup helps a non-convex deep network.
Question 15 · Generative Models · hard
In a Generative Adversarial Network, let y = D(G(z)) be the discriminator's output probability that a generated (fake) image is real. Early in training the generator is still weak, so the discriminator confidently rejects its output, giving y = 0.02. Compare the original minimax generator loss L_G = log(1 − y) with the non-saturating generator loss L_G = −log(y) by computing the magnitude of dL_G/dy at y = 0.02 for each — which comparison correctly explains why the non-saturating loss avoids the vanishing-gradient problem in this regime?
For the same y = 0.02, the minimax loss log(1 − y) actually yields the larger gradient magnitude of 50, while the non-saturating loss −log y yields only about 1.02, so replacing the objective would weaken training signal for a weak generator.
Since D(G(z)) = 0.02 sits so close to zero, gradients from both the minimax and non-saturating losses vanish toward zero in this regime, leaving the generator with no useful learning signal from either formulation.
With D(G(z)) = 0.02, the minimax loss log(1 − y) has |dL_G/dy| = 1/0.98 ≈ 1.02, while the non-saturating loss −log y has |dL_G/dy| = 1/0.02 = 50 — about 49× stronger signal exactly when D confidently rejects early fakes.
Both loss formulations produce an identical gradient magnitude of about 1.02 at y = 0.02 because log(1 − y) and −log y differ only by a constant that cancels under differentiation with respect to y.
Answer: C. With D(G(z)) = 0.02, the minimax loss log(1 − y) has |dL_G/dy| = 1/0.98 ≈ 1.02, while the non-saturating loss −log y has |dL_G/dy| = 1/0.02 = 50 — about 49× stronger signal exactly when D confidently rejects early fakes.
ExplanationFor the minimax objective, L_G = log(1 − y), so dL_G/dy = −1/(1 − y). At y = 0.02 this is −1/0.98 ≈ −1.02, a magnitude of only about 1.02 even though the generator is being badly fooled: this is the vanishing-gradient problem — exactly when the generator most needs a strong corrective push, log(1 − y) is nearly flat because it saturates as y → 0. For the non-saturating objective, L_G = −log(y), so dL_G/dy = −1/y. At y = 0.02 this is −1/0.02 = −50, a magnitude of 50, which is about 49 times larger than the minimax gradient at the same point (50 / 1.02 ≈ 49). This is precisely Goodfellow's 2014 fix: instead of the generator minimizing log(1 − D(G(z))), it minimizes −log D(G(z)) (equivalently maximizes log D(G(z))), and this reparameterization keeps gradients steep exactly where D(G(z)) is small. The two loss curves are not related by an additive constant, so their derivatives genuinely differ everywhere in (0,1) — they only coincide, and both go to −∞, in the opposite limit y → 1, not at y = 0.02.
Question 16 · Object Detection · hard
In YOLO, each grid cell predicts B bounding boxes. If B=3 and you have a 13×13 grid with an image containing 2 dogs and 1 cat (3 objects), how many bounding box predictions are made in total, and why might this cause 'duplicate detections' if NMS is not applied?
Total predictions: 3 (number of objects), regardless of grid size.
Total predictions: 2+1=3 objects × 3 boxes/object = 9 boxes.
Answer: D. Total predictions: 13×13 grid × 3 boxes/cell = 169×3 = 507 bounding boxes.
ExplanationYOLO's architecture divides the image into a 13×13 grid, and each cell outputs B=3 candidate boxes, each carrying (x, y, w, h, confidence) plus class probabilities. Total box predictions: 169 cells × 3 boxes = 507 boxes, regardless of how many real objects (here, 3) are in the image. Duplicate detections arise because the network always emits all 507 boxes: an object's true center falls in one cell, but that cell's other boxes and neighboring cells often also produce high-confidence boxes overlapping the same object. Without non-max suppression (NMS) to discard redundant boxes that have high IoU with a higher-confidence box for the same object, several of these overlapping raw predictions survive as separate reported detections of what is really a single dog or cat.
Question 17 · Object Detection · hard
In Faster R-CNN (and its Mask R-CNN extension), RoIAlign is used in place of RoIPool to extract fixed-size features from each region proposal. Why does this change improve detection accuracy, particularly for small objects?
Bilinear interpolation lets RoIAlign compute feature values at exact fractional sampling points, avoiding the harsh coordinate rounding that RoIPool performs — a misalignment that costs small objects the most since even a few pixels of drift can shift the pooled features off the true object.
A larger set of region proposals is generated by RPN before pooling, when RoIAlign is used, giving small objects more candidate boxes to be classified correctly.
Average pooling replaces the max pooling operation inside RoIAlign, smoothing out noisy activations that would otherwise dominate the pooled features for small objects.
A stricter, lower IoU threshold is applied during NMS by RoIAlign before pooling begins, so fewer overlapping proposals survive to compete with small-object detections.
Answer: A. Bilinear interpolation lets RoIAlign compute feature values at exact fractional sampling points, avoiding the harsh coordinate rounding that RoIPool performs — a misalignment that costs small objects the most since even a few pixels of drift can shift the pooled features off the true object.
ExplanationRoIPool quantizes RoI boundaries and then quantizes again when dividing each cell into a fixed grid, snapping coordinates to the nearest integer twice. This double rounding shifts the pooled feature location relative to the actual object, and the error is proportionally larger for small objects because their whole extent may span only a handful of feature-map cells — a one- or two-pixel shift can move the sample entirely off the object. RoIAlign removes this by sampling feature values at exact, non-quantized fractional locations using bilinear interpolation, then aggregating those samples, so the extracted features stay spatially aligned with the true RoI regardless of object size. This alignment fix, not a change to proposal generation, pooling type, or NMS behavior, was the actual contribution introduced alongside Mask R-CNN. The claim about generating more proposals conflates RoIAlign, a feature-extraction step that runs after RPN and NMS have already fixed the set of proposals, with the proposal-generation stage itself. The claim about swapping in average pooling misdescribes RoIAlign, which works with either max or average aggregation and made no such swap its defining change. The claim about a stricter NMS threshold assigns RoIAlign a suppression role it does not have — NMS thresholds are set during the RPN and classification stages, not inside the pooling operation.
Question 18 · Image Segmentation · hard
Instance Segmentation outputs both pixel-wise class labels and instance IDs. Mask R-CNN extends Faster R-CNN by adding a mask head that predicts a binary mask (foreground/background) for each RoI. For an image with 5 detected objects, what is the shape of the mask predictions, and how are they post-processed?
Mask head outputs 5 binary masks, each of shape (M, M), where M is the mask resolution (typically 14 or 28); each mask is resized back to the RoI's bounding-box size and thresholded to produce the final per-object segmentation
Mask head outputs a single (H, W, num_classes) tensor predicting class probability per pixel, same as semantic segmentation
Mask head outputs (5, num_classes) class probabilities for each of the 5 objects, not spatial masks
Mask head outputs (M, M, 5) where M is mask resolution and 5 is the number of detected objects, stacking all masks in one tensor
Answer: A. Mask head outputs 5 binary masks, each of shape (M, M), where M is the mask resolution (typically 14 or 28); each mask is resized back to the RoI's bounding-box size and thresholded to produce the final per-object segmentation
ExplanationFirst, Mask R-CNN adds a mask branch to Faster R-CNN's outputs. For each RoI (region of interest), the mask head is a small FCN that predicts a binary mask: mask_i ∈ R^(M×M) for object i (M typically 14 or 28). So the output for 5 objects is 5 separate masks, each of shape (M, M), produced independently from 5 separate RoI-pooled feature maps rather than as one combined tensor. Common misconceptions: that masks are class-specific (they're object-instance-specific; a mask for dog_0 is different from dog_1), or that masks predict class probabilities (they're binary foreground/background masks per instance, not a (5, num_classes) probability table). It also differs from semantic segmentation's single shared (H, W, num_classes) map, and from stacking all masks into one (M, M, 5) tensor — that framing wrongly treats the 5 object masks as channels of one shared spatial output, when each is independently generated per RoI and later resized and thresholded against its own bounding box.
Question 19 · Batch/Layer Normalization · hard
Suppose during Group Normalization (GN), instead of normalizing across batch (BN) or features (LN), you divide channels into groups and normalize within each group. For C=256 channels and num_groups=32, each group has 256/32=8 channels. For a sample with shape (H, W, 256), what are the statistics shapes and why is GN useful?
GN computes 1 (μ, σ) pair shared across all channels and spatial dims, similar to instance normalization but global
GN computes 256 (μ, σ) pairs, one per channel, identical to LN but over spatial dimensions instead of feature dimensions
GN computes 32 (μ, σ) pairs, one per group.
GN requires multiple samples in a batch to compute meaningful statistics.
Answer: C. GN computes 32 (μ, σ) pairs, one per group.
ExplanationFirst, group Normalization (GN) is a middle ground between BN and LN. For C=256 channels divided into G=32 groups: each group has C/G = 8 channels. Then, normalization is per-sample, per-group: for each sample and each group, compute μ and σ over the group's 8 channels and all spatial positions (H, W). Common misconceptions: that GN requires large batch_size (it doesn't; it's independent), or that GN is identical to LN (LN is a single (μ, σ) per sample; GN is multiple per-group stats). Thus, with G=32 groups per sample, GN produces 32 (μ, σ) pairs — the correct answer — making it batch-size independent and effective for small-batch training such as object detection and segmentation.
Question 20 · Reinforcement Learning · hard
Consider a policy gradient update of the form ∇J(θ) ≈ (1/N) Σᵢ ∇log π(aᵢ|sᵢ;θ) · (Gᵢ − b(sᵢ)), where Gᵢ is the sampled return following action aᵢ taken in state sᵢ. Two baselines are compared for the same batch of trajectories: b₁(s) = 0 (no baseline at all) and b₂(s) = V^π(s), the true state-value function. Which statement correctly describes how the expected value and the variance of the resulting gradient estimate differ between using b₁ and using b₂?
Subtracting a state-only baseline like V^π(s) does not change the expected gradient, because summing π(a|s;θ)∇log π(a|s;θ) over all actions equals ∇θ of the total probability, which is always zero; however, using b₂ = V^π(s) shrinks how far (Gᵢ − b(sᵢ)) spreads from zero for each state, which typically lowers the variance of the gradient estimate compared to using no baseline at all.
Choosing b(s) = V^π(s) biases the gradient estimate downward whenever the sampled return exceeds the state's average, since the subtraction systematically removes positive contributions before the expectation over actions is taken.
The zero-baseline estimator has lower variance than the value-function baseline, because subtracting V^π(s) removes information about which actions perform better than the state's average, leaving only noise in the remaining signal.
Both estimators remain unbiased, but pairing sampled returns with the value function increases variance rather than reducing it, since V^π(s) introduces an extra stochastic term that adds to the randomness already present in the sampled return G.
Answer: A. Subtracting a state-only baseline like V^π(s) does not change the expected gradient, because summing π(a|s;θ)∇log π(a|s;θ) over all actions equals ∇θ of the total probability, which is always zero; however, using b₂ = V^π(s) shrinks how far (Gᵢ − b(sᵢ)) spreads from zero for each state, which typically lowers the variance of the gradient estimate compared to using no baseline at all.
ExplanationThe key fact about policy-gradient baselines is that subtracting any function of the state alone — never the action — leaves the expected gradient unchanged. For a fixed state s, Σ_a π(a|s;θ)∇log π(a|s;θ) = ∇θ Σ_a π(a|s;θ) = ∇θ(1) = 0, so E_{a~π(·|s)}[∇log π(a|s;θ) · b(s)] = b(s) · 0 = 0 for any choice of b(s), including b₂(s) = V^π(s). This means both b₁ = 0 and b₂ = V^π(s) give an unbiased estimate of ∇J(θ) — the baseline term contributes nothing to the expectation regardless of its value, so neither choice introduces bias. Where the two baselines differ is variance. The quantity (Gᵢ − b(sᵢ)) measures how much better or worse a sampled return is than the baseline. With b₁ = 0, this is just the raw return, which can swing widely across episodes purely from environment and policy stochasticity, even when the actions taken were equally good. With b₂ = V^π(s), the term (Gᵢ − V^π(sᵢ)) is centered near zero for each state — it isolates how much better this particular trajectory did than what was already expected from that state — so its typical magnitude, and therefore the variance of the gradient estimate, is much smaller. This is precisely why REINFORCE-with-baseline trains faster and more stably than plain REINFORCE: the expected update direction is identical, but the sampled updates used to estimate it are far less noisy.