Analyze a MaxPool2d(kernel_size=2, stride=2) layer applied to a feature map of shape (32, 64, 28, 28). Calculate the output shape, explain why max pooling preserves spatial locality better than average pooling in CNNs, and derive how it affects gradient flow during backpropagation?
The output shape is (32, 64, 14, 14) because max pooling selects the highest activation per region, which preserves feature importance and produces sparse gradients that flow only through max locations (dL/dx = dL/dy_max at the max-index position only), whereas average pooling distributes gradients uniformly across all positions and dilutes critical feature signals
Given kernel_size=2, MaxPool2d halves the channel dimension rather than the spatial dimensions, producing an output shape of (32, 32, 28, 28)
Max pooling and average pooling have identical gradient behavior because both reduce spatial size by factor of 2
With kernel_size=2 and stride=2, each spatial dimension shrinks by exactly 1 pixel per pooling step, giving an output shape of (32, 64, 27, 27)
Answer: A. The output shape is (32, 64, 14, 14) because max pooling selects the highest activation per region, which preserves feature importance and produces sparse gradients that flow only through max locations (dL/dx = dL/dy_max at the max-index position only), whereas average pooling distributes gradients uniformly across all positions and dilutes critical feature signals
ExplanationFirst, MaxPool2d: output_size = (28 - 2)/2 + 1 = 14, so the output shape is (32, 64, 14, 14). Max pooling selects the maximum value per 2×2 region. During the backward pass, gradients flow only through the position that produced the max (sparse), focusing the gradient signal on high-activation features like edges and textures. Average pooling instead computes y = (x_1+x_2+x_3+x_4)/4, so all four positions receive an equal gradient dL/dx_i = dL/dy/4, distributing gradients uniformly across the region. This means max pooling's sparse gradient pattern emphasizes important features, while average pooling's uniform distribution dilutes the signal from any single dominant feature.
Question 42 · NLP · hard
In the skip-gram model trained with negative sampling, one training step for a single (target, context) word pair scores the true context word plus k negative samples drawn from the vocabulary — each scored with a dot product followed by a sigmoid — instead of scoring all V vocabulary words the way full softmax does. For a vocabulary of V = 20,000 words and k = 4 negative samples per positive pair, what speedup factor does negative sampling achieve over full softmax, measured as the ratio of dot products computed per training step?
Negative sampling scores 5 dot products per training step (1 positive pair + 4 negative pairs) against full softmax's 20,000, giving a speedup factor of 20,000 / 5 = 4,000x.
Only the four negative samples are scored per step, so the speedup factor is 20,000 / 4 = 5,000x, since the positive pair's score is already available for free from the initial embedding lookup.
Hierarchical softmax and negative sampling achieve the same speedup here because both replace full softmax with binary-tree traversal, giving a speedup factor of log2(20,000) ≈ 14.3x.
Because negative sampling still must normalize probabilities across the full vocabulary during backpropagation, the speedup factor is only 2x, reflecting a halved forward-pass cost.
Answer: A. Negative sampling scores 5 dot products per training step (1 positive pair + 4 negative pairs) against full softmax's 20,000, giving a speedup factor of 20,000 / 5 = 4,000x.
ExplanationFull softmax must compute a dot-product score for every word in the vocabulary in order to normalize the probability distribution, so it performs V = 20,000 dot products per training step. Negative sampling replaces this with a binary classification objective: maximize log(σ(v_target · v_context)) for the one true context word, plus Σ log(σ(−v_target · v_neg)) for each of the k = 4 randomly drawn negative words. That is 1 positive pair + 4 negative pairs = 5 dot-product-and-sigmoid evaluations total, none of which require summing over the rest of the vocabulary — no partition function is computed at all. The speedup in dot products per step is therefore V / (k+1) = 20,000 / 5 = 4,000x. The claim that only the 4 negative samples need scoring ignores that the positive pair's score is exactly what the objective is built around — it must be computed explicitly, not obtained "for free." The claim invoking log2(V) ≈ 14.3x describes hierarchical softmax, a different technique that replaces the flat V-way classification with a binary-tree path of decisions — it does not apply to negative sampling, which uses no tree at all. The claim that negative sampling still normalizes over the full vocabulary is the core misconception the technique is designed to eliminate: negative sampling deliberately avoids computing the softmax partition function, which is precisely why it is so much faster than full softmax.
Question 43 · CNN architecture · hard
Given a Conv2D layer: nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1) followed by nn.MaxPool2d(2), with input tensor shape [32, 3, 224, 224], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [32, 64, 224, 224], after MaxPool: [32, 64, 112, 112]; params = 64*(3*3*3+1) = 1,792; FLOPS = 86,704,128; memory = 98.0 MB, because each filter slides across spatial dimensions computing dot products
Output: [32, 3, 224, 224] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [32, 64, 448, 448] because convolution upsamples by factor of stride, doubling spatial resolution
Parameters = 576 without bias, and FLOPS = 192 because only channel-wise operations count toward computation, when in fact each of the 64 filters must convolve across all 3 input channels rather than just one
Answer: A. Output after Conv2D: [32, 64, 224, 224], after MaxPool: [32, 64, 112, 112]; params = 64*(3*3*3+1) = 1,792; FLOPS = 86,704,128; memory = 98.0 MB, because each filter slides across spatial dimensions computing dot products
ExplanationStep-by-step calculation: (1) Conv2D output spatial size: H_out = floor((224 + 2×1 - 3)/1) + 1 = floor((223)/1) + 1 = 224. W_out = 224 (same formula). (2) Output shape = [32, 64, 224, 224]. (3) MaxPool2d(2): H = 224/2 = 112, W = 224/2 = 112. Final: [32, 64, 112, 112]. (4) Parameters: 64 filters × (3×3×3 weights + 1 bias) = 64 × 28 = 1,792. (5) FLOPS: 64 × 3×3×3 × 224×224 = 86,704,128 FLOPs. Therefore, the memory for feature maps = 32 × 64 × 112 × 112 × 4 bytes = 98.0 MB. Because backpropagation requires storing activations for gradient computation, the actual training memory is approximately 2-3× this value.
Question 44 · CNN architecture · hard
A CNN layer is defined as Conv2D(in_channels=64, out_channels=128, kernel_size=5, stride=1, padding=2) and applied to an input tensor of shape [16, 64, 32, 32], where 16 is the batch size; what are the output tensor's spatial dimensions and the total number of trainable parameters (including biases) in this layer?
With padding=2 and stride=1 exactly preserving spatial resolution for a 5×5 kernel, the output tensor has shape [16, 128, 32, 32] and the layer contains 204,928 trainable parameters, computed as 128 × (5×5×64 + 1) = 128 × 1601 = 204,928.
Ignoring the padding's effect on spatial size, the output tensor would be [16, 128, 28, 28] with 204,800 trainable parameters, computed as 128 × 5 × 5 × 64 without adding the bias term.
Treating each output channel as needing only one weight per input channel regardless of kernel size, the parameter count would be just 8,192 (128 × 64), while the output shape stays [16, 128, 32, 32].
Assuming padding always doubles the input's spatial dimensions before convolution, the output tensor would be [16, 128, 64, 64], despite the parameter count still equaling 204,928.
Answer: A. With padding=2 and stride=1 exactly preserving spatial resolution for a 5×5 kernel, the output tensor has shape [16, 128, 32, 32] and the layer contains 204,928 trainable parameters, computed as 128 × (5×5×64 + 1) = 128 × 1601 = 204,928.
ExplanationFor a convolution with stride 1, the output spatial size follows H_out = floor((H_in + 2P − K)/S) + 1. Substituting H_in=32, P=2, K=5, S=1 gives floor((32+4−5)/1)+1 = floor(31)+1 = 32, so the spatial dimensions stay 32×32 — this is the "same padding" condition, which holds for stride 1 whenever P = (K−1)/2. The output shape is therefore [16, 128, 32, 32]. Each of the 128 output filters convolves over all 64 input channels with a 5×5 receptive field, giving 5×5×64 = 1600 weights per filter, plus 1 bias per filter, for 1601 parameters each; multiplying by 128 output filters gives 128 × 1601 = 204,928 trainable parameters. The reasoning that ignores padding's role in preserving spatial size lands on 28×28 instead of 32×32, and separately drops the bias term, undercounting parameters as 204,800. The reasoning that treats the parameter count as independent of kernel size (128 × 64 = 8,192) overlooks that every position within the 5×5 kernel window carries its own learned weight per input channel — a 1×1 kernel would give that count, not a 5×5 one. The reasoning that has padding double the spatial dimensions confuses zero-padding, which adds border pixels to help preserve resolution during a normal convolution, with upsampling operations such as transposed convolution.
Question 45 · CNN architecture · hard
Given a Conv2D layer: nn.Conv2d(in_channels=3, out_channels=32, kernel_size=5, stride=1, padding=2) followed by nn.MaxPool2d(2), with input tensor shape [64, 3, 32, 32], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [64, 32, 32, 32], after MaxPool: [64, 32, 16, 16]; params = 32*(5*5*3+1) = 2,432; FLOPS = 2,457,600; memory = 2.0 MB, because each filter slides across spatial dimensions computing dot products
Output: [64, 3, 32, 32] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [64, 32, 64, 64] because convolution upsamples by factor of stride, doubling spatial resolution
Parameters = 800 without bias, and FLOPS = 96 because only channel-wise operations count toward computation
Answer: A. Output after Conv2D: [64, 32, 32, 32], after MaxPool: [64, 32, 16, 16]; params = 32*(5*5*3+1) = 2,432; FLOPS = 2,457,600; memory = 2.0 MB, because each filter slides across spatial dimensions computing dot products
Given a Conv2D layer: nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1) followed by nn.MaxPool2d(2), with input tensor shape [8, 256, 28, 28], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [8, 512, 28, 28], after MaxPool: [8, 512, 14, 14]; params = 512*(3*3*256+1) = 1,180,160; FLOPS = 924,844,032; memory = 3.1 MB, because each filter slides across spatial dimensions computing dot products
Output: [8, 256, 28, 28] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [8, 512, 56, 56] because convolution upsamples by factor of stride, doubling spatial resolution — this interpretation would only apply to a significantly different algorithm with different preconditions and postconditions
Parameters = 4608 without bias, and FLOPS = 131072 because only channel-wise operations count toward computation
Answer: A. Output after Conv2D: [8, 512, 28, 28], after MaxPool: [8, 512, 14, 14]; params = 512*(3*3*256+1) = 1,180,160; FLOPS = 924,844,032; memory = 3.1 MB, because each filter slides across spatial dimensions computing dot products
ExplanationStep-by-step calculation: (1) Conv2D output spatial size: H_out = floor((28 + 2×1 - 3)/1) + 1 = floor((27)/1) + 1 = 28. W_out = 28 (same formula). (2) Output shape = [8, 512, 28, 28]. (3) MaxPool2d(2): H = 28/2 = 14, W = 28/2 = 14. Final: [8, 512, 14, 14]. (4) Parameters: 512 filters × (3×3×256 weights + 1 bias) = 512 × 2305 = 1,180,160. (5) FLOPS: 512 × 3×3×256 × 28×28 = 924,844,032 multiply-add operations. Therefore, the memory for feature maps = 8 × 512 × 14 × 14 × 4 bytes = 3.1 MB. Because backpropagation requires storing activations for gradient computation, the actual training memory is approximately 2-3× this value.
Question 47 · CNN architecture · hard
Given a Conv2D layer: nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, stride=2, padding=0), with input tensor shape [32, 3, 299, 299], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [32, 32, 149, 149]; params = 32*(3*3*3+1) = 896; FLOPS = 19,181,664; memory = 86.7 MB, because each filter slides across spatial dimensions computing dot products
Output: [32, 3, 299, 299] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [32, 32, 598, 598] because convolution upsamples by factor of stride, doubling spatial resolution
Parameters = 288 without bias, and FLOPS = 96 because only channel-wise operations count toward computation
Answer: A. Output after Conv2D: [32, 32, 149, 149]; params = 32*(3*3*3+1) = 896; FLOPS = 19,181,664; memory = 86.7 MB, because each filter slides across spatial dimensions computing dot products
ExplanationStep-by-step calculation: (1) Conv2D output spatial size: H_out = floor((299 + 2×0 - 3)/2) + 1 = floor((296)/2) + 1 = 149. W_out = 149 (same formula). (2) Output shape = [32, 32, 149, 149]. (3) Parameters: 32 filters × (3×3×3 weights + 1 bias) = 32 × 28 = 896. (4) FLOPS: 32 × 3×3×3 × 149×149 = 19,181,664 multiply-add operations. Therefore, the memory for feature maps = 32 × 32 × 149 × 149 × 4 bytes = 86.7 MB. Because backpropagation requires storing activations for gradient computation, the actual training memory is approximately 2-3× this value.
Question 48 · CNN architecture · hard
Given a Conv2D layer: nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1), with input tensor shape [16, 512, 14, 14], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [16, 512, 14, 14]; params = 512*(3*3*512+1) = 2,359,808; FLOPS = 462,422,016; memory = 6.1 MB, because each filter slides across spatial dimensions computing dot products
Output: [16, 512, 14, 14] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [16, 512, 28, 28] because convolution upsamples by factor of stride, doubling spatial resolution
Parameters = 4608 without bias, and FLOPS = 262144 because only channel-wise operations count toward computation
Answer: A. Output after Conv2D: [16, 512, 14, 14]; params = 512*(3*3*512+1) = 2,359,808; FLOPS = 462,422,016; memory = 6.1 MB, because each filter slides across spatial dimensions computing dot products
ExplanationStep-by-step calculation: (1) Conv2D output spatial size: H_out = floor((14 + 2×1 - 3)/1) + 1 = floor((13)/1) + 1 = 14. W_out = 14 (same formula). (2) Output shape = [16, 512, 14, 14]. (3) Parameters: 512 filters × (3×3×512 weights + 1 bias) = 512 × 4609 = 2,359,808. (4) FLOPS: 512 × 3×3×512 × 14×14 = 462,422,016 multiply-add operations. Therefore, the memory for feature maps = 16 × 512 × 14 × 14 × 4 bytes = 6.1 MB. Because backpropagation requires storing activations for gradient computation, the actual training memory is approximately 2-3× this value.
Question 49 · CNN architecture · hard
Given a Conv2D layer: nn.Conv2d(in_channels=64, out_channels=64, kernel_size=1, stride=1, padding=0), with input tensor shape [32, 64, 56, 56], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [32, 64, 56, 56]; params = 64*(1*1*64+1) = 4,160; FLOPS = 411,041,792 (across the batch of 32); memory = 24.5 MB, because each filter slides across spatial dimensions computing dot products
Output: [32, 64, 56, 56] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [32, 64, 112, 112] because convolution upsamples by factor of stride, doubling spatial resolution
Parameters = 64 without bias, and FLOPS = 4096 because only channel-wise operations count toward computation
Answer: A. Output after Conv2D: [32, 64, 56, 56]; params = 64*(1*1*64+1) = 4,160; FLOPS = 411,041,792 (across the batch of 32); memory = 24.5 MB, because each filter slides across spatial dimensions computing dot products
ExplanationStep-by-step calculation: (1) Conv2D output spatial size: H_out = floor((56 + 2×0 - 1)/1) + 1 = floor((55)/1) + 1 = 56. W_out = 56 (same formula). (2) Output shape = [32, 64, 56, 56]. (3) Parameters: 64 filters × (1×1×64 weights + 1 bias) = 64 × 65 = 4,160 (parameter count does not depend on batch size, since weights are shared across every image in the batch). (4) FLOPS per image: 64 × 1×1×64 × 56×56 = 12,845,056 multiply-add operations; since the input tensor's batch dimension is 32, the total FLOPS for the full batch = 12,845,056 × 32 = 411,041,792 multiply-add operations. (5) Memory for feature maps = 32 × 64 × 56 × 56 × 4 bytes = 24.5 MB, which also accounts for the batch dimension of 32. Because backpropagation requires storing activations for gradient computation, the actual training memory is approximately 2-3× this value.
Question 50 · CNN architecture · hard
Given a Conv2D layer: nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1), with input tensor shape [32, 128, 28, 28], calculate the output dimensions, parameter count, and FLOPS, then analyze the memory footprint?
Output after Conv2D: [32, 128, 28, 28]; params = 128*(3*3*128+1) = 147,584; FLOPS = 115,605,504; memory = 12.3 MB, because each filter slides across spatial dimensions computing dot products
Output: [32, 128, 28, 28] unchanged because padding preserves all dimensions regardless of kernel size and stride
Output: [32, 128, 56, 56] because convolution upsamples by factor of stride, doubling spatial resolution
Parameters = 1152 without bias, and FLOPS = 16384 because only channel-wise operations count toward computation
Answer: A. Output after Conv2D: [32, 128, 28, 28]; params = 128*(3*3*128+1) = 147,584; FLOPS = 115,605,504; memory = 12.3 MB, because each filter slides across spatial dimensions computing dot products
ExplanationStep-by-step calculation: (1) Conv2D output spatial size: H_out = floor((28 + 2×1 - 3)/1) + 1 = floor((27)/1) + 1 = 28. W_out = 28 (same formula). (2) Output shape = [32, 128, 28, 28]. (3) Parameters: 128 filters × (3×3×128 weights + 1 bias) = 128 × 1153 = 147,584. (4) FLOPS: 128 × 3×3×128 × 28×28 = 115,605,504 multiply-add operations. Therefore, the memory for feature maps = 32 × 128 × 28 × 28 × 4 bytes = 12.3 MB. Because backpropagation requires storing activations for gradient computation, the actual training memory is approximately 2-3× this value.
Question 51 · LSTM architecture · hard
Consider the following PyTorch layer definition:
```python
import torch.nn as nn
lstm = nn.LSTM(input_size=32, hidden_size=64, num_layers=1, batch_first=True)
```
For this single-layer LSTM, what is the total number of trainable parameters, and which weight matrix type — input-to-hidden or hidden-to-hidden — contributes the larger share?
Total trainable parameters = 4×[(64×32)+(64×64)+64+64] = 25,088; the hidden-to-hidden weight matrices contribute 16,384 parameters (roughly two-thirds of the total), more than the 8,192 contributed by the input-to-hidden matrices, because hidden_size (64) exceeds input_size (32)
A single weight matrix pair transforms the input and hidden state at each time step, giving total trainable parameters = (64×32)+(64×64)+64+64 = 6,272, since PyTorch shares one set of weights across all gate computations
The total trainable parameter count is 25,088, but the input-to-hidden weight matrices contribute the larger share at 16,384 parameters compared to 8,192 parameters from the hidden-to-hidden matrices
Because the LSTM cell uses three gating mechanisms rather than four, total trainable parameters = 3×[(64×32)+(64×64)+64+64] = 18,816
Answer: A. Total trainable parameters = 4×[(64×32)+(64×64)+64+64] = 25,088; the hidden-to-hidden weight matrices contribute 16,384 parameters (roughly two-thirds of the total), more than the 8,192 contributed by the input-to-hidden matrices, because hidden_size (64) exceeds input_size (32)
ExplanationEach of the 4 LSTM gates (forget, input, cell candidate, output) has its own input-to-hidden weight matrix of shape (hidden_size × input_size) = 64×32 = 2,048 parameters, its own hidden-to-hidden weight matrix of shape (hidden_size × hidden_size) = 64×64 = 4,096 parameters, and its own pair of bias vectors (bias_ih and bias_hh, each of length hidden_size) totaling 64+64 = 128 parameters. Per gate that is 2,048 + 4,096 + 128 = 6,272 parameters, and across all 4 gates: 4 × 6,272 = 25,088 total trainable parameters. Grouping by matrix type instead of by gate: the input-to-hidden matrices sum to 4 × 2,048 = 8,192 parameters, while the hidden-to-hidden matrices sum to 4 × 4,096 = 16,384 parameters — exactly double, since hidden_size (64) is twice input_size (32). The hidden-to-hidden matrices therefore account for the larger share, about 65% (16,384 / 25,088) of the layer's parameters, which is why increasing hidden_size grows LSTM parameter count faster than increasing input_size does.
Question 52 · GRU architecture · hard
Given nn.GRU(input_size=64, hidden_size=128, num_layers=1, batch_first=True) processing an input tensor of shape [32, 20, 64], what is the total number of trainable parameters in this GRU layer, and how does the update gate z_t mathematically combine the previous hidden state h_{t-1} with the candidate hidden state ñ_t to produce h_t?
This GRU layer has 74,496 trainable parameters, since each of the 3 gates contributes W_ih (128×64) + W_hh (128×128) + two bias vectors (128+128) = 24,832 parameters, and the update gate blends states as h_t = (1 − z_t) ⊙ ñ_t + z_t ⊙ h_{t−1}, so z_t near 1 keeps the previous hidden state nearly unchanged.
Because GRU cells omit bias vectors entirely, the total comes to 73,728 parameters, calculated as 3 × (64×128 + 128×128) across the single layer, with the update gate combining states as h_t = (1 − z_t) ⊙ ñ_t + z_t ⊙ h_{t−1}.
Treating this as a four-gate cell like an LSTM gives 99,328 parameters in total, and a reset gate value near 0 causes the network to fully carry forward the previous hidden state into the candidate computation unchanged.
With 74,496 total trainable parameters correctly computed across the 3 gates, the hidden state update actually follows h_t = z_t ⊙ ñ_t + (1 − z_t) ⊙ h_{t−1}, so a value of z_t near 1 causes the layer to adopt the freshly computed candidate state.
Answer: A. This GRU layer has 74,496 trainable parameters, since each of the 3 gates contributes W_ih (128×64) + W_hh (128×128) + two bias vectors (128+128) = 24,832 parameters, and the update gate blends states as h_t = (1 − z_t) ⊙ ñ_t + z_t ⊙ h_{t−1}, so z_t near 1 keeps the previous hidden state nearly unchanged.
ExplanationEach GRU layer contains 3 gates — reset (r_t), update (z_t), and candidate/new (ñ_t) — and each gate has its own input-to-hidden weight matrix W_ih of shape (hidden_size, input_size), hidden-to-hidden weight matrix W_hh of shape (hidden_size, hidden_size), and two separate bias vectors b_ih and b_hh of length hidden_size. For this layer: W_ih is 128×64 = 8,192 parameters, W_hh is 128×128 = 16,384 parameters, and the two biases add 128+128 = 256 parameters, giving 8,192+16,384+256 = 24,832 parameters per gate. Multiplying by the 3 gates gives 3 × 24,832 = 74,496 total trainable parameters; since num_layers=1, there is no further multiplication across layers. The batch size (32) and sequence length (20) in the input tensor do not affect the parameter count at all, because the same weight matrices are reused at every one of the 20 time steps. For the hidden-state update, the update gate combines states as h_t = (1 − z_t) ⊙ ñ_t + z_t ⊙ h_{t−1}: z_t is a sigmoid output between 0 and 1 that acts as an interpolation weight, so when z_t is close to 1 the layer keeps the previous hidden state h_{t−1} almost unchanged (letting information persist across time steps), and when z_t is close to 0 the hidden state is replaced almost entirely by the freshly computed candidate ñ_t.
Question 53 · LSTM architecture · hard
Given nn.LSTM(input_size=128, hidden_size=256, num_layers=2, batch_first=True) with the default bias=True, processing an input tensor of shape [32, 200, 128], what is the total number of trainable parameters in this LSTM, and how does the forget gate's sigmoid output regulate the cell state across all 200 time steps?
921,600 total parameters, since each layer contributes 4 gates of size (input_size×hidden_size + hidden_size² + 2×hidden_size), with Layer 2 correctly using input_size = hidden_size = 256 because it consumes Layer 1's hidden output
Only 917,504 parameters result if the two bias vectors PyTorch adds per gate are left out and just the input-to-hidden and hidden-to-hidden weight matrices are counted for all 4 gates
Applying input_size = 128 to both layers instead of switching Layer 2 to the 256-dimensional hidden state it actually receives yields 790,528 parameters
Treating the LSTM cell as using one combined weight matrix per layer rather than four independent gates for forget, input, cell, and output gives 230,400 parameters
Answer: A. 921,600 total parameters, since each layer contributes 4 gates of size (input_size×hidden_size + hidden_size² + 2×hidden_size), with Layer 2 correctly using input_size = hidden_size = 256 because it consumes Layer 1's hidden output
ExplanationFor nn.LSTM(input_size=128, hidden_size=256, num_layers=2, batch_first=True) with bias=True, each of the 4 gates (forget, input, cell, output) has its own W_ih, W_hh, and two bias vectors b_ih and b_hh, since PyTorch keeps every gate's parameters separate rather than fusing them. Layer 1: input_size=128, hidden_size=256, so each gate has 128×256 (W_ih) + 256×256 (W_hh) + 256 (b_ih) + 256 (b_hh) = 32,768 + 65,536 + 512 = 98,816 parameters; across 4 gates that is 395,264. Layer 2 receives Layer 1's 256-dimensional hidden output as its input, so its effective input_size is 256, not 128: each gate has 256×256 + 256×256 + 512 = 131,584 parameters, giving 526,336 across 4 gates. Summing both layers: 395,264 + 526,336 = 921,600 total trainable parameters. At every one of the 200 time steps, the forget gate applies a sigmoid to a linear combination of x_t and h_{t-1}, producing values in (0,1) that scale c_{t-1} element-wise before new candidate values (from the input and cell gates) are added to form c_t; because this pathway is additive and multiplicative by a gate that can sit near 1.0, gradients flowing backward through the cell state avoid the repeated shrinking that causes vanishing gradients in a vanilla RNN, while the output gate's sigmoid separately controls how much of the updated c_t is exposed as h_t.
Question 54 · LSTM architecture · hard
Given nn.LSTM(input_size=768, hidden_size=512, num_layers=2, batch_first=True) with input tensor [8, 512, 768], calculate the total parameter count and analyze how the 4 gates (forget, input, cell, output) control information flow through 512 time steps?
Total parameters = 4 gates × (input_size×hidden + hidden×hidden + bias) per layer = 4,726,784 for 2 layers; the forget, input, cell, output gates use sigmoid/tanh activations to control what information passes through, because each gate learns independent weight matrices
Parameters = 768 × 512 = 393,216 because LSTM only has one weight matrix connecting input to hidden state, ignoring the separate input-to-hidden and hidden-to-hidden matrices that each of the four gates requires
The LSTM processes all 512 time steps simultaneously in parallel because recurrent connections are computed via matrix multiplication across the sequence dimension
Parameters = 512² = 262,144 because only hidden-to-hidden connections have trainable weights, ignoring the input-to-hidden weight matrices and bias terms that each gate also learns
Answer: A. Total parameters = 4 gates × (input_size×hidden + hidden×hidden + bias) per layer = 4,726,784 for 2 layers; the forget, input, cell, output gates use sigmoid/tanh activations to control what information passes through, because each gate learns independent weight matrices
ExplanationStep-by-step parameter calculation: (1) LSTM has 4 gates: forget, input, cell, output. (2) Layer 1 parameters: each gate has W_ih (768×512) + W_hh (512×512) + b_ih + b_hh. Per gate: 768×512 + 512×512 + 2×512 = 656384. All 4 gates: 4 × 656384 = 2625536. (3) Layer 2 parameters: input_size = hidden_size = 512, so per gate = 512×512 + 512×512 + 2×512 = 525312. All 4 gates: 4 × 525312 = 2101248. Therefore total = Layer1 + Layer2 = 2625536 + 2101248 = 4,726,784 parameters. Because the forget gate (sigmoid output 0-1) controls how much previous cell state to retain, vanishing gradients are mitigated: gradient flows through the cell state with multiplicative factor close to 1.0 when forget gate ≈ 1.
Question 55 · GRU architecture · hard
Consider the following PyTorch layer definition:
```python
import torch.nn as nn
gru = nn.GRU(input_size=64, hidden_size=128, num_layers=1, batch_first=True, bias=True)
```
How many total learnable parameters does this single-layer GRU contain?
Multiplying three gates by [128×64 + 128×128 + 128 + 128] gives 74,496 learnable parameters, since each of the reset, update, and candidate computations has its own input-to-hidden weight matrix, hidden-to-hidden weight matrix, and two bias vectors, exactly as PyTorch implements nn.GRU.
Counting only the reset and update gates as independently parameterized gives 2 × [128×64 + 128×128 + 128 + 128] = 49,664 learnable parameters, with the candidate hidden state computed by reusing the update gate's weight matrices.
Using one shared bias vector per gate instead of two gives 3 × [128×64 + 128×128 + 128] = 74,112 learnable parameters, since the input-side and hidden-side projections would be added together before a single bias term is applied.
Applying a single weight matrix and bias pair across all three gates gives 128×64 + 128×128 + 128 + 128 = 24,832 learnable parameters, treating the reset, update, and candidate computations as sharing one transformation.
Answer: A. Multiplying three gates by [128×64 + 128×128 + 128 + 128] gives 74,496 learnable parameters, since each of the reset, update, and candidate computations has its own input-to-hidden weight matrix, hidden-to-hidden weight matrix, and two bias vectors, exactly as PyTorch implements nn.GRU.
ExplanationA GRU layer creates three independent gate computations per layer: the reset gate, the update gate, and the candidate (new) hidden state. PyTorch's nn.GRU gives each of these three computations its own input-to-hidden weight matrix of shape (hidden_size, input_size), its own hidden-to-hidden weight matrix of shape (hidden_size, hidden_size), and two separate bias vectors (bias_ih and bias_hh) of length hidden_size. For input_size=64 and hidden_size=128, one gate's parameters are 128×64 + 128×128 + 128 + 128 = 8,192 + 16,384 + 128 + 128 = 24,832. Multiplying by the three gates gives 3 × 24,832 = 74,496 total learnable parameters for this single GRU layer. Unlike an LSTM, a GRU has no cell state and no forget gate: the update gate alone controls, for every hidden unit, the blend between the previous hidden state and the newly computed candidate state, while the reset gate determines how much of the previous hidden state is allowed to influence that candidate computation.
Question 56 · LSTM architecture · hard
Given nn.LSTM(input_size=64, hidden_size=128, num_layers=2, batch_first=True) processing an input tensor of shape [8, 20, 64], what is the total number of trainable parameters, and how does the forget gate help gradients survive across the 20 time steps?
Total trainable parameters equal 231,424 — 99,328 from the first LSTM layer (which reads the 64-dimensional input) plus 132,096 from the second layer (which reads the previous layer's 128-dimensional hidden state) — and the forget gate's sigmoid output near 1 lets gradients flow through the cell state across all 20 time steps largely unattenuated, curbing vanishing gradients.
Applying the 64-dimensional input size to every layer, including the second, gives 198,656 trainable parameters, while the output gate alone determines how much of the cell state reaches the next time step's hidden state.
Excluding both bias vectors from the count, on the reasoning that PyTorch fixes LSTM biases rather than training them, yields 229,376 trainable parameters.
Treating the forget and input gates as sharing a single weight matrix, so that only three independent gates remain, produces 173,568 trainable parameters.
Answer: A. Total trainable parameters equal 231,424 — 99,328 from the first LSTM layer (which reads the 64-dimensional input) plus 132,096 from the second layer (which reads the previous layer's 128-dimensional hidden state) — and the forget gate's sigmoid output near 1 lets gradients flow through the cell state across all 20 time steps largely unattenuated, curbing vanishing gradients.
ExplanationEach LSTM layer has 4 gates (forget, input, candidate/cell, output), and PyTorch gives every gate its own input-to-hidden matrix W_ih, hidden-to-hidden matrix W_hh, and two bias vectors b_ih and b_hh (matching the cuDNN kernel layout). Layer 1 reads the raw 64-dimensional input, so per gate it needs 64×128 (W_ih) + 128×128 (W_hh) + 128 + 128 (biases) = 8,192 + 16,384 + 256 = 24,832 parameters; across 4 gates that totals 99,328. Layer 2 does not see the original 64-dimensional input at all — it consumes the 128-dimensional hidden state produced by layer 1 — so per gate it needs 128×128 + 128×128 + 128 + 128 = 33,024 parameters, totaling 132,096 across 4 gates. Summing both layers: 99,328 + 132,096 = 231,424 trainable parameters. Across the 20 time steps, the forget gate outputs a sigmoid value in (0,1) that scales the previous cell state c(t-1) before new candidate information is added; when this gate saturates near 1, the multiplicative term in the cell-state recurrence stays close to 1, so gradients backpropagated through the cell state avoid the repeated small-magnitude multiplications that cause vanishing gradients in a plain RNN.
Question 57 · GRU architecture · hard
Given `nn.GRU(input_size=100, hidden_size=50, num_layers=1, batch_first=True)` with the default bias enabled, what is the total number of trainable parameters, and which structural feature of the GRU's gate design does that count reflect?
This configuration yields 22,800 total parameters, since PyTorch's GRU gives each of the 3 gates (reset, update, candidate) its own input-to-hidden weight matrix, hidden-to-hidden weight matrix, and two independent bias vectors (b_ih and b_hh)
The count works out to 22,650 total parameters, because each gate is assumed to share a single combined bias vector rather than keeping the input bias and hidden bias separate, which undercounts the true PyTorch parameterization
Just 7,600 parameters are required, because one shared reset/update/candidate transformation is assumed to govern all gating behavior, so the weight matrices are not multiplied by the number of gates
Treating this as a 4-gate cell gives 30,400 total parameters, because the architecture is assumed to have input, forget, cell, and output gates the way an LSTM does, rather than GRU's 3 gates
Answer: A. This configuration yields 22,800 total parameters, since PyTorch's GRU gives each of the 3 gates (reset, update, candidate) its own input-to-hidden weight matrix, hidden-to-hidden weight matrix, and two independent bias vectors (b_ih and b_hh)
ExplanationPyTorch's nn.GRU allocates each of the three gates (reset, update, candidate) its own input-to-hidden weight matrix of shape (hidden_size, input_size), its own hidden-to-hidden weight matrix of shape (hidden_size, hidden_size), and two independent bias vectors, b_ih and b_hh, matching the real GRU equations where the reset and update gates each need separate learnable transforms of both x_t and h_(t-1), and the candidate state applies the reset gate to h_(t-1) before its own transform. For input_size=100, hidden_size=50, one layer: input-to-hidden weights = 3 x 50 x 100 = 15,000; hidden-to-hidden weights = 3 x 50 x 50 = 7,500; biases = 2 x 3 x 50 = 300, where the factor of 2 comes from keeping b_ih and b_hh separate rather than merging them. Total = 15,000 + 7,500 + 300 = 22,800. Merging the two bias vectors into one undercounts by 150, giving 22,650. Assuming a single shared transformation governs reset, update, and candidate together ignores that each gate needs independently learned weights, giving only 7,600. Assuming four gates conflates GRU's 3-gate design with LSTM's 4-gate design (input, forget, cell, output), inflating the count to 30,400.
Question 58 · LSTM architecture · hard
Given nn.LSTM(input_size=512, hidden_size=1024, num_layers=1, batch_first=True) with input tensor [4, 256, 512], calculate the total parameter count and analyze how the 4 gates (forget, input, cell, output) control information flow through 256 time steps?
Total parameters = 4 gates × (input_size×hidden + hidden×hidden + bias) per layer = 6,299,648 for 1 layer; the forget, input, cell, output gates use sigmoid/tanh activations to control what information passes through, because each gate learns independent weight matrices
Parameters = 512 × 1024 = 524,288 because LSTM only has one weight matrix connecting input to hidden state, ignoring the separate hidden-to-hidden weights and per-gate biases entirely
The LSTM processes all 256 time steps simultaneously in parallel because recurrent connections are computed via matrix multiplication across the sequence dimension
Parameters = 1024^2 = 1,048,576 because only hidden-to-hidden connections have trainable weights, treating the input-to-hidden projection and all gate biases as non-trainable constants
Answer: A. Total parameters = 4 gates × (input_size×hidden + hidden×hidden + bias) per layer = 6,299,648 for 1 layer; the forget, input, cell, output gates use sigmoid/tanh activations to control what information passes through, because each gate learns independent weight matrices
ExplanationStep-by-step parameter calculation: (1) LSTM has 4 gates: forget, input, cell, output. (2) Each gate has its own weight matrices W_ih (512×1024) and W_hh (1024×1024), plus biases b_ih and b_hh (1024 each). Per gate: 512×1024 + 1024×1024 + 2×1024 = 524,288 + 1,048,576 + 2,048 = 1,574,912. (3) With num_layers=1, total = 4 gates × 1,574,912 = 6,299,648 parameters. Functionally, the forget gate (sigmoid output between 0 and 1) controls how much of the previous cell state to retain, which mitigates vanishing gradients: the gradient flows through the cell state with a multiplicative factor close to 1.0 whenever the forget gate is near 1, allowing information to propagate across many of the 256 time steps.
Question 59 · GAN training dynamics · hard
In a GAN, the discriminator D outputs a probability via D(x) = sigmoid(l), where l is its logit for a given input. For the original minimax generator loss L_G = log(1 − D(G(z))), the chain rule gives dL_G/dl = −D(G(z)) (since d/dl log(1 − sigmoid(l)) = −sigmoid(l)). For the non-saturating generator loss L_G' = −log(D(G(z))), the chain rule gives dL_G'/dl = D(G(z)) − 1. Early in training, the discriminator confidently rejects a batch of generated images, so D(G(z)) = 0.02 for that batch. Given these two gradient expressions evaluated at D(G(z)) = 0.02, which statement correctly compares their magnitudes and explains the resulting training dynamics?
Computing both derivatives directly: the original loss gives |−0.02| = 0.02 gradient magnitude, while the non-saturating loss gives |0.02 − 1| = 0.98 — a roughly 49-fold increase, so the non-saturating formulation supplies far more signal for the generator precisely when the discriminator is winning decisively.
Substituting D(G(z)) = 0.02 shows the minimax loss actually produces the larger gradient magnitude of 0.98, while the non-saturating loss produces only 0.02, meaning the original formulation is preferred whenever the discriminator becomes highly confident.
Both formulations produce an identical gradient magnitude of 0.02 at this point, since the sigmoid derivative term D(1−D) cancels the logarithmic term in exactly the same way regardless of which loss function is differentiated.
Because D(G(z)) = 0.02 makes the original loss value log(1−0.02) nearly zero, that near-zero loss corresponds to a gradient magnitude of 0.98 for the original formulation and only 0.02 for the non-saturating one.
Answer: A. Computing both derivatives directly: the original loss gives |−0.02| = 0.02 gradient magnitude, while the non-saturating loss gives |0.02 − 1| = 0.98 — a roughly 49-fold increase, so the non-saturating formulation supplies far more signal for the generator precisely when the discriminator is winning decisively.
ExplanationBy the chain rule, d/dl log(1 − sigmoid(l)) = −sigmoid(l), so the original minimax generator loss has gradient dL_G/dl = −D(G(z)); at D(G(z)) = 0.02 this has magnitude 0.02. For the non-saturating loss L_G' = −log(D(G(z))), the chain rule gives d/dl[−log(sigmoid(l))] = −(1 − sigmoid(l)) = D(G(z)) − 1; at D(G(z)) = 0.02 this has magnitude |0.02 − 1| = 0.98. The ratio 0.98 / 0.02 = 49, so the non-saturating loss delivers roughly 49 times more gradient signal to the generator at this point. This is the vanishing-gradient problem identified in the original GAN paper (Goodfellow et al., 2014): when the discriminator confidently rejects generated samples (D(G(z)) → 0), the minimax generator loss saturates and gives almost no useful gradient — exactly when the generator most needs a strong learning signal. Switching to the non-saturating loss −log(D(G(z))) keeps gradients large under precisely these conditions, which is why it is the version used in practice instead of the original minimax formulation.
Question 60 · GAN training dynamics · hard
Given a DCGAN with Generator: z(64) -> ConvTranspose2d -> ... -> output(3, 32, 32) and Discriminator: input(3, 32, 32) -> Conv2d -> ... -> sigmoid, with Adam optimizer (lr_D=0.0002, lr_G=0.0002), batch_size=128, analyze the training dynamics and predict when mode collapse might occur during 100 epochs?
Mode collapse risk peaks in the first 10-20 of the 100 epochs: because lr_D and lr_G are both 0.0002, the discriminator's simpler binary real/fake decision converges faster than the generator's harder image-synthesis task, so D loss approaches 0 with D(real) -> 1.0 and D(fake) -> 0.0 early on, and G's gradient from log(1 - D(G(z))) -> log(1.0) = 0 vanishes before G's weights (updated with batch_size=128 samples per step) have moved far enough to cover diverse modes
Mode collapse never occurs with DCGAN architecture because the convolutional structure of ConvTranspose2d layers forces spatial diversity in every generated image, regardless of how skewed the discriminator's D(real) versus D(fake) outputs become during training
Mode collapse only happens when batch_size < 32, therefore batch_size=128 is safe from this failure mode
The discriminator always converges before the generator because it has a simpler objective function, so mode collapse cannot occur since a fully-converged discriminator will always reject a generator's collapsed, low-diversity outputs as obviously unrealistic
Answer: A. Mode collapse risk peaks in the first 10-20 of the 100 epochs: because lr_D and lr_G are both 0.0002, the discriminator's simpler binary real/fake decision converges faster than the generator's harder image-synthesis task, so D loss approaches 0 with D(real) -> 1.0 and D(fake) -> 0.0 early on, and G's gradient from log(1 - D(G(z))) -> log(1.0) = 0 vanishes before G's weights (updated with batch_size=128 samples per step) have moved far enough to cover diverse modes
ExplanationStep-by-step GAN analysis: (1) GAN minimax objective: min_G max_D E[log(D(x))] + E[log(1-D(G(z)))]. (2) Optimal D: D*(x) = p_data(x) / (p_data(x) + p_g(x)). (3) When D is too strong: D(G(z)) -> 0, so the gradient of log(1-D(G(z))) with respect to G's parameters shrinks toward 0/(1-0) = 0 (vanishing gradient). (4) Mode collapse follows: once G's gradient signal is this weak, G stops exploring the latent space and instead exploits the few outputs that still slip past D, mapping many z values to nearly the same image. (5) Timing: with lr_D = lr_G = 0.0002, both networks take equally sized optimizer steps, but discriminating 3x32x32 images as real/fake is a far simpler decision boundary than synthesizing them from a 64-dim noise vector, and with batch_size=128 giving D 128 real and 128 fake examples every step, D's loss typically bottoms out well before G's has caught up. This pushes the vanishing-gradient regime -- and with it the highest mode-collapse risk -- into roughly the first 10-20 of the 100 training epochs, rather than late in training once G has had time to close the gap. (6) Prevention: balance the lr_G/lr_D ratio, use label smoothing (real labels = 0.9 instead of 1.0), and add a feature matching loss.