During training with mini-batch SGD, your batch has 4 samples with individual losses [0.5, 1.2, 0.3, 0.8]. The learning rate is 0.01 and the gradient of the mean loss with respect to weight w is computed as -2.5. What is the new value of w if w_current = 3.0?
Mean loss = (0.5+1.2+0.3+0.8)/4 = 0.7. w_new = w - lr × gradient = 3.0 - 0.01 × (-2.5) = 3.0 + 0.025 = 3.025
w_new = 3.0 - 0.01 × 2.5 = 2.975; the negative gradient means we should subtract a positive value
w_new = 3.0 - 0.01 × (-2.5) × 4 = 3.1; multiply gradient by batch size to undo the averaging
w_new = 3.0 - (0.7 × 0.01) = 2.993; the update uses the loss value, not the gradient
Answer: A. Mean loss = (0.5+1.2+0.3+0.8)/4 = 0.7. w_new = w - lr × gradient = 3.0 - 0.01 × (-2.5) = 3.0 + 0.025 = 3.025
ExplanationSGD update rule: w_new = w_old - lr × dL/dw. Given w=3.0, lr=0.01, dL/dw=-2.5: w_new = 3.0 - 0.01(-2.5) = 3.0 + 0.025 = 3.025. The negative gradient means the loss decreases when w increases, so SGD pushes w upward. The mean loss (0.7) is used to compute the gradient but doesn't directly enter the update.
Question 142 · CNN Parameter Counting · hard
A Conv2D layer has 32 filters of size 3×3 applied to an input with 3 channels (RGB). With bias, how many parameters does this layer have? Calculate the contribution of each component and explain why the kernel dimensions include input channels?
Each filter: 3×3×3 = 27 weights (kernel covers all input channels). 32 filters: 32×27 = 864 weights + 32 biases = 896 total
Each filter: 3×3 = 9 weights (shared across channels). 32 filters: 32×9 = 288 weights + 32 biases = 320 total
Answer: A. Each filter: 3×3×3 = 27 weights (kernel covers all input channels). 32 filters: 32×27 = 864 weights + 32 biases = 896 total
ExplanationEach conv filter has spatial size 3×3 and must span all 3 input channels, so each filter has 3×3×3 = 27 weight parameters. With 32 filters: 32×27 = 864 weights. Each filter has 1 bias, so 32 biases. Total = 864 + 32 = 896. Common mistake: option B forgets to multiply by input channels (each filter is 3D, not 2D).
Question 143 · Early Stopping and Overfitting · hard
You train a model for 10 epochs. Training loss: [2.1, 1.5, 0.9, 0.5, 0.3, 0.2, 0.15, 0.12, 0.10, 0.09]. Validation loss: [2.0, 1.6, 1.1, 0.8, 0.75, 0.78, 0.82, 0.90, 0.95, 1.01]. At which epoch should you apply early stopping, and what phenomenon is occurring?
Stop at epoch 5 (val loss 0.75, the minimum). After epoch 5, validation loss increases while training loss keeps decreasing — this is overfitting. The growing gap between training and validation loss confirms the model is memorizing training data
Stop at epoch 10 because training loss is still decreasing, meaning the model is still learning useful patterns
Stop at epoch 1 because that is when validation loss is closest to training loss (gap = 0.1), indicating the best generalization
Stop at epoch 3 because the rate of validation loss decrease slows down after that point
Answer: A. Stop at epoch 5 (val loss 0.75, the minimum). After epoch 5, validation loss increases while training loss keeps decreasing — this is overfitting. The growing gap between training and validation loss confirms the model is memorizing training data
ExplanationValidation loss decreases until epoch 5 (0.75) then starts increasing: 0.78, 0.82, 0.90, 0.95, 1.01. Meanwhile training loss keeps dropping to 0.09. This divergence is the hallmark of overfitting — the model fits training noise rather than generalizable patterns. Early stopping saves the model at epoch 5 (lowest validation loss). Patience of 1-2 epochs is typically used in practice to avoid stopping on a temporary fluctuation before confirming the upward trend is real.
Question 144 · Dropout Regularization · hard
A hidden layer in a neural network has activations [4.0, 8.0, 6.0, 2.0]. During one training pass with inverted dropout at rate p=0.2 (keep probability 1-p=0.8), neurons 1 and 4 are dropped (set to 0). What are the correct training-time output vector and the correct inference-time output vector?
With inverted dropout applied at train time, neurons 1 and 4 are zeroed and survivors are divided by the keep probability (1-p) = 0.8, yielding a training output of [0, 10.0, 7.5, 0]; inference then uses the raw activations unchanged: [4.0, 8.0, 6.0, 2.0].
Classic (non-inverted) dropout zeroes neurons 1 and 4 without any rescaling at train time, yielding [0, 8.0, 6.0, 0]; the compensation is deferred to inference, where every activation is multiplied by the keep probability 0.8, giving [3.2, 6.4, 4.8, 1.6].
Simply zeroing neurons 1 and 4 with no rescaling anywhere gives a training output of [0, 8.0, 6.0, 0]; since dropout is switched off at test time, inference then uses the unmodified activations [4.0, 8.0, 6.0, 2.0].
Dividing the surviving activations by the dropout rate p = 0.2 itself (instead of the keep probability) after zeroing neurons 1 and 4 produces a training output of [0, 40.0, 30.0, 0], while inference again uses the raw, unmodified activations [4.0, 8.0, 6.0, 2.0].
Answer: A. With inverted dropout applied at train time, neurons 1 and 4 are zeroed and survivors are divided by the keep probability (1-p) = 0.8, yielding a training output of [0, 10.0, 7.5, 0]; inference then uses the raw activations unchanged: [4.0, 8.0, 6.0, 2.0].
ExplanationInverted dropout applies its compensation during training, not at test time. With p=0.2 the keep probability is 1-p=0.8, so after zeroing neurons 1 and 4 the surviving activations 8.0 and 6.0 are each divided by 0.8, giving 10.0 and 7.5 — a training output of [0, 10.0, 7.5, 0]. Because this rescaling already happened during training, inference needs no adjustment at all: the layer simply passes through the raw activations [4.0, 8.0, 6.0, 2.0]. The classic (non-inverted) formulation instead skips rescaling during training and multiplies by the keep probability at inference — a valid but different scheme, not what inverted dropout specifies here. Skipping rescaling altogether, or dividing by the dropout rate p instead of the keep probability (1-p), both break the expected-value guarantee that dropout relies on to keep activation magnitudes consistent between training and inference.
Question 145 · Binary Cross-Entropy Loss · hard
You compute binary cross-entropy loss for a single sample with true label y=1 and predicted probability p=0.8. Using ln(0.8) = -0.223 and ln(0.2) = -1.609, what is the loss?
ExplanationBCE = -[y·ln(p) + (1-y)·ln(1-p)]. For y=1, p=0.8: BCE = -[1×ln(0.8) + 0×ln(0.2)] = -[(-0.223) + 0] = 0.223. When y=1, only the -ln(p) term survives. If the model predicted p=1.0 perfectly, loss would be 0. As p→0, loss→∞, penalizing confident wrong predictions severely. This result occurs because each step follows deterministically from the computation rules.
Question 146 · Max Pooling · hard
A max pooling layer with kernel size 2×2 and stride 2 is applied to this 4×4 feature map:
[[1, 3, 2, 4],
[5, 6, 7, 8],
[3, 2, 1, 0],
[1, 4, 3, 2]]
What is the output? Analyze the computation step by step and determine the exact numerical answer?
[[3.75, 5.25], [2.5, 1.5]] — this is average pooling, not max pooling
[[1, 0], [1, 0]] — this is min pooling, taking the smallest value in each block
[[6, 8, 4, 3]] — the output is a single row because max pooling flattens the feature map
Answer: A. [[6, 8], [4, 3]] — each 2×2 block: max(1,3,5,6)=6, max(2,4,7,8)=8, max(3,2,1,4)=4, max(1,0,3,2)=3
Explanation2×2 max pooling with stride 2 divides the 4×4 input into four non-overlapping 2×2 blocks. Top-left [1,3,5,6]→max=6. Top-right [2,4,7,8]→max=8. Bottom-left [3,2,1,4]→max=4. Bottom-right [1,0,3,2]→max=3. Output: [[6,8],[4,3]], a 2×2 feature map. Spatial dimensions halve: 4×4→2×2.
Question 147 · Batch Normalization · hard
Batch normalization normalizes activations: x_norm = (x - mean_batch) / sqrt(var_batch + epsilon). For a mini-batch with values [2, 4, 6, 8], what are the normalized values? (Use epsilon = 0). What value is returned?
mean = 5, var = 5. x_norm = [(2-5)/sqrt(5), (4-5)/sqrt(5), (6-5)/sqrt(5), (8-5)/sqrt(5)] = [-1.342, -0.447, 0.447, 1.342]
mean = 5, var = 20. x_norm = [-3/sqrt(20), -1/sqrt(20), 1/sqrt(20), 3/sqrt(20)] = [-0.671, -0.224, 0.224, 0.671]; using population variance N instead of sample variance
x_norm = [2/8, 4/8, 6/8, 8/8] = [0.25, 0.5, 0.75, 1.0]; normalization divides by the max value
x_norm = [0, 0, 0, 0]; all values are replaced by zero in batch normalization
Answer: A. mean = 5, var = 5. x_norm = [(2-5)/sqrt(5), (4-5)/sqrt(5), (6-5)/sqrt(5), (8-5)/sqrt(5)] = [-1.342, -0.447, 0.447, 1.342]
ExplanationBatch of [2,4,6,8]. mean = (2+4+6+8)/4 = 5. Variance = [(2-5)^2 + (4-5)^2 + (6-5)^2 + (8-5)^2]/4 = [9+1+1+9]/4 = 20/4 = 5. Note: BN uses population variance (divide by N, not N-1). sqrt(5) ≈ 2.236. x_norm: (2-5)/2.236 = -1.342, (4-5)/2.236 = -0.447, (6-5)/2.236 = 0.447, (8-5)/2.236 = 1.342. The resulting values are symmetric around 0 and have unit variance, which is exactly the effect batch normalization is designed to produce.
Question 148 · Forward Pass Computation · hard
You have a 2-layer neural network: input(4) → hidden(3, ReLU) → output(2, softmax). The hidden layer weights are W1 = [[1,0,-1],[0,1,0],[-1,0,1],[0,-1,0]], bias b1 = [0,0,0]. For input x = [1, 2, -1, 1], what is the hidden layer output after ReLU?
Question 149 · Xavier/Glorot Initialization · hard
You have a neural network layer with input size 512 and output size 256. You apply weight initialization using Xavier/Glorot uniform: weights sampled from U(-limit, limit) where limit = sqrt(6 / (fan_in + fan_out)). What is the initialization range for this layer?
limit = sqrt(6 / (512 + 256)) = sqrt(6/768) = sqrt(0.00781) = 0.0884. Weights are initialized in [-0.0884, 0.0884]. This keeps the variance of activations roughly equal across layers to prevent vanishing or exploding signals
limit = sqrt(6 / 512) = 0.108 because only fan_in matters; the output dimension does not affect initialization
limit = sqrt(2 / 512) = 0.0625 because Xavier uses 2/fan_in, not 6/(fan_in+fan_out)
limit = 1/sqrt(512) = 0.0442 because the standard normal scaled by 1/sqrt(n) is used
Answer: A. limit = sqrt(6 / (512 + 256)) = sqrt(6/768) = sqrt(0.00781) = 0.0884. Weights are initialized in [-0.0884, 0.0884]. This keeps the variance of activations roughly equal across layers to prevent vanishing or exploding signals
ExplanationXavier uniform initialization uses limit = sqrt(6/(fan_in + fan_out)). fan_in=512, fan_out=256. limit = sqrt(6/768) = sqrt(0.007813) = 0.08839. Weights are drawn from U(-0.0884, 0.0884). The 6 comes from the variance of a uniform distribution: Var(U(-a,a)) = a²/3, and we want Var = 2/(fan_in+fan_out), so a² = 6/(fan_in+fan_out). This produces stable gradient flow because the variance of outputs roughly matches the variance of inputs.
Question 150 · Sigmoid Backpropagation · hard
During backpropagation through a sigmoid layer, the forward pass output is sigma = 0.8. What is the local gradient d(sigma)/d(z), and if the upstream gradient is delta = 0.5, what gradient flows backward?
Sigmoid derivative: sigma(1-sigma) = 0.8 × 0.2 = 0.16. Backward gradient = upstream × local = 0.5 × 0.16 = 0.08. The sigmoid derivative is maximized at sigma=0.5 (0.25) and shrinks toward 0 at extremes, which causes vanishing gradients
Derivative = 1/sigma = 1/0.8 = 1.25. Backward = 0.5 × 1.25 = 0.625; the gradient of sigmoid is its reciprocal
Derivative = sigma² = 0.64. Backward = 0.5 × 0.64 = 0.32; the squared output is the local gradient
Derivative = 1.0 always. Backward = 0.5; sigmoid has gradient 1 everywhere like a linear function
Answer: A. Sigmoid derivative: sigma(1-sigma) = 0.8 × 0.2 = 0.16. Backward gradient = upstream × local = 0.5 × 0.16 = 0.08. The sigmoid derivative is maximized at sigma=0.5 (0.25) and shrinks toward 0 at extremes, which causes vanishing gradients
ExplanationThe sigmoid function sigma(z) has a clean derivative: d(sigma)/dz = sigma(1-sigma). Given sigma=0.8: local gradient = 0.8(1-0.8) = 0.8×0.2 = 0.16. By the chain rule, the gradient flowing backward = upstream_gradient × local_gradient = 0.5 × 0.16 = 0.08. Note how the gradient shrinks: from 0.5 upstream to 0.08 downstream. This happens because sigma=0.8 is in the saturated region. At sigma=0.5, the gradient would be 0.5×0.25=0.125 — the maximum possible through sigmoid.
Question 151 · 1D Convolution · hard
A 1D convolution with kernel [1, -2, 1] is applied to signal [3, 1, 4, 1, 5, 9] with stride 1 and no padding. What is the output, and what operation does this kernel approximate?
Output length = 6-3+1 = 4. Convolutions: 3×1+1×(-2)+4×1=5, 1×1+4×(-2)+1×1=-6, 4×1+1×(-2)+5×1=7, 1×1+5×(-2)+9×1=0. Output: [5, -6, 7, 0]. This kernel approximates the second derivative (discrete Laplacian) because it computes f(i-1) - 2f(i) + f(i+1)
Output: [3, -2, 4, -2, 5, -18] — multiply each element by the corresponding kernel weight
Output: [5, -6, 7, 0, 0, 0] — zero-pad the output to match input length
Output: [0, -6, 7, 5] — convolution reverses the kernel before applying, changing the output order
Answer: A. Output length = 6-3+1 = 4. Convolutions: 3×1+1×(-2)+4×1=5, 1×1+4×(-2)+1×1=-6, 4×1+1×(-2)+5×1=7, 1×1+5×(-2)+9×1=0. Output: [5, -6, 7, 0]. This kernel approximates the second derivative (discrete Laplacian) because it computes f(i-1) - 2f(i) + f(i+1)
ExplanationKernel [1,-2,1] slides across the input with stride 1. Position 0: 3(1)+1(-2)+4(1) = 3-2+4 = 5. Position 1: 1(1)+4(-2)+1(1) = 1-8+1 = -6. Position 2: 4(1)+1(-2)+5(1) = 4-2+5 = 7. Position 3: 1(1)+5(-2)+9(1) = 1-10+9 = 0. Output: [5,-6,7,0]. The kernel [1,-2,1] computes f[i-1]-2f[i]+f[i+1], which is the discrete second derivative. Positive values indicate the signal curves upward (concave up), negative values indicate concavity downward.
Question 152 · Learning Rate Warmup · hard
You train with learning rate warmup: starting from lr=0 and linearly increasing to lr=0.001 over 1000 steps, then constant. What is the learning rate at step 250, step 500, and step 1500?
Step 250: lr = 0.001 × (250/1000) = 0.00025. Step 500: lr = 0.001 × (500/1000) = 0.0005. Step 1500: lr = 0.001 (constant after warmup). Warmup prevents large early updates when the model weights are still random and gradients are noisy
Step 250: lr = 0.001/250 = 0.000004. Step 500: lr = 0.001/500 = 0.000002. Step 1500: lr = 0.001/1500; the learning rate always decays
Step 250: lr = 0.001. Step 500: lr = 0.001. Step 1500: lr = 0.001; warmup means the learning rate stays constant from the start
Step 250: lr = 0.00025. Step 500: lr = 0.0005. Step 1500: lr = 0.0 because the learning rate drops to zero after warmup
Answer: A. Step 250: lr = 0.001 × (250/1000) = 0.00025. Step 500: lr = 0.001 × (500/1000) = 0.0005. Step 1500: lr = 0.001 (constant after warmup). Warmup prevents large early updates when the model weights are still random and gradients are noisy
ExplanationLinear warmup from 0 to target_lr over warmup_steps: lr(t) = target_lr × min(t/warmup_steps, 1). Step 250: 0.001 × (250/1000) = 0.001 × 0.25 = 0.00025. Step 500: 0.001 × (500/1000) = 0.0005. Step 1000: 0.001 × 1.0 = 0.001 (warmup complete). Step 1500: 0.001 (constant, since 1500/1000 > 1, clamped to 1). Warmup is critical for transformers because initial random weights produce large, unstable gradients that can derail training.
Question 153 · CNN Architecture Analysis · hard
A neural network has layers: Conv(3→16, 3×3) → Conv(16→32, 3×3) → FC(32×6×6→128) → FC(128→10). Starting from a 3-channel 8×8 input with no padding and stride 1, what is the output size after each conv layer, and how many total parameters does the FC part have?
Conv1 output: 6×6×16 (8-3+1=6). Conv2 output: 4×4×32 (6-3+1=4). But FC input is 32×4×4=512, not 32×6×6. If the architecture says FC(32×6×6→128), there is a mismatch. Assuming it should be FC(512→128): 512×128+128=65,664. FC(128→10): 128×10+10=1,290. Total FC params: 66,954
Conv1: 8×8×16 (same padding). Conv2: 8×8×32. FC1: 8×8×32×128 = 262,144 + 128. FC2: 128×10+10. Total FC params: 263,562
Conv1: 6×6×16. Conv2: 4×4×32. FC1: 32×6×6×128 = 147,456 + 128 = 147,584. FC2: 1,290. Total: 148,874; using the stated dimensions without checking spatial size
Conv outputs don't reduce spatial size because convolution preserves dimensions by default
Answer: A. Conv1 output: 6×6×16 (8-3+1=6). Conv2 output: 4×4×32 (6-3+1=4). But FC input is 32×4×4=512, not 32×6×6. If the architecture says FC(32×6×6→128), there is a mismatch. Assuming it should be FC(512→128): 512×128+128=65,664. FC(128→10): 128×10+10=1,290. Total FC params: 66,954
ExplanationWithout padding, output_size = input_size - kernel_size + 1. Conv1: 8-3+1 = 6, so output is 6×6×16. Conv2: 6-3+1 = 4, so output is 4×4×32. Flattened: 32×4×4 = 512 neurons. The stated FC(32×6×6→128) is architecturally inconsistent — the actual flattened size is 512, not 1152. Assuming correction to FC(512→128): params = 512×128 + 128 = 65,664. FC(128→10): 128×10 + 10 = 1,290. Total FC: 66,954. This demonstrates why checking spatial dimensions through the network is critical.
Question 154 · Data Augmentation · hard
You apply data augmentation during training: random horizontal flip (p=0.5), random rotation (±15°), and color jitter (brightness ±0.2). If your original dataset has 1000 images and you train for 10 epochs, how many unique augmented images does the model see, and why does this help?
The model sees 10,000 images total (1000 × 10 epochs), but each pass through the same image produces a different augmentation. So approximately 10,000 unique augmented versions are generated because the random transforms differ each time. This acts as implicit regularization, reducing overfitting by making the model robust to variations
The model sees exactly 1000 images because augmentation just preprocesses the dataset once before training begins
The model sees 1000 × 2^3 = 8000 images because each augmentation doubles the dataset size
The model sees 10 × 3 × 1000 = 30,000 images because each augmentation type creates a separate copy per epoch
Answer: A. The model sees 10,000 images total (1000 × 10 epochs), but each pass through the same image produces a different augmentation. So approximately 10,000 unique augmented versions are generated because the random transforms differ each time. This acts as implicit regularization, reducing overfitting by making the model robust to variations
ExplanationOnline augmentation applies random transforms each time an image is loaded during training. With 1000 images × 10 epochs = 10,000 forward passes. Each pass has different random parameters: different flip decision, different rotation angle, different brightness shift. So the model effectively sees ~10,000 unique versions (virtually no two identical because continuous random parameters ensure uniqueness). This acts as regularization because the model cannot memorize specific pixel patterns, producing better generalization to unseen data.
Question 155 · GRU Update Gate · hard
In a GRU (Gated Recurrent Unit), the update gate z_t = sigmoid(W_z × [h_{t-1}, x_t]). If z_t = 0.9 for a particular timestep, what does this mean for the hidden state update h_t = z_t × h_{t-1} + (1 - z_t) × h_candidate?
h_t = 0.9 × h_{t-1} + 0.1 × h_candidate. The update gate at 0.9 means the cell retains 90% of its previous hidden state and incorporates only 10% new information. This allows the GRU to preserve long-term memory by keeping z_t close to 1, effectively creating a highway for gradient flow
h_t = h_candidate entirely because z_t=0.9 rounds to 1 which selects the candidate state
h_t = 0.9 × h_candidate + 0.1 × h_{t-1}; the gate controls how much NEW information to accept (90%), not how much to remember
z_t=0.9 means 90% of neurons are dropped, similar to dropout regularization
Answer: A. h_t = 0.9 × h_{t-1} + 0.1 × h_candidate. The update gate at 0.9 means the cell retains 90% of its previous hidden state and incorporates only 10% new information. This allows the GRU to preserve long-term memory by keeping z_t close to 1, effectively creating a highway for gradient flow
ExplanationThe GRU update equation h_t = z_t × h_{t-1} + (1-z_t) × h_candidate is a linear interpolation between old state and candidate. With z_t=0.9: h_t = 0.9×h_{t-1} + 0.1×h_candidate. This means 90% old memory retained, 10% new information incorporated. The update gate learns when to remember (z_t→1, carry forward) vs when to update (z_t→0, accept new info). For sequences where a past event matters for future predictions, the GRU can hold z_t near 1 for many timesteps, effectively creating a skip connection through time.
Question 156 · Transposed Convolution · hard
You compute the output of a transposed convolution (deconvolution) with kernel size 3×3, stride 2, and no padding on a 2×2 input feature map filled with value 1. What is the output spatial size and how is it computed?
Output size = stride × (input - 1) + kernel = 2×(2-1) + 3 = 5. The output is 5×5. Transposed convolution upsamples by inserting stride-1=1 zeros between input values, then applying a standard convolution. Each input pixel's kernel placement overlaps, creating a checkerboard pattern before the convolution smooths it
Output size = input × stride = 2×2 = 4, giving a 4×4 output because transposed convolution simply scales up by stride
Output size = input + kernel - 1 = 2+3-1 = 4, giving a 4×4 output identical to standard convolution output formula
Output size = input × kernel = 2×3 = 6, giving a 6×6 output because each input pixel spawns a full kernel
Answer: A. Output size = stride × (input - 1) + kernel = 2×(2-1) + 3 = 5. The output is 5×5. Transposed convolution upsamples by inserting stride-1=1 zeros between input values, then applying a standard convolution. Each input pixel's kernel placement overlaps, creating a checkerboard pattern before the convolution smooths it
ExplanationTransposed convolution output size formula: o = s(i-1) + k - 2p where s=stride, i=input, k=kernel, p=padding. With s=2, i=2, k=3, p=0: o = 2(2-1) + 3 - 0 = 2 + 3 = 5. So the output is 5×5. Mechanically, it works by: (1) inserting stride-1=1 zero between each input value, making a 3×3 padded input, (2) applying a standard convolution with the kernel. Transposed convolutions are used in decoder networks, GANs, and segmentation models for learned upsampling because they produce spatial resolution increases.
Question 157 · Gradient Accumulation · hard
You implement gradient accumulation over 4 mini-batches of size 8 before one optimizer step. How does this compare to training with a single batch of size 32, and what is the effective batch size? Analyze the computation step by step and determine the exact answer?
Effective batch size = 4 × 8 = 32. The gradients from 4 batches of 8 are summed (or averaged), then one optimizer step is taken. This is mathematically equivalent to a single batch of 32 (assuming the loss is averaged per batch), but uses only 8 samples of GPU memory at a time. The tradeoff is 4× more forward/backward passes for the same update
Effective batch size = 8 because only the last batch's gradients are used; the first 3 are overwritten
Effective batch size = 4 because we average 4 gradient vectors, reducing the batch dimension
Gradient accumulation gives different results from batch size 32 because the running mean of gradients introduces bias
Answer: A. Effective batch size = 4 × 8 = 32. The gradients from 4 batches of 8 are summed (or averaged), then one optimizer step is taken. This is mathematically equivalent to a single batch of 32 (assuming the loss is averaged per batch), but uses only 8 samples of GPU memory at a time. The tradeoff is 4× more forward/backward passes for the same update
ExplanationGradient accumulation sums gradients across multiple forward-backward passes before updating weights. With 4 accumulation steps of batch 8: effective batch = 4×8 = 32. Each mini-batch computes loss on 8 samples and backprops, adding gradients to a running sum. After 4 steps, divide by 4 (or scale loss by 1/4), then optimizer.step(). This gives the same gradient as processing all 32 samples at once because gradient of a sum equals sum of gradients. The key benefit: you get large-batch training dynamics while fitting in GPU memory limited to 8 samples.
Question 158 · Pre-Activation Residual Block · hard
In a residual block with pre-activation (BatchNorm → ReLU → Conv), the input x has values at a specific position: [1.5, -0.3, 2.1]. After BatchNorm the values become [0.8, -1.2, 1.4]. What are the values after ReLU, and what is the final output of the residual block if the Conv produces [0.1, -0.1, 0.3]?
After ReLU: [0.8, 0.0, 1.4] — ReLU clips -1.2 to 0. Conv produces [0.1, -0.1, 0.3]. Residual output = x + Conv(ReLU(BN(x))) = [1.5+0.1, -0.3+(-0.1), 2.1+0.3] = [1.6, -0.4, 2.4]. The skip connection adds the ORIGINAL input, bypassing BN and ReLU
After ReLU: [0.8, -1.2, 1.4] — ReLU is not applied in pre-activation blocks. Output = [1.6, -1.5, 2.4]
After ReLU: [0.8, 0.0, 1.4]. Output = [0.1, -0.1, 0.3] — no skip connection in pre-activation ResNet
After ReLU: [0.8, 0.0, 1.4]. Output = [0.8+0.1, 0.0+(-0.1), 1.4+0.3] = [0.9, -0.1, 1.7]; skip connection adds BN-ReLU output, not original input
Answer: A. After ReLU: [0.8, 0.0, 1.4] — ReLU clips -1.2 to 0. Conv produces [0.1, -0.1, 0.3]. Residual output = x + Conv(ReLU(BN(x))) = [1.5+0.1, -0.3+(-0.1), 2.1+0.3] = [1.6, -0.4, 2.4]. The skip connection adds the ORIGINAL input, bypassing BN and ReLU
ExplanationPre-activation order: BN → ReLU → Conv, then add skip. BN transforms [1.5, -0.3, 2.1] → [0.8, -1.2, 1.4]. ReLU: max(0, 0.8)=0.8, max(0, -1.2)=0.0, max(0, 1.4)=1.4 → [0.8, 0.0, 1.4]. Conv maps this to [0.1, -0.1, 0.3]. Skip connection adds ORIGINAL input x: output = [1.5+0.1, -0.3-0.1, 2.1+0.3] = [1.6, -0.4, 2.4]. The skip bypasses all transformations, which is what makes residual connections powerful — gradients flow directly through the identity path.
Question 159 · Weight Decay in SGD · hard
You train a network with weight decay lambda=0.001 and SGD. The gradient of the data loss with respect to weight w=2.5 is dL/dw=-0.8. Learning rate is 0.01. What is the total gradient (including weight decay) and the updated weight?
Weight decay gradient = lambda × w = 0.001 × 2.5 = 0.0025. Total gradient = -0.8 + 0.0025 = -0.7975. w_new = 2.5 - 0.01 × (-0.7975) = 2.5 + 0.007975 = 2.507975. Weight decay pulls weights toward zero, but here the data gradient dominates, pushing w higher; this works because weight decay adds the penalty directly to the weight update step
Total gradient = -0.8 × 0.001 = -0.0008. w_new = 2.5 + 0.0008 = 2.5008; weight decay multiplies the gradient
w_new = 2.5 - 0.01 × (-0.8) - 0.001 × 2.5 = 2.508 - 0.0025 = 2.5055; weight decay is subtracted separately from the update
w_new = (1 - 0.001) × 2.5 - 0.01 × (-0.8) = 2.4975 + 0.008 = 2.5055; weight decay is applied as multiplicative factor first
Answer: A. Weight decay gradient = lambda × w = 0.001 × 2.5 = 0.0025. Total gradient = -0.8 + 0.0025 = -0.7975. w_new = 2.5 - 0.01 × (-0.7975) = 2.5 + 0.007975 = 2.507975. Weight decay pulls weights toward zero, but here the data gradient dominates, pushing w higher; this works because weight decay adds the penalty directly to the weight update step
ExplanationL2 regularization adds (lambda/2)||w||² to the loss, giving gradient = dL_data/dw + lambda×w. Data gradient: -0.8. Regularization gradient: 0.001 × 2.5 = 0.0025. Total: -0.8 + 0.0025 = -0.7975. Update: w = 2.5 - 0.01×(-0.7975) = 2.5 + 0.007975 = 2.507975. Note: option D shows decoupled weight decay (AdamW style) where decay is applied multiplicatively: w *= (1-lr×lambda), then gradient step. Standard L2 regularization (this question) adds the decay to the gradient instead.
Question 160 · Depthwise Separable Convolution · hard
A depthwise separable convolution processes a 32×32×64 input with 128 output channels using 3×3 kernels. Compare the parameter counts of standard convolution vs depthwise separable convolution (depthwise + 1×1 pointwise). What is the parameter reduction factor?
Standard: 3×3×64×128 = 73,728. Depthwise: 3×3×64 = 576 (one 3×3 kernel per input channel). Pointwise: 1×1×64×128 = 8,192. Total separable: 576+8,192 = 8,768. Ratio: 73,728/8,768 = 8.4×. Depthwise separable uses 8.4 times fewer parameters with similar representational power; this occurs because depthwise separable convolution splits into per-channel and pointwise operations
Standard: 3×3×128 = 1,152. Depthwise separable: 3×3×64 = 576. Standard uses exactly 2× more parameters
Both have identical parameter counts because separable convolution is just a different computation order
Standard: 73,728. Separable: 73,728 + 8,192 = 81,920; separable has MORE parameters because it adds a pointwise layer
Answer: A. Standard: 3×3×64×128 = 73,728. Depthwise: 3×3×64 = 576 (one 3×3 kernel per input channel). Pointwise: 1×1×64×128 = 8,192. Total separable: 576+8,192 = 8,768. Ratio: 73,728/8,768 = 8.4×. Depthwise separable uses 8.4 times fewer parameters with similar representational power; this occurs because depthwise separable convolution splits into per-channel and pointwise operations
ExplanationStandard conv: each of 128 filters has size 3×3×64 = 576 weights. Total: 128×576 = 73,728. Depthwise conv: 64 separate 3×3 filters (one per channel), total = 64×9 = 576. Pointwise (1×1) conv: 64→128 channels, total = 64×128 = 8,192. Separable total: 576+8,192 = 8,768. Reduction factor: 73,728/8,768 ≈ 8.4×. The general formula for reduction is approximately 1/C_out + 1/k², which for k=3 and C_out=128 gives 1/128+1/9 ≈ 0.119. MobileNet achieves near-identical accuracy with this 8× parameter reduction. This reduction occurs because the depthwise step processes each channel independently, and the pointwise step mixes channels with 1x1 convolutions.