A common screening exercise for incoming AI research interns works like this: hand them a published paper and a laptop, and ask them to reproduce Table 2. Not summarize it, not critique it: reproduce it, in code, with numbers that land within the paper's reported margin. Almost everyone who has read the paper carefully still fails this test on the first attempt. The reason is not carelessness. It is that a paper's Methods section and an executable training script are two different artifacts, and the translation between them loses information that the authors either omitted as "obvious" or genuinely did not think to record.
This chapter assumes you already know how to extract an algorithm from a paper (that discipline, including the pass-by-pass reading strategy, is covered elsewhere in this course). Here the question is narrower and more mechanical: once you have the algorithm box in front of you, what does it take to turn it into code that actually produces the claimed result, and how do you know, with evidence rather than hope, that your version is faithful? We will work through one real, citable case end to end: the Lottery Ticket Hypothesis of Jonathan Frankle and Michael Carbin (ICLR 2019), a paper whose central algorithm is short enough to implement in an afternoon and whose reproduction history is well documented enough to teach real lessons about where translations go wrong.
Reproducibility is not one thing
Computing science bodies (the ACM's artifact review terminology is the most widely cited version) distinguish two properties that get flattened into the single word "reproducible" in casual conversation. Reproducibility means: given the same code, the same data, and the same environment, a different team gets the same result. Replicability means: given only the paper's description, a different team writes independent code and still reaches the same conclusion. A paper can be reproducible (its GitHub repo runs and prints the reported number) while being poorly replicable (nobody who did not copy that repo can get the same number from the paper's text alone). For a student without access to the authors' original codebase, replicability is almost always the harder and more instructive target, because it forces you to locate every decision the paper's prose left implicit.
Those implicit decisions cluster into a few recurring categories: hyperparameters mentioned once in a caption and never again (learning rate schedule, weight decay, batch size); preprocessing steps assumed to be "standard" for a dataset but not pinned to a specific version; random seeds and how many of them were averaged before reporting a number; and software versions, since a convolution or a pruning threshold computed by one BLAS library or one PyTorch minor version can differ in its last few bits from another. None of this is a defect in the paper. It is simply that natural-language description and executable specification are not the same format, and something is always lost crossing from one to the other. Your job in a reproduction is to find what was lost and put it back with a justified, stated choice, not a silent guess.
The algorithm: iterative magnitude pruning
Frankle and Carbin's paper makes a specific, falsifiable claim: inside a randomly initialized dense neural network there exists a much smaller subnetwork (a "winning ticket") that, when trained in isolation starting from that same original initialization, matches the accuracy of the full dense network in a comparable number of training iterations. The procedure they use to find such a subnetwork is called iterative magnitude pruning (IMP), and its algorithm box is genuinely short:
- Randomly initialize a network with weights θ₀.
- Train the network to convergence, arriving at weights θ.
- Prune a fraction p of the weights with the smallest magnitude, layer by layer, producing a binary mask m.
- Reset the surviving weights, the ones the mask kept, to their original values in θ₀ (not to their trained values in θ).
- Repeat from step 2 on the smaller masked network, for n rounds.
Read as prose, this looks unambiguous. It is not. Step 3 says "prune by magnitude" but does not say whether the threshold is computed globally across the whole network or independently per layer (the paper uses per-layer). It does not say whether biases and batch-norm parameters are included in the pruning population (they are typically excluded). It does not specify what happens when the target fraction does not divide evenly into an integer number of weights. None of these gaps are visible until you sit down to write the function that actually decides which weights to zero out, which is exactly why "I understood the algorithm" and "I can reproduce the algorithm" are different claims.
Writing the harness
Below is a small, fully self-contained implementation of the IMP loop on a toy network, small enough that every number in its output can be traced by hand rather than taken on faith. It deliberately makes each of the ambiguous choices above explicit rather than hiding them inside a library call.
import torch
import torch.nn as nn
import copy
torch.manual_seed(42)
class TinyMLP(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(4, 8)
self.fc2 = nn.Linear(8, 2)
def forward(self, x):
return self.fc2(torch.relu(self.fc1(x)))
model = TinyMLP()
theta0 = copy.deepcopy(model.state_dict()) # frozen initial weights, theta_0
# Synthetic dataset: 8 points, binary labels
X = torch.randn(8, 4)
y = torch.randint(0, 2, (8,))
# Mask only the weight matrices, not biases -- an explicit, stated choice
masks = {name: torch.ones_like(p)
for name, p in model.named_parameters() if "weight" in name}
def apply_masks(model, masks):
with torch.no_grad():
for name, p in model.named_parameters():
if name in masks:
p.mul_(masks[name])
def train(model, epochs=50, lr=0.1):
opt = torch.optim.SGD(model.parameters(), lr=lr)
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
opt.zero_grad()
loss = loss_fn(model(X), y)
loss.backward()
opt.step()
apply_masks(model, masks) # keep pruned weights fixed at zero
def prune_round(model, masks, prune_fraction=0.2):
with torch.no_grad():
for name, p in model.named_parameters():
if name in masks:
mask = masks[name]
alive = p[mask.bool()].abs()
k = int(prune_fraction * alive.numel())
if k == 0:
continue
threshold = torch.topk(alive, k, largest=False).values.max()
masks[name] = mask * (p.abs() > threshold).float()
def reset_to_theta0(model, theta0, masks):
model.load_state_dict(copy.deepcopy(theta0))
apply_masks(model, masks)
A few of these lines are the actual translation decisions the paper leaves as prose. apply_masks is called after every optimizer step rather than by zeroing gradients before the step, which is a simplification: production implementations usually zero the gradient of pruned weights too, so that the optimizer's momentum buffer for a dead weight does not keep nudging it away from zero between mask applications. Because this harness re-zeros every single step, the discrepancy never has a chance to accumulate, so the simplification is harmless here, but it is exactly the kind of detail that would matter more in a larger model with heavier momentum, and it is the kind of thing a paper's Methods section never mentions either way.
Running the loop and printing the surviving weight fraction before each round:
for round_idx in range(5):
total = sum(m.numel() for m in masks.values())
remaining = sum(m.sum().item() for m in masks.values())
print(f"Round {round_idx}: remaining = {remaining/total:.4f}")
train(model)
prune_round(model, masks, prune_fraction=0.2)
reset_to_theta0(model, theta0, masks)
Trace this by hand before running it, the way any reproduction should be checked. fc1.weight has 8×4 = 32 entries and fc2.weight has 2×8 = 16 entries, for 48 prunable weights total. At round 0 nothing has been pruned yet, so remaining = 48/48 = 1.0000. The first prune_round call then removes int(0.2×32)=6 weights from fc1 and int(0.2×16)=3 from fc2, leaving 26+13=39 alive, so round 1 prints 39/48 = 0.8125. Repeating this arithmetic, each round's alive count feeding the next round's 20% cut: round 2 leaves 21+11=32 alive (0.6667), round 3 leaves 17+9=26 alive (0.5417), round 4 leaves 14+8=22, tracing the per-layer counts precisely (32→26→21→17→14 for fc1, 16→13→11→9→8 for fc2) gives round 4 a printed value of 22/48 = 0.4583. Notice something important: this entire sequence, 1.0000, 0.8125, 0.6667, 0.5417, 0.4583, depends only on counting integers. It does not depend on the actual trained weight values at all, so it is exactly reproducible on any machine, any PyTorch version, any GPU, even though the loss value printed alongside it in a fuller version of this script would not be, since floating-point summation order differs across BLAS libraries and hardware. That split, between what a reproduction can guarantee bit-for-bit and what it cannot, is itself one of the most useful things to identify before you start comparing your numbers to a paper's table.
The misconception: pruning is not resetting
Most students who have heard of "pruning neural networks" before encountering this paper have in mind an older and different technique: Song Han, Jeff Pool, John Tran, and William Dally's "Learning both Weights and Connections for Efficient Neural Networks" (NIPS 2015). That method also prunes small-magnitude weights iteratively, but after each pruning step it fine-tunes the surviving weights starting from their current trained values, the goal being a smaller network for faster inference, with no claim about what would happen if you trained that sparse structure from scratch.
The step that makes Frankle and Carbin's result a different scientific claim, and the step students most often skip when reimplementing from memory, is line 4 of the algorithm box: resetting surviving weights to θ₀, their value before any training happened, rather than continuing from their already-trained value. If you delete that reset (i.e., you fine-tune the pruned network instead of rewinding it), you have correctly implemented Han et al.'s 2015 method and have not tested the lottery ticket claim at all, no matter how faithfully you copied the rest of the loop. In the code above, that distinction lives entirely in reset_to_theta0, and it is worth staring at that one function until it's clear why deleting the model.load_state_dict(copy.deepcopy(theta0)) line and instead simply continuing training would silently turn this into a different, older experiment.
The full loop, traced against its output
Making the comparison honest: engineering discipline
A toy MLP is small enough to trace by hand, but the same loop applied to a real convolutional network on CIFAR-10, which is the scale Frankle and Carbin actually study, cannot be hand-verified. At that scale the reproduction question shifts from "did I implement the algorithm correctly" to "how do I know my run and the paper's run are comparable at all," and that requires infrastructure, not just correct code.
Start with determinism. Even with torch.manual_seed fixed, a training run is not fully deterministic by default: cuDNN selects convolution algorithms based on a runtime autotuning pass that can vary between hardware, and floating-point summation order in parallel reductions is not associative, so the same seed on two different GPUs can diverge after enough steps. Calling torch.use_deterministic_algorithms(True) and setting torch.backends.cudnn.deterministic = True removes most of this, at some cost to training speed, and should be treated as mandatory for a reproduction run, not optional. Seeding needs to cover every generator actually in use: Python's random, NumPy's, PyTorch's CPU generator, and PyTorch's CUDA generator are four separate streams, and a data loader with multiple worker processes needs its own seeding scheme or shuffling order becomes another silent source of divergence.
Next, pin the environment. A requirements.txt with exact version pins (not torch>=2.0 but torch==2.3.1), recorded alongside the CUDA and driver version, turns "it doesn't match" from a mystery into a diffable fact. Log every hyperparameter as structured configuration (a YAML or JSON file saved alongside the run's checkpoints), not as command-line flags that vanish from history the moment the terminal closes. NeurIPS has required submitting authors to complete a reproducibility checklist since 2019, an initiative associated with Joelle Pineau's push for empirical rigor in machine learning research, and the checklist's central demand is exactly this: state the hyperparameters, state the number of seeds averaged, state the compute used, in a form a stranger can act on without emailing you.
Finally, decide in advance what counts as a successful reproduction, before you look at your result. A useful compare-to-paper harness never asserts exact equality of floats; it asserts that your metric falls inside a tolerance band, ideally derived from the paper's own reported variance across seeds (if they report a standard deviation across three or five runs, your single run should land within a few of those standard deviations of the mean), or, failing that, a stated band such as one percentage point of accuracy. Writing that assertion down before running the experiment, rather than eyeballing the number afterward and deciding it "basically matches," is the difference between a reproduction and a rationalization.
Active recall
Attempt each question before reading its answer.
1. A classmate says "I reproduced the paper: I downloaded the authors' GitHub repo, ran it, and got their exact number." Is this reproducibility or replicability, in the ACM sense used above, and what would it take to upgrade it to the other one?
2. In the IMP algorithm box, which single step distinguishes a "winning ticket" search from ordinary magnitude pruning with fine-tuning (Han et al., 2015), and what would you see in a training curve if a reimplementation accidentally omitted it?
3. Using the same 48-parameter network (32 weights in fc1, 16 in fc2), suppose prune_fraction is changed from 0.2 to 0.1. Trace, round by round, how many prune rounds it takes before the remaining fraction first drops below 50%, and compare that to how many rounds it took at 0.2. Is the answer exactly double? Why or why not?
4. Two people run the exact same seeded PyTorch script on different GPUs and get losses that agree to three decimal places for the first few steps, then start to diverge. Name two concrete causes, and describe what you would put in requirements.txt and your training script to close the gap as much as possible.
5. A 2020 follow-up paper (Frankle, Dziugaite, Roy, and Carbin, "Linear Mode Connectivity and the Lottery Ticket Hypothesis," ICML 2020) found that literally rewinding to iteration 0 stops working once you move to deeper networks trained with larger learning rates. What did they rewind to instead, and what does this imply for someone trying to reproduce the original paper's qualitative claim on a much larger model than the paper itself used?
6. In prune_round, why is threshold computed separately for fc1.weight and fc2.weight rather than once across both tensors concatenated together, and what would change in the round-by-round remaining-fraction trace if you switched to a single global threshold?
Answer 1. Running someone else's code on their own data is reproducibility, not replicability: it verifies that the artifact is not broken, but says nothing about whether the paper's description alone (without that specific codebase) is enough to recover the result. To upgrade to replicability, your classmate would need to close the GitHub tab, read only the paper's text, and write an independent implementation, then check whether that independent version also lands near the reported number. Passing the first test and failing the second is common, and is exactly the gap this chapter is about.
Answer 2. The distinguishing step is resetting surviving weights to θ₀ before the next training round, rather than continuing training from their already-updated values. If a reimplementation skips that reset and simply keeps fine-tuning, the loss curve at the start of each new round would look smooth and continuous, dropping from wherever the previous round's loss ended, because the weights never moved. With the correct reset in place, the loss curve should show a visible jump upward at the start of each round (the network briefly gets worse), since the surviving weights have been yanked back to their random initial values and have to relearn from there. That visible jump is a cheap, concrete sanity check that the reset is actually happening in your code.
Answer 3. At prune_fraction = 0.2, tracing the per-layer counts (32→26→21→17→14 for fc1, 16→13→11→9→8 for fc2) shows the total remaining fraction drops below 50% after the 4th prune round, landing at 22/48 = 0.4583 (round 3's total, 26/48 = 0.5417, is still above 50%). At prune_fraction = 0.1, tracing the same two layers with k = int(0.1 × alive) each round (fc1: 32→29→27→25→23→21→19→18→17→16→15, fc2: 16→15→14→13→12→11→10→9→9→9→9, note fc2 stalls once int(0.1×9)=0) shows the total hits exactly 24/48 = 0.5000 after 10 rounds and only drops below 50% (23/48 = 0.4792) at round 11. So halving the pruning rate multiplies the number of rounds needed by roughly 2.75, not exactly 2. The naive "half the rate, double the rounds" intuition is wrong for two compounding reasons: integer truncation means some rounds prune zero weights from a layer once its alive count gets small relative to the rate, and the two layers are pruned independently, so the smaller layer (fc2) stalls out earlier and stops contributing further reductions. Since each IMP round means training the network from scratch, this also means the p=0.1 search costs roughly 2.75x the compute of the p=0.2 search to reach comparable sparsity, a real cost difference a paper's stated "pruning rate" hyperparameter can hide unless you trace it.
Answer 4. Two concrete causes: cuDNN's autotuned convolution algorithms can select different (non-bit-identical) kernels on different GPU architectures, and parallel floating-point reductions are not strictly associative, so summation order differences between hardware accumulate small errors that compound over many training steps. To close the gap, pin exact library versions in requirements.txt (including the CUDA toolkit and cuDNN version, not just torch), and in the script itself call torch.use_deterministic_algorithms(True) and set torch.backends.cudnn.deterministic = True (accepting the resulting slowdown), plus seed all four generators (Python, NumPy, torch CPU, torch CUDA) rather than only torch.manual_seed.
Answer 5. Instead of rewinding surviving weights all the way to their value at initialization (iteration 0), Frankle, Dziugaite, Roy, and Carbin rewind them to their value a small number of steps into training. The reason, tied to their linear mode connectivity findings, is that for deeper networks trained with the larger learning rates typical of that scale, SGD's noise in the very first steps can send otherwise-identical runs into different regions of the loss landscape, so literal iteration-0 rewinding produces subnetworks that are not reliably trainable in isolation. The implication for reproduction is important: faithfully copying the original 2019 paper's exact procedure (rewind to iteration 0) may reproduce the original paper's own small-CNN, MNIST/CIFAR-scale numbers reasonably well, since that is the regime it was tested in, but applying that same literal procedure to a materially larger model is reproducing the code, not the scientific claim, and will often fail to show the effect the paper is famous for. A correct larger-scale reproduction has to incorporate the later refinement, which is not stated anywhere in the original paper because it had not been discovered yet.
Answer 6. Magnitude is not comparable across layers with different scales: a layer initialized with a different fan-in (as fc1 and fc2 are, having 4 and 8 input units respectively) will have systematically different typical weight magnitudes under standard initialization schemes, so a single global threshold would prune almost entirely from whichever layer happens to have larger typical magnitudes, leaving the other layer nearly untouched, rather than removing a consistent 20% from each. Switching to a global threshold would break the clean per-layer arithmetic used in the trace above (32→26, 16→13, and so on), since the number pruned from each layer would no longer be a fixed fraction of that layer's count. It would instead depend on the actual trained weight values, which means, unlike the per-layer version, the round-by-round remaining-fraction sequence would no longer be predictable from counts alone and would become a genuinely random-seed-dependent quantity, exactly the kind of hidden dependency a reproduction needs to notice and state explicitly rather than discover by accident.
Think About It
Think about this: How would you explain reproducing research: from paper to code to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind reproducing research: from paper to code, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.