A fraud-detection team at a UPI payments company ships a new model and reports it in a Slack message: "97% accuracy on the fraud classifier." Leadership is pleased. A junior engineer on the team checks one number before celebrating: out of 10,000 transactions in the test set, how many are actually fraud? The answer is 200 — 2%. A model that predicts "not fraud" on every single transaction, without looking at a single feature, scores 98% accuracy. The celebrated model, at 97%, is worse than doing nothing. This is not a hypothetical trap; it is the single most common way machine learning results get reported wrong in industry and in student projects alike, and it exists because the team skipped the two disciplines this chapter is about: establishing a baseline before claiming a number means anything, and running an ablation study before claiming which part of the system is responsible for that number.
Baselines: a number is not evidence until it is compared to something
A baseline is a reference system, evaluated on the exact same test data with the exact same metric as your model, that tells you what "no skill" or "prior skill" looks like. Without one, 97% is just a number floating in space. Four baselines matter in practice, and a rigorous experiment report typically states more than one:
Majority-class (naive) baseline. Predict the most frequent label for every input, ignoring the features entirely. For the UPI fraud set above, "always predict not-fraud" gets 98% accuracy precisely because fraud is rare — 9,800 of 10,000 transactions are correctly called non-fraud by doing nothing at all. This baseline is what exposes the "accuracy paradox": on imbalanced data, accuracy rewards you for agreeing with the majority class and tells you almost nothing about whether you caught the minority class you actually care about.
Random baseline. Predict labels according to the class distribution (or uniformly at random) with no information from the input. Useful mainly for multi-class problems with many roughly-balanced classes, where the majority baseline is uninformative — e.g., a 12-way scene classifier has an ~8.3% random baseline.
Prior-system baseline. The best previously deployed or previously published system on this exact task — a rule-based fraud filter your company already runs, or the accuracy reported in the paper you are trying to beat. This is the baseline that determines whether your new model is worth the engineering cost of shipping it.
Human-performance baseline. Where feasible, what a trained human annotator scores on the same test set. Common in NLP and vision benchmarks (SQuAD, ImageNet) as an upper reference — not a floor to beat, but a ceiling that tells you how much headroom plausibly remains.
Note what a baseline is not: it is not "my model before I tuned the hyperparameters," and it is not "an earlier checkpoint of the same architecture." Those are useful sanity checks, but they are not baselines in the experiment-design sense — a baseline is an independently justified reference point that exists whether or not you built a model at all.
Ablation studies: isolating which piece of your system is doing the work
Once a model clears its baselines, a second question follows immediately: which part of the model is responsible for the win? A system rarely has one design choice — it has a feature set, an architecture, a pretraining objective, a loss function, a data augmentation pipeline, each contributing an unknown amount to the final metric. An ablation study answers this by removing or disabling exactly one component, re-evaluating the system with everything else held fixed, and attributing the resulting change in the metric to that component.
The name comes from surgery — "ablation" means removal of tissue to study what that tissue was doing — and the underlying logic is the same one you already use in a physics or chemistry practical: to find out whether a specific variable affects an outcome, you hold every other variable constant and change only that one. This is John Stuart Mill's "method of difference," and it is the entire justification for why an ablation study is trustworthy evidence rather than a guess: if the only thing that changed between two runs is one component, then the only thing that can explain a metric difference is that component.
This is also exactly why an ablation is a different kind of experiment from a baseline comparison. A baseline is an external reference — a different system entirely, often with a different architecture, feature set, and training procedure, so a gap against a baseline tells you "we are better than the alternative" but not "here is why." An ablation is internal and controlled — same system, same data, same training procedure, one component toggled — so a gap in an ablation tells you "here is why," at the cost of not telling you anything about how you compare to the outside world. A complete experiment section in a paper, or a complete evaluation of a production model, needs both.
Worked example: ablating the UPI fraud detector, fully derived
Take the fraud classifier from the opening scenario. It uses four input features: transaction amount, time-of-day, a device-fingerprint-change flag (has this device paid from this account before?), a transaction-velocity count (how many transactions from this account in the past ten minutes), and a merchant-category code. Test set: 10,000 transactions, 200 fraudulent (2% prevalence). Two baselines are established first:
- Majority-class baseline: predict "not fraud" always → accuracy = 9,800 / 10,000 = 98.0%, but recall on the fraud class = 0/200 = 0, so F1 (the harmonic mean of precision and recall, the metric that actually matters when the positive class is rare) = 0.00.
- Prior system baseline: the rule-based filter currently in production scores F1 = 0.62 on this same test set.
The full model, using all four features, produces this confusion matrix on the 200 fraud cases and 9,800 legitimate cases: 170 true positives, 30 false negatives, 40 false positives, 9,760 true negatives. Precision and recall are derived directly from these four counts:
Precision = TP / (TP + FP) = 170 / (170 + 40) = 170 / 210 = 0.8095
Recall = TP / (TP + FN) = 170 / (170 + 30) = 170 / 200 = 0.8500
F1 = 2 · P · R / (P + R)
= 2 · 0.8095 · 0.8500 / (0.8095 + 0.8500)
= 1.3762 / 1.6595
= 0.8293The full model clears both baselines comfortably (0.8293 against 0.62 and 0.00). Now the ablation: remove one feature at a time, retrain the same model architecture on the remaining three features, and re-evaluate on the identical test set.
Remove device-fingerprint-change. New confusion matrix: 150 TP, 50 FN, 55 FP. Precision = 150/205 = 0.7317, Recall = 150/200 = 0.7500, F1 = 2(0.7317)(0.7500)/(0.7317+0.7500) = 0.7407. Drop from full model = 0.8293 − 0.7407 = 0.0885.
Remove transaction-velocity. New confusion matrix: 165 TP, 35 FN, 42 FP. Precision = 165/207 = 0.7971, Recall = 165/200 = 0.8250, F1 = 2(0.7971)(0.8250)/(0.7971+0.8250) = 0.8108. Drop = 0.8293 − 0.8108 = 0.0185.
Remove merchant-category. New confusion matrix: 168 TP, 32 FN, 39 FP. Precision = 168/207 = 0.8116, Recall = 168/200 = 0.8400, F1 = 2(0.8116)(0.8400)/(0.8116+0.8400) = 0.8256. Drop = 0.8293 − 0.8256 = 0.0037.
The ablation table now ranks the four features by how much of the model's total lift over baseline each one is responsible for: device-fingerprint-change contributes an F1 drop of 0.0885 if removed — roughly 24 times the contribution of merchant-category's 0.0037. Time-of-day and amount were never removed in this study, so this table says nothing about their contribution; a full ablation would ablate every feature (and combinations of features) to build a complete picture, and would also report a no-feature or single-feature-only floor. What this partial study already licenses, though, is an engineering decision the accuracy number alone could never justify: merchant-category can likely be dropped from the production feature pipeline — cutting a database lookup from every inference call — for a cost of 0.0037 F1, while device-fingerprint-change must be kept and, if anything, investigated further since it is doing most of the work.
Ablation studies in real published research
This pattern — hold the system fixed, remove one piece, measure the gap — is exactly how three foundational deep learning papers justify their central claims, and reading their ablation tables is a useful habit before trusting any paper's abstract.
He, Zhang, Ren, and Sun, "Deep Residual Learning for Image Recognition" (2016), motivate the residual (skip) connection with a controlled comparison: a plain 34-layer convolutional network and a 34-layer network with residual connections added, everything else — depth, width, training data, optimizer — held identical. The plain deeper network trains to a higher error than a shallower plain network (the "degradation problem": adding depth without residual connections makes optimization harder, not easier), while the residual version of the deeper network outperforms the residual version of the shallower one. The ablation is the entire argument of the paper: residual connections, and only residual connections, are what let depth help instead of hurt.
Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019), run an ablation on their own pretraining objective: a version of the model trained without the next-sentence-prediction task, and a version that replaces bidirectional masked-language-model pretraining with a left-to-right (unidirectional) objective, holding model size and data fixed. Both ablated variants lose accuracy on downstream tasks such as MNLI and SQuAD, with the loss from removing bidirectionality substantially larger than the loss from removing next-sentence-prediction — evidence, produced by the ablation table and nowhere else in the paper, for which of the two design choices actually deserves credit for BERT's results.
Vaswani et al., "Attention Is All You Need" (2017), ablate the number of attention heads in the Transformer while holding total model capacity fixed. The result is not monotonic: too few heads hurts translation quality, but so does too many — there is an interior optimum. This is a case where the ablation reveals something a single headline number never would: "more attention heads is better" is false, and only the ablation sweep shows that.
The misconception this chapter has to correct
Every one of the ablations above invites a conclusion that feels obvious and is not automatically true: "performance dropped when I removed component X, therefore X is functioning the way I designed it to function." This is a confound, not a proof, and it is the single most common mistake in student and even published ablation studies. When you remove a component, you are almost always changing more than one thing at once — you are usually also changing the model's parameter count, its effective capacity, and sometimes its training dynamics (a smaller model may need a different learning rate or more epochs to converge fairly). A performance drop after removing device-fingerprint-change could mean the feature is causally informative about fraud, exactly as hypothesized — or it could partly reflect that the model simply has fewer input dimensions to fit with, a capacity effect that has nothing to do with what that specific feature encodes.
The fix is to control for capacity, not just for the component's presence. A rigorous ablation replaces the removed piece with a "matched" of the same parameter count — a randomly initialized, frozen version of the same feature transform, or noise of the same dimensionality — rather than simply deleting it. If the model with the real component beats the model with the size-matched dummy, the case for "this specific component's content matters" is much stronger than a raw removal can support. Lipton and Steinhardt, "Troubling Trends in Machine Learning Scholarship" (2018), catalogue exactly this failure mode as a recurring problem in published ablation studies: components get removed without controlling for the confounds their removal introduces, and the paper's causal story ends up resting on a comparison that was never actually isolated to one variable.
A second, smaller misconception worth naming: students often treat "ablation" and "hyperparameter search" as the same activity because both involve running many variants of a model. They are not the same. A hyperparameter search (learning rate, batch size, number of layers) asks "what is the best setting of this knob?" and optimizes toward a maximum. An ablation asks "how much does this component matter at all?" and its answer is a magnitude of contribution, not a best setting — you don't tune whether next-sentence-prediction is in the model, you measure what happens with and without it.
Reading and running an ablation study: the mechanism
The chart makes the two disciplines of this chapter visible at once. The gray bars are baselines — external reference systems evaluated once, sitting to the left of the model entirely. The blue bar is the full model, the single reference every ablation is measured against; the dashed line at its height is drawn all the way across so the gap between it and each amber bar is a directly comparable visual quantity, which is the F1 drop attributable to that one feature. Reading left to right: the distance from "Majority baseline" to "Full model" is the total achievement of building the classifier at all; the distance from "Full model" down to each amber bar is how much of that achievement traces to one specific design choice. A paper, or a production experiment report, that shows only the blue bar has told you the model works. Only the gray bars and the amber bars together tell you whether it was worth building, and which part of it to keep.
Active recall
Attempt each question before reading its answer.
1. A student builds a binary classifier for a dataset where the positive class is 50% of examples (perfectly balanced) and reports "my model beats the majority-class baseline's accuracy of 50% by 30 points, so it's clearly a strong model." Is the majority-class baseline the most informative baseline to cite here? Why or why not?
2. Using the fraud detector's full-model confusion matrix (170 TP, 30 FN, 40 FP, 9,760 TN), compute accuracy. Explain why accuracy alone, even at this respectable-looking value, would have been a misleading number to report as the headline result.
3. A researcher ablates a "self-attention" component from a Transformer by replacing it with a randomly initialized, frozen (untrained) attention layer of the same parameter count, rather than deleting it outright. What confound does this design control for that a simple deletion would not, and why does that make the resulting comparison more trustworthy?
4. Suppose the fraud team relabels their test set after an audit and the true fraud prevalence turns out to be 5% (500 of 10,000 transactions) rather than 2%, with everything else about the four ablation experiments unchanged in method. (a) Recompute the majority-class baseline's accuracy. (b) Does the majority-class baseline's F1 score change? (c) Does the earlier ranking of which feature mattered most (device-fingerprint-change > transaction-velocity > merchant-category) still hold under the new prevalence, and how would you actually find out?
5. Why is an ablation study, by itself, never sufficient to justify shipping a model — what question does it leave completely unanswered that only a baseline comparison answers?
6. A published ablation table reports only one training run per variant (one number per row, no repeated trials). What is the specific risk of trusting a small gap in that table — say, a 0.3-point F1 difference between two rows — and what would you ask the authors to report instead?
Answers
1. No — on a balanced dataset the majority-class baseline (50%) and the random baseline (also ~50% for two classes) coincide and are both weak, uninformative floors; nearly any trained model clears them easily, so beating one by 30 points is a low bar. The baseline that would actually test whether the model is "clearly strong" is a prior-system or published state-of-the-art baseline on the same task — if the best existing system already scores 78%, then "50 → 80" looks impressive against the naive baseline but is only a 2-point improvement over the system that actually matters to beat.
2. Accuracy = (TP + TN) / total = (170 + 9,760) / 10,000 = 9,930 / 10,000 = 99.3%. This number is misleading precisely because it is dominated by the 9,760 easy true negatives among 9,800 legitimate transactions — a model could get most of that 99.3% simply by being decent at the easy majority class while missing a meaningful fraction of fraud (30 of 200 missed here, a 15% miss rate on the class that actually costs the business money). F1 = 0.8293, computed from precision and recall on the positive class only, is the number that actually reflects performance on the rare, costly class and is why it — not accuracy — was used as the metric throughout the worked example.
3. Simple deletion changes two things simultaneously: it removes whatever function self-attention was performing, and it also removes the associated parameters, changing the model's total capacity and possibly its optimization dynamics. A performance drop after simple deletion can't distinguish "attention's specific computation mattered" from "the model just had fewer parameters to work with." Replacing it with a frozen, randomly-initialized layer of matched parameter count holds capacity fixed while still removing the trained, meaningful computation — so a further drop against this matched control isolates the contribution of what self-attention actually learns, which is the claim the ablation is trying to support.
4. (a) New total = 10,000, new fraud count = 500, non-fraud = 9,500, so majority-class accuracy = 9,500/10,000 = 95.0% (dropped 3 points from 98.0%, exactly tracking the 3-point increase in positive-class prevalence). (b) No — the majority-class baseline still predicts "not fraud" on every single example regardless of how many positives exist in the data, so its recall on the fraud class is still 0/500 = 0 and its F1 is still 0.00; F1 for a constant-negative predictor is invariant to prevalence, only accuracy moves. (c) Not necessarily, and it cannot be answered from the numbers already given — prevalence is a property of the label distribution, and it can interact with how predictive each feature is (e.g., transaction-velocity might correlate with the newly-included fraud cases differently than it did in the original 2% sample if the audit's relabeling systematically reclassified a particular kind of transaction). The only correct answer is that all four ablation runs — full model and the three single-feature-removed variants — must be re-trained and re-evaluated on the corrected labels before the ranking can be claimed to hold; reusing the old ranking on new data is exactly the kind of unearned inference this chapter has been arguing against.
5. An ablation study only ever compares your system to itself with pieces removed — every number in the table is upper-bounded by your own full model's performance, so it can tell you that device-fingerprint-change contributes 0.0885 F1 within this architecture, but it says nothing about whether 0.8293 F1 overall is good enough to beat what's already in production, or good enough to be worth the cost of deploying. That question — is this system, as a whole, better than the alternative — is answered only by a baseline comparison against an external reference (the prior rule-based filter, a competitor's published system, or human performance), never by an ablation.
6. A single run's score is a sample from a distribution — differences of a few tenths of a point between two variants can easily be within the noise produced by random weight initialization, data shuffling order, or dropout masking, especially on smaller datasets. Trusting a 0.3-point gap without knowing that noise level risks attributing to a design choice a difference that would flip sign on a second run with a different random seed. The right ask is for the authors to report results averaged over multiple random seeds with a spread — standard deviation or a confidence interval — so the gap can be judged against the run-to-run variance, and, ideally, a significance test (e.g., a paired t-test across matched seeds) confirming the gap is unlikely to be noise.
Think About It
Think about this: How would you explain experiment design: ablation studies and baselines to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind experiment design: ablation studies and baselines, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.