The dataset that broke ten benchmarks
In 2021, three researchers — Curtis Northcutt, Anish Athalye, and Jonas Mueller — ran a deceptively simple audit. They took the ten most-cited test sets in machine learning, including ImageNet, MNIST, and CIFAR-10, and checked whether the labels were actually correct. Every single one of the ten sets contained label errors, at rates ranging from under 1% to well above 5%, averaging a little more than 3% across the board. On ImageNet's validation set — the benchmark that has anchored image-classification leaderboards for over a decade — correcting the mislabeled examples was enough to change which model architecture ranked first. Models that looked state-of-the-art were, in part, state-of-the-art at fitting somebody's typo.
This is the central fact this chapter is built around: a model cannot be more correct than the data it was told was correct. Every layer of a transformer, every regularizer, every learning-rate schedule you tuned in earlier chapters operates on the assumption that the label next to an example means what it says. Dataset curation is the discipline of making that assumption defensible — and, when it isn't fully defensible, of knowing precisely how much it is costing you. A related study by Nithya Sambasivan and colleagues (CHI 2021) interviewed AI practitioners building high-stakes systems across India and other Global South deployments and found that data problems discovered late in a pipeline routinely cascade into compounding downstream harms — a bad label caught after deployment costs far more than the same label caught before training. That asymmetry is why curation belongs at the front of the MLOps lifecycle, not as cleanup after a model underperforms.
What "data quality" actually means
Treat "quality" as a bundle of independently measurable properties, not a vague virtue. For a dataset, the six that matter in practice are:
Accuracy — does the label match ground truth? This is the dimension the Northcutt audit measured, and the one this chapter's worked example quantifies.
Completeness — are required fields present for every record, or are there silent nulls that a downstream feature pipeline will quietly coerce into zero?
Consistency — is the same real-world entity represented the same way everywhere? A transaction timestamp stored as IST in one source system and UTC in another is a consistency failure that will not throw an error; it will just make your model's "time since last transaction" feature wrong by 5.5 hours for a subset of rows.
Uniqueness — are there duplicate or near-duplicate records inflating the weight of certain examples, or worse, leaking across a train/test split?
Timeliness — does the data reflect the distribution the model will actually see in production, or has the underlying process drifted since collection?
Representativeness — does the sample cover the population the model must generalize to, including its rare but consequential slices?
These dimensions interact. A dataset can be 100% complete and perfectly consistent while still being useless, because every label was assigned by an annotator who systematically missed a hard-to-detect class. That is the failure mode the rest of this chapter works through in detail, because it is the one most students underestimate.
Worked example: how a little label noise wrecks a rare-event detector
Consider a dataset a fintech team is building to detect fraudulent UPI transactions — a realistic instance of the class-imbalance problem that dominates fraud, medical screening, and manufacturing-defect detection. Fraud is rare: assume the true rate is 0.5%. Out of 100,000 transactions, that means:
True fraud = 500. True legitimate = 99,500.
Labels are produced by a mix of automated rules and human review, and like any labeling process, it is not perfect. Suppose two independent noise processes are at work: 1% of genuinely legitimate transactions get mislabeled "fraud" (an overzealous rule flags an unusual-but-innocent purchase), and 10% of genuinely fraudulent transactions get mislabeled "legit" (sophisticated fraud is, definitionally, the kind that's hard for a rule or a reviewer to catch — this is why the miss rate on the minority class is so much higher than the false-alarm rate on the majority class).
Trace the counts:
Legit transactions mislabeled "fraud": 99,500 × 0.01 = 995.
Fraud transactions mislabeled "legit": 500 × 0.10 = 50, which means 500 − 50 = 450 fraud transactions keep their correct "fraud" label.
Total transactions carrying a noisy "fraud" label: 450 (correct) + 995 (false) = 1,445.
Total transactions carrying a noisy "legit" label: 98,505 (correct) + 50 (missed fraud) = 98,555. Check: 1,445 + 98,555 = 100,000. ✓
Now ask the question that matters: what is the best precision any classifier could achieve, if it learned to reproduce these noisy labels perfectly? This is a useful idealization — a classifier that has essentially memorized the labeling function, which is the ceiling every real, imperfect model is trying to approach. Its predicted-fraud set is exactly the 1,445 examples carrying a noisy "fraud" label. Of those, only 450 are actually fraud.
Precision ceiling = 450 / 1,445 = 31.1%.
Recall ceiling = 450 / 500 = 90%.
Read that again: a labeling process with a 1% false-positive rate on the majority class and a 10% miss rate on the minority class caps the achievable precision at 31%, no matter how good the architecture, the optimizer, or the amount of compute thrown at it. Recall looks fine at 90% — which is exactly the trap, because a team monitoring recall alone would conclude the system is working. It is the base-rate effect from Bayesian reasoning, in the same family as why a highly accurate medical test still produces mostly false positives when screening a rare disease: rarity amplifies the damage a fixed false-positive rate does to precision.
This generalizes cleanly. Let r be the true prevalence of the rare class, p the false-positive noise rate on the majority class, and q the miss rate on the minority class. Then:
Precision ceiling = (1 − q)·r / [(1 − q)·r + p·(1 − r)]
Plugging in r = 0.005, p = 0.01, q = 0.10 reproduces 31.1% exactly. Two things fall out of this formula that are worth internalizing: recall depends only on q (the miss rate), never on prevalence — which is why the 90% figure above will hold regardless of how rare fraud is — while precision is brutally sensitive to prevalence, because the false-positive noise is being compared against a shrinking pool of true positives. This is precisely the mechanism Curtis Northcutt formalized more generally in "Confident Learning: Estimating Uncertainty in Dataset Labels" (Northcutt, Jiang, and Chuang, JAIR 2021): label noise doesn't just add random jitter to a model's performance, it imposes a hard, computable ceiling that depends on the joint distribution of true and noisy labels.
A curation pipeline you can actually build
Fixing this in production is not "look at the data more carefully." It is a sequence of concrete, automatable gates, each targeting one of the six quality dimensions above, run before a single training epoch happens. The diagram below shows the seven gates for the UPI fraud dataset, and — using the exact numbers derived above — what happens to the precision ceiling once label cleaning is done properly instead of skipped.
The seven gates map directly onto the quality dimensions from the previous section. Schema validation catches completeness and consistency failures before they propagate. Deduplication protects uniqueness, and — critically for evaluation — prevents a near-duplicate transaction from landing in both the train and test split, which silently inflates test accuracy by letting the model "recognize" a record it effectively memorized during training. Label QA, combining inter-annotator agreement (IAA) statistics with a confident-learning-style disagreement check, is the gate that would have caught the 995 false "fraud" labels and the 50 missed frauds in the worked example. Leakage audit removes features that are only known after the outcome is decided (a chargeback flag set by the bank after investigating a case is not something a real-time model has access to at prediction time, and training on it teaches the model to shortcut). Temporal splitting matters specifically for fraud because fraud tactics evolve; a random 80/20 split would let the model train on transactions from next month and test on transactions from this month, silently leaking future fraud patterns backward in time.
Catching the problems in code, not by eye
Deduplication is the cheapest gate to implement and the easiest to skip. On the UPI dataset, an exact-key duplicate check on the transaction identifier looks like this:
import pandas as pd
df = pd.DataFrame({
"txn_id": ["T1", "T2", "T3", "T2", "T4"],
"amount": [500, 1200, 75, 1200, 3000],
"label": ["legit", "fraud", "legit", "fraud", "legit"],
})
exact_dupes = df[df.duplicated(subset=["txn_id"], keep=False)]
print(exact_dupes)
Tracing this by hand: df.duplicated(subset=["txn_id"], keep=False) marks every row whose txn_id appears more than once as True, on both occurrences (that is what keep=False does — the default, keep="first", would hide the first copy and only flag the second). Row index 1 and row index 3 both carry txn_id = "T2", so exact_dupes contains exactly those two rows — both showing amount 1200 and label "fraud" — while rows 0, 2, and 4 are excluded. In a fraud dataset, a duplicated transaction record is not a harmless repeat; it doubles that example's influence on the loss function and, if one copy lands in train and the other in test, it manufactures an artificially easy test case.
Label QA is harder because there is no ground truth to check against directly — that is the entire problem. A practical approximation of Northcutt's confident-learning idea is to train a model with cross-validation and flag every example where the model's out-of-fold prediction disagrees with the label it was given. Disagreement doesn't prove the label is wrong, but it concentrates review effort on the examples most likely to be wrong, instead of auditing all 100,000 by hand:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
rng = np.random.default_rng(seed=7)
n_legit, n_fraud = 9950, 50 # 0.5% base rate, scaled for a runnable demo
X_legit = rng.normal(loc=0.0, scale=1.0, size=(n_legit, 2))
X_fraud = rng.normal(loc=2.5, scale=1.0, size=(n_fraud, 2))
X = np.vstack([X_legit, X_fraud])
y_true = np.array([0] * n_legit + [1] * n_fraud)
# Inject the same noise process as the worked example
y_noisy = y_true.copy()
legit_idx = np.where(y_true == 0)[0]
fraud_idx = np.where(y_true == 1)[0]
flip_legit = rng.choice(legit_idx, size=int(0.01 * len(legit_idx)), replace=False)
flip_fraud = rng.choice(fraud_idx, size=int(0.10 * len(fraud_idx)), replace=False)
y_noisy[flip_legit] = 1
y_noisy[flip_fraud] = 0
clf = LogisticRegression()
oof_proba = cross_val_predict(clf, X, y_noisy, cv=5, method="predict_proba")[:, 1]
oof_pred = (oof_proba >= 0.5).astype(int)
suspect_mask = oof_pred != y_noisy
print("Suspected mislabeled examples:", suspect_mask.sum())
Every variable used is defined before use: rng, n_legit/n_fraud, the feature arrays, y_true, and y_noisy are all built in order, and cross_val_predict with an integer cv on a classification target uses stratified folds internally, so both classes are present in every fold despite fraud being only 0.5% of the data. This will not recover every flipped label — logistic regression is a simple linear model and the synthetic classes here overlap somewhat by construction — but it is a legitimate, cheap first pass: examples where a held-out model confidently disagrees with the recorded label are exactly the examples a human reviewer should look at first, rather than sampling 100,000 rows uniformly at random.
Common misconception: "more data fixes bad labels"
Students who have just spent a year on scaling laws and large training runs tend to assume that data quality problems are a volume problem — that doubling the dataset dilutes the noise. Look back at the precision-ceiling formula: precision = (1 − q)·r / [(1 − q)·r + p·(1 − r)]. Every term is a rate, not a count. Doubling N to 200,000 transactions while keeping the same noise rates p = 1% and q = 10% and the same prevalence r = 0.5% leaves the formula, and therefore the 31.1% ceiling, completely unchanged — you now have 900 correctly labeled fraud examples and 1,990 false "fraud" labels instead of 450 and 995, but the ratio is identical. More data collected through the same flawed labeling process reproduces the same ceiling at higher confidence; it does not raise the ceiling. The only way to raise it is to change p or q directly — a better labeling process, adjudication of disagreements, or a more careful rule for flagging edge cases. This is precisely why the fourth active-recall question below asks you to hold the noise rates fixed and change only the base rate: it is the base rate, not the volume, that moves the ceiling in this formula, and telling those two apart is the whole point of understanding the mechanism instead of memorizing the number.
Documenting what you built
A curated dataset that isn't documented decays back into an undocumented one the moment its creator changes teams. Gebru et al., first circulated in 2018 and published in a more complete form in Communications of the ACM in 2021, proposed "datasheets for datasets": a structured record answering who collected the data and why, what preprocessing and label-QA steps were applied (exactly the seven gates above), what the known label-noise characteristics are, and what the dataset should not be used for. For the UPI dataset, a datasheet entry stating "1% legit-to-fraud and 10% fraud-to-legit noise, measured via a 2,000-transaction human-adjudicated audit sample" is worth more to the next engineer than any amount of model-card boilerplate about the classifier trained on top of it, because it tells them exactly where the precision ceiling in their own downstream experiments is coming from.
Active recall
Attempt each question before reading the answer beneath it.
1. Name the six data quality dimensions introduced in this chapter and give one UPI-dataset example of each.
2. Using the worked example's numbers (100,000 transactions, 0.5% true fraud rate, 1% legit→fraud noise, 10% fraud→legit noise), compute the recall ceiling and show your work.
3. A teammate says: "Our held-out test accuracy is 99.8%, so the label quality must be fine." What is wrong with this reasoning?
4. Suppose the true fraud rate is actually 2% instead of 0.5% (so 2,000 fraud and 98,000 legit out of 100,000), with the same noise rates as before (1% legit→fraud, 10% fraud→legit). Recompute the noisy "fraud" label count and the precision ceiling, and explain why the recall ceiling does not change.
5. Which two stages in the pipeline diagram specifically guard against data leakage, and why does a fraud-detection dataset need a temporal split rather than a random one?
6. A colleague proposes fixing the 995 false "fraud" legit-transaction labels by simply deleting those rows rather than relabeling them. What risk does this introduce?
Answers
1. Accuracy — does a transaction actually match its fraud/legit label. Completeness — is the merchant category code present for every row. Consistency — are all timestamps stored in the same timezone across source systems. Uniqueness — no transaction ID appears twice across the dataset. Timeliness — does the fraud pattern distribution reflect this month's tactics, not two years ago. Representativeness — does the dataset include enough examples of rare fraud types (e.g., account-takeover fraud), not just the common ones.
2. Recall ceiling = correctly labeled fraud ÷ true fraud = 450 / 500 = 90%. This is fixed once you know that 10% of the 500 true fraud transactions (50 of them) were mislabeled "legit," leaving 450 correctly labeled; recall is defined only over the true-positive population, so it never involves the 995 false-positive legit-mislabeled-as-fraud count at all.
3. At a 0.5% true fraud rate, a model that predicts "legit" for every single transaction achieves 99.5% accuracy while catching zero fraud. 99.8% accuracy is barely better than that trivial baseline and says almost nothing about whether the rare, high-stakes class is being handled correctly. Worse, accuracy is computed against the same recorded labels the model was trained on — if those labels are noisy, a model that has learned to reproduce the noise will still score well on accuracy against that same noisy test set. Accuracy needs to be replaced with precision/recall measured against a small, carefully human-adjudicated gold-standard audit sample, not the bulk noisy label set.
4. Legit mislabeled "fraud": 98,000 × 0.01 = 980. Fraud mislabeled "legit": 2,000 × 0.10 = 200, so 1,800 fraud transactions keep the correct label. Noisy "fraud" label count = 1,800 + 980 = 2,780. Precision ceiling = 1,800 / 2,780 = 64.75%, roughly double the original 31.1%. The recall ceiling is unchanged at 1,800/2,000 = 90%, because recall depends only on the 10% miss rate q, which was held fixed — it is structurally independent of how rare the positive class is. The ripple is entirely on precision: the same absolute false-positive noise (roughly 980–995 mislabeled legit transactions either way) is now diluted across a much larger pool of true positives, so it damages precision far less. This is the formula's key lesson — prevalence, not the noise rate alone, sets how catastrophic a fixed labeling error rate becomes.
5. The Leakage Audit gate (drops features only knowable after the outcome, like a bank's post-hoc chargeback flag) and the Temporal Split gate (ensures test transactions are always later in time than training transactions) both guard against leakage. Fraud specifically needs a temporal split because fraud tactics evolve — a random split would let a model see transactions from after the test period during training, effectively letting it learn from fraud patterns that, at true prediction time, would not yet have existed. That produces an optimistic evaluation that will not survive contact with production.
6. Deleting the 995 flagged rows instead of sending them for a second annotation pass removes them from the dataset entirely rather than correcting them — but these are disproportionately the borderline, most fraud-like legitimate transactions, since that is exactly why the rule flagged them in the first place. Deleting them means the training set loses its hardest, most informative "hard negative" examples near the decision boundary, biasing the remaining legit class toward easy, obviously-clean transactions. The model then never learns to distinguish genuinely tricky legitimate purchases from fraud, and precision in production — on the real distribution, which still contains those hard cases — will be worse than the audit predicted. The correct fix is adjudication: route flagged examples to a second, ideally more senior or specialized reviewer, and correct the label rather than discard the row.
Think About It
Think about this: How would you explain dataset curation and data quality to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind dataset curation and data quality, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.