Feature Selection: Choosing What Matters

The Fraud Model That Learned Too Much

Every time someone scans a QR code at a Chennai coffee cart or sends ₹500 to a friend, that payment moves through UPI, the Unified Payments Interface built by the National Payments Corporation of India (NPCI) and launched in 2016. Somewhere in the fraction of a second between "tap to pay" and "payment successful," a machine learning model decides whether that particular transaction looks safe or suspicious.

Picture the data science team at a payments company building that fraud model. Every transaction that flows through their systems comes with dozens of fields attached: the amount, the time of day, the sender's account age, how many times this sender has paid this receiver before, whether the receiver was added as a new payee in the last hour, the sender's device operating system, their phone's battery percentage at the moment of payment, the color theme of their payment app, the number of contacts saved on their phone, and forty-odd columns besides. A junior engineer on the team reasons, not unreasonably, that more information should only help: feed the model everything, and let it work out what matters.

Three weeks later, the results are in, and they are not encouraging. The model trained on all 46 fields is not more accurate than a simpler version. On transactions it has never seen before, it is slightly worse. It also takes noticeably longer to score each transaction, which matters when UPI is processing an enormous volume of payments every single day and each one has to be approved or flagged in a fraction of a second. And when an auditor asks why a particular transfer got blocked, the honest answer satisfies nobody: "a combination of 46 signals, including phone battery level, pushed a hidden score past a threshold."

The problem was never a shortage of data. It was too much of the wrong kind. This is exactly the problem feature selection exists to solve.

What Exactly Is a Feature?

In machine learning, a feature is a measurable input variable: one column in a dataset that a model is allowed to look at when making a prediction. If a dataset contains transaction amount, time of day, and sender's account age as columns, each of those is a feature. The thing being predicted (fraudulent or not, house price, exam score) is the target, and it is not a feature; everything else in the table is a candidate.

Feature selection is the process of choosing which of those candidate columns a model should actually use, and discarding the rest, not because the discarded columns are wrong or badly measured, but because keeping them does not help, and often actively hurts, the model's ability to predict correctly on new data. It is worth separating this from a related idea, feature engineering, which is about creating new columns: computing "transaction amount ÷ this sender's average transaction amount over the last 30 days" out of two raw fields, for instance. Feature engineering expands the pool of candidates; feature selection decides which candidates, raw or engineered, earn a place in the final model.

Why More Features Can Make a Model Worse

It feels intuitive that giving a model more information should never hurt. In practice, five things go wrong when a dataset carries far more columns than it needs.

• The curse of dimensionality. As the number of features grows, the space of possible feature combinations explodes, and a fixed amount of training data ends up covering that space more and more thinly.
• Noise. A feature with no real relationship to the target (a customer's phone wallpaper color, say) still contains random ups and downs, and a flexible enough model can mistake that randomness for a pattern.
• Redundancy. Two features carrying nearly the same information (a temperature reading in both Celsius and Fahrenheit, for instance) add no new signal, but they can confuse a model about how much credit each one deserves.
• Cost. Every extra feature means extra data to collect, clean, store, and feed through the model at prediction time, an expense that grows fast at the scale of a national payments system processing transactions every second.
• Interpretability. A model built on six well-understood signals can be explained to an auditor or a regulator. A model tangled up in forty-six signals usually cannot.

The curse of dimensionality is worth slowing down on, because the name sounds abstract but the effect is very concrete. Suppose 100 students' mock exam results are grouped by one feature: attendance, say, split into low, medium, and high. That gives three buckets, roughly 33 students in each, enough to see a real pattern per bucket. Add a second feature, hours studied, also split three ways, and the buckets become 3 × 3 = 9, roughly 11 students each. This is thinner, but still usable. Add a third feature and it becomes 27 buckets, about 4 students each. By 10 features split three ways each, the number of buckets is 3^10 = 59,049, for a dataset of only 100 students. Almost every bucket is empty, and the few that are not contain a single student each: one data point pretending to be a pattern. This is precisely the mechanism overfitting exploits: a model with too many features relative to its training data stops learning genuine relationships and starts memorizing coincidences that will not repeat on the next student, or the next transaction.

Redundancy causes a related but distinct problem called multicollinearity: when two or more features are highly correlated with each other, a linear model struggles to decide how much of the outcome to credit to each one, and the resulting weights can become unstable. Small changes in the training data can swing a feature's assigned importance wildly, even though the model's overall predictions barely change. Dropping the redundant copy fixes this without losing any real information.

Three Families of Feature Selection Methods

Once the problem is clear (too many columns, not enough of them pulling their weight), there are three broad strategies for deciding what to keep. They differ mainly in how much they trust a specific model to make that decision.

• Filter methods score each feature on its own, using statistics such as correlation and variance, before any model is trained. They are fast and do not care which algorithm will eventually be used, which also means they can miss a feature that only matters in combination with another one.
• Wrapper methods actually train a model on different subsets of features and keep whichever subset performs best on held-out data. They catch interactions between features that filter methods miss, at the cost of training many models instead of one.
• Embedded methods build feature selection into the training process of a single model. The model decides, as a side effect of learning, which features deserve non-zero weight or high importance. They sit in the middle, nearly as cheap as a filter and nearly as thorough as a wrapper.

A well-run data science team, including the one behind the fraud model, typically chains more than one of these: a fast filter pass to eliminate the obviously useless columns, followed by a more careful wrapper or embedded pass on whatever survives.

Filter Methods: Judging a Feature Before Any Model Sees It

The simplest filter check needs no formula at all: if a feature has almost the same value in every row of the dataset (every transaction happened over "UPI", every student is in "Grade 10, Section B"), it has zero, or close to zero, variance, and therefore carries no information that could distinguish one row from another. A variance threshold filter drops any feature whose variance falls below a chosen cutoff, without even glancing at the target column.

The more common filter tool, when both the feature and the target are numeric, is the correlation coefficient, usually Pearson's r. It is a single number between -1 and +1 that measures how strongly two variables move together in a straight-line relationship. An r close to +1 means that as one variable rises, the other reliably rises too; close to -1 means one reliably falls as the other rises; close to 0 means knowing one variable tells you almost nothing about the other, at least not in a linear sense. For a feature X and target Y measured across n rows, it is calculated as:

r = Σ(x - mean(x))(y - mean(y)) / sqrt( Σ(x - mean(x))² × Σ(y - mean(y))² )

which looks dense until it is run through an actual dataset, one row at a time.

Correlation only catches straight-line relationships, and plenty of real ones are not straight. Mutual information is a filter score built for that gap: it measures how much knowing one variable reduces uncertainty about another, whether the pattern is a line, a curve, or something more tangled, which is useful for a feature like transaction hour, where fraud risk might spike at odd hours without following any straight trend. For a categorical feature paired with a categorical target, say payment app used against a fraud yes-or-no label, a chi-square test checks whether the two are related more than chance alone would produce. Scikit-learn ships all three as ready-made functions under sklearn.feature_selection: f_regression for correlation-style scores, mutual_info_regression or mutual_info_classif for mutual information, and chi2 for the chi-square test.

Worked Example: Which Features Predict a Mock Score?

An AICI Grade 10 batch sits a mock board exam, and eight students' records are pulled for a quick check: how many hours each one studied that week, how many siblings they have, and their score out of 100.

Student   Hours studied/week   Siblings   Mock score
   A              5                0          58
   B              8                2          72
   C              3                2          50
   D             10                3          88
   E              6                3          65
   F             12                0          91
   G              4                1          55
   H              9                1          80

Hours studied is the obvious candidate feature. Number of siblings is thrown in almost as a joke; there is no reason it should predict an exam score. But that instinct is worth testing with numbers instead of just trusting it.

Start with the mean of each column. Hours studied sums to 57 across 8 students, so mean(hours) = 57 / 8 = 7.125. Scores sum to 559, so mean(score) = 559 / 8 = 69.875. Every student's deviation from these means is then computed, multiplied, and squared:

Student   dx=hours-7.125   dy=score-69.875   dx·dy      dx²       dy²
   A         -2.125            -11.875        25.234     4.516    141.016
   B          0.875              2.125         1.859     0.766      4.516
   C         -4.125            -19.875        81.984    17.016    395.016
   D          2.875             18.125        52.109     8.266    328.516
   E         -1.125             -4.875         5.484     1.266     23.766
   F          4.875             21.125       102.984    23.766    446.266
   G         -3.125            -14.875        46.484     9.766    221.266
   H          1.875             10.125        18.984     3.516    102.516
                                     Sum:     335.125    68.875   1662.875

Plugging the three sums into the formula:

r(hours, score) = 335.125 / sqrt(68.875 × 1662.875)
                = 335.125 / sqrt(114,530.52)
                = 335.125 / 338.42
                ≈ 0.990

Now repeat the same process for siblings. Siblings sum to 12 across 8 students, so mean(siblings) = 12 / 8 = 1.5. The score deviations (dy) are unchanged from the table above; only the siblings side needs recomputing:

Student   Siblings   dx=sib-1.5   dx²     dx·dy (using dy above)
   A          0         -1.5      2.25       17.812
   B          2          0.5      0.25        1.062
   C          2          0.5      0.25       -9.938
   D          3          1.5      2.25       27.188
   E          3          1.5      2.25       -7.312
   F          0         -1.5      2.25      -31.688
   G          1         -0.5      0.25        7.438
   H          1         -0.5      0.25       -5.062
                                Sum:  10.00    -0.500

r(siblings, score) = -0.500 / sqrt(10 × 1662.875)
                    = -0.500 / sqrt(16,628.75)
                    = -0.500 / 128.95
                    ≈ -0.004

Hours studied comes out at r ≈ 0.990, about as close to a perfect straight-line relationship as real data ever gets: study time and exam performance genuinely move together. Number of siblings comes out at r ≈ -0.004, indistinguishable from zero. The instinct was right, and now it is a number. This feature carries no usable signal about exam performance, and a filter method would drop it before a single model is trained, saving both the compute and the risk of overfitting to it.

Confirming It With Code

The arithmetic above is exactly what pandas automates with a single method call:

import pandas as pd

data = {
    "hours_studied": [5, 8, 3, 10, 6, 12, 4, 9],
    "siblings":      [0, 2, 2, 3, 3, 0, 1, 1],
    "mock_score":    [58, 72, 50, 88, 65, 91, 55, 80]
}
df = pd.DataFrame(data)

print(df.corr(numeric_only=True)["mock_score"])

hours_studied    0.990253
siblings        -0.003877
mock_score       1.000000
Name: mock_score, dtype: float64

The numbers match the hand calculation, down to the rounding. A target always correlates perfectly with itself, which is why mock_score shows 1.0 against its own name: a useful sanity check that the code is doing what it claims.

Variance thresholding is just as mechanical. Suppose a third column records every student's attendance as a flat 100%:

import pandas as pd
from sklearn.feature_selection import VarianceThreshold

features = pd.DataFrame({
    "hours_studied":  [5, 8, 3, 10, 6, 12, 4, 9],
    "siblings":       [0, 2, 2, 3, 3, 0, 1, 1],
    "attendance_pct": [100, 100, 100, 100, 100, 100, 100, 100]
})

selector = VarianceThreshold(threshold=0.0)
selector.fit(features)
print(list(features.columns[selector.get_support()]))

['hours_studied', 'siblings']

attendance_pct is dropped automatically, not because attendance never matters, but because a column that never changes cannot explain why one student scored 91 and another scored 50. Notice that this filter kept siblings, even though the correlation check just showed it is useless: variance thresholding only asks whether a feature varies at all, not whether it varies in a way that lines up with the target. Real pipelines chain multiple filters instead of trusting just one.

Wrapper and Embedded Methods: Letting a Model Help Decide

Filter methods are fast, but they judge each feature in isolation. They cannot notice that two features are only powerful in combination, or that one feature stops mattering once a stronger, correlated feature is already in the model. Wrapper methods fix this by actually training models on different feature subsets and comparing results. Recursive feature elimination is the standard example: train a model on every available feature, note which feature contributes the least, remove it, retrain on what remains, and repeat until only the desired number of features is left. It is thorough and it accounts for interactions, but training dozens of models to select from dozens of features is the kind of computational expense a real-time fraud system cannot always afford.

Embedded methods reach a practical middle ground: selection happens as a side effect of training one model, not as a separate stage. Two examples cover most real use:

• Lasso regression (L1 regularization) adds a penalty to the training objective proportional to the sum of the absolute values of the model's weights. Shrinking weights this way (rather than penalizing their squares, as Ridge regression does) has a distinctive effect: it can push unhelpful weights all the way to exactly zero, which removes that feature from the model entirely.
• Tree-based feature importance comes from how decision trees and random forests are built: at each split, the tree picks whichever feature most reduces prediction error, and a feature that is rarely or never picked for a useful split ends up with an importance score near zero.

Running the mock-score data through a small decision tree confirms what the correlation check already suggested:

import pandas as pd
from sklearn.tree import DecisionTreeRegressor

df = pd.DataFrame({
    "hours_studied": [5, 8, 3, 10, 6, 12, 4, 9],
    "siblings":      [0, 2, 2, 3, 3, 0, 1, 1],
    "mock_score":    [58, 72, 50, 88, 65, 91, 55, 80]
})

X = df[["hours_studied", "siblings"]]
y = df["mock_score"]

tree = DecisionTreeRegressor(max_depth=2, random_state=0)
tree.fit(X, y)

for name, importance in zip(X.columns, tree.feature_importances_):
    print(f"{name}: {importance:.3f}")

hours_studied: 1.000
siblings: 0.000

The tree assigns every unit of importance to hours studied and none to siblings. This is a small, tidy dataset built to make the point clearly: hours studied alone already predicts scores almost perfectly, so the tree never needs to split on siblings. It is worth being honest about the limits of this demonstration. With far more irrelevant columns, far less data, or a tree allowed to grow deeper than max_depth=2, a flexible model can and does start finding accidental patterns in pure noise. That is overfitting again, wearing a different disguise. Feature selection has to happen before a model is trusted, not as a formality run once and forgotten.

Back to the Fraud Model

Return to the payments company's data science team, now armed with all three families of technique. A variance-threshold pass removes columns that never change across transactions (the payment protocol field, for instance, when every row in the dataset is a UPI transaction). A correlation pass clears out fields like phone wallpaper color and battery percentage, whose relationship to fraud turns out to be statistically indistinguishable from zero, exactly like siblings and mock scores. What remains goes through an embedded pass, a Lasso-regularized or tree-based model trained on the survivors, which narrows the list further, down to a compact, high-signal set: something like transaction amount relative to the sender's usual amount, time since the payee was first added, and how many prior transactions exist between this sender and receiver.

The rebuilt model, running on perhaps 8 features instead of 46, scores new transactions more accurately. There is no leftover noise for it to overfit to. It also runs fast enough for split-second decisions at national scale, and when an auditor asks why a transaction was blocked, the team can point to two or three specific, defensible signals instead of shrugging at a black box. None of this came from collecting more data. It came from being disciplined about which columns earned a place in the model: the same discipline that turned a shaky guess about siblings and a mock exam score into a number, confirmed twice over, of essentially zero.

Think About It

Think about this: How would you explain feature selection: choosing what matters to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind feature selection: choosing what matters, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.