Dimensionality Reduction: PCA and t-SNE

Open any serious cricket statistics site and click on a batter's profile. You will find a wall of numbers: batting average, strike rate, centuries, fifties, boundary percentage, dot ball percentage, average against pace, average against spin, powerplay strike rate, death overs strike rate, and more. Now imagine you are a data analyst at an IPL franchise, three weeks before the auction, and you have to compare two hundred such players and decide who fits the team's game plan. Nobody can hold fifteen numbers per player in working memory. Nobody can even draw a fifteen dimensional scatter plot, because human eyes only know how to read two or three dimensions at once.

Yet analysts do this every year. They take a spreadsheet with hundreds of rows and dozens of columns and turn it into a single chart where "aggressive finishers" cluster in one corner and "steady anchors" cluster in another, and a scout can see the whole shape of the player pool in one glance. The technique that makes this possible is called dimensionality reduction. Its two most important tools are Principal Component Analysis (PCA), the workhorse of compression and preprocessing, and t-SNE, the specialist built for visualization.

Too many columns to see

In machine learning, each measured quantity in your data, batting average, strike rate, number of sixes, is called a feature. Each feature is also called a dimension, because if you treat every feature as an axis, one data point (one player) becomes a single point floating in that many dimensional space. Two features give you a scatter plot on paper. Three features give you a point floating in a cube, which you can still picture. Fifteen features give you a point in fifteen dimensional space, which no human being can visualize directly, no matter how much they practice.

High dimensional data causes more trouble than just being hard to draw. As the number of features grows, the space those points live in grows explosively, and the data inside it gets thin. Here is a clean way to see why. Suppose you want to place points densely enough to cover a line segment of length ten, say one point every unit: that takes 10 points. Covering a 10 by 10 square at the same density takes 100 points. A 10 by 10 by 10 cube takes 1,000 points. Every extra dimension multiplies the number of points you need by 10. By the time you reach just 10 dimensions, matching that density needs 10 billion points. Real datasets never have that many rows, so in high dimensions your data is always sparse, scattered thinly through a mostly empty space. This effect, where distances start to lose their meaning and models need exponentially more data as dimensions grow, is called the curse of dimensionality.

Dimensionality reduction answers this: build a smaller set of new features, usually two or three, that keep as much of the useful structure of the original data as possible. What counts as "useful structure" can mean two different things, and that difference is exactly what separates PCA from t-SNE.

PCA: rotating your point of view

PCA starts from one idea: variance is information. If every player in your dataset had the exact same strike rate, that column would be useless for telling players apart. A feature only helps you compare players to the extent that it actually varies across them. So the direction in which the data spreads out the most is a reasonable stand in for the direction that carries the most distinguishing information.

PCA takes this literally. Picture your data plotted in two dimensions, sixes on one axis and fifties on the other, forming a cloud of dots that leans diagonally, because bigger hitters of sixes also tend to convert more fifties. PCA looks for a new axis, at any angle, not necessarily matching the original sixes or fifties axes, along which that cloud is most stretched out. That new axis is called the first principal component. It then looks for a second axis, perpendicular to the first, that captures as much of the remaining spread as possible: the second principal component. In two dimensions there is only one perpendicular direction left, so this second step is automatic; in higher dimensions PCA keeps repeating the process, each new axis perpendicular to all the earlier ones, until it has as many new axes as the original data had features.

The payoff is that these new axes come ranked by importance. If the first one or two capture almost all of the spread in the data, you can describe every player with just one or two numbers instead of fifteen, losing very little.

Finding the exact direction of maximum spread by testing every possible angle would take forever. Fortunately, there is a shortcut rooted in linear algebra. For any square matrix, an eigenvector is a special direction that the matrix does not rotate: it only stretches or shrinks vectors pointing that way, and the amount of stretching is called the eigenvalue. It turns out, and this is a genuinely elegant, provable result you will meet in more depth in later years, that the directions of maximum variance in a dataset are exactly the eigenvectors of its covariance matrix, a small table that records how every feature varies with every other feature. The amount of variance captured along each direction equals its eigenvalue. That turns "search over every possible angle" into "solve one matrix equation," arithmetic you already know how to do for a small matrix.

Worked example: compressing two stats into one

Consider four hypothetical T20 players, each described by two stats: sixes hit and fifties scored in a season.

Player   Sixes (X)   Fifties (Y)
  A          2            1
  B          4            3
  C          6            5
  D          8            3

Step 1: Center the data. PCA always starts by subtracting the mean of each column, so the cloud of points is centered on the origin. The mean of X is (2 + 4 + 6 + 8) / 4 = 5, and the mean of Y is (1 + 3 + 5 + 3) / 4 = 3. Subtracting these means gives the centered coordinates:

Player   X - 5   Y - 3
  A        -3      -2
  B        -1       0
  C         1       2
  D         3       0

Step 2: Build the covariance matrix. The variance of X is the average of the squared centered X values: (9 + 1 + 1 + 9) / 4 = 5. The variance of Y is (4 + 0 + 4 + 0) / 4 = 2. The covariance between X and Y, which measures whether they rise and fall together, is the average of their products: ((-3)(-2) + (-1)(0) + (1)(2) + (3)(0)) / 4 = 8 / 4 = 2. Arranged as a matrix, with variances on the diagonal and the covariance on the off diagonal:

        [ 5   2 ]
  C  =  [ 2   2 ]

Step 3: Solve for the eigenvalues. An eigenvalue λ of this matrix satisfies det(C - λI) = 0, where I is the identity matrix. Writing that out:

  (5 - λ)(2 - λ) - (2)(2) = 0
  10 - 5λ - 2λ + λ² - 4 = 0
  λ² - 7λ + 6 = 0

That last line is a quadratic equation, the same kind you already solve in Class 10 mathematics by factoring. It factors cleanly into (λ - 1)(λ - 6) = 0, giving two eigenvalues: λ = 1 and λ = 6. As a check, their sum, 7, equals the sum of the diagonal entries of C (5 + 2), and their product, 6, equals the determinant of C (5×2 - 2×2 = 6). Both checks confirm the arithmetic.

Step 4: Find the eigenvector for the larger eigenvalue. The larger eigenvalue, 6, marks the first principal component, the direction of maximum spread. Its eigenvector v satisfies (C - 6I)v = 0:

  [ -1   2 ] [v1]   [0]
  [  2  -4 ] [v2] = [0]

The first row gives -v1 + 2v2 = 0, so v1 = 2v2. Taking v2 = 1 gives the direction (2, 1). Any centered point (x, y) can now be projected onto this direction by computing 2x + y. Doing that for all four players: A gives 2(-3) + (-2) = -8, B gives 2(-1) + 0 = -2, C gives 2(1) + 2 = 4, and D gives 2(3) + 0 = 6. Already, this single number ranks the players the way you would expect from their raw stats: A lowest, then B, then C, then D highest, capturing a combined sense of "power and consistency" that neither sixes nor fifties alone shows as cleanly.

To make this projection mathematically exact, so that its variance equals the eigenvalue 6 precisely, the direction (2, 1) needs to be rescaled to unit length. Its length is √(2² + 1²) = √5, so the unit eigenvector is (2/√5, 1/√5), approximately (0.894, 0.447). The properly scaled scores become -8/√5 ≈ -3.58, -2/√5 ≈ -0.89, 4/√5 ≈ 1.79, and 6/√5 ≈ 2.68. Squaring these four numbers, summing them, and dividing by 4 gives exactly 6, matching the eigenvalue and confirming the calculation end to end.

The total variance in the original two dimensional data is the sum of both eigenvalues, 6 + 1 = 7, which also equals the sum of the two original variances, 5 + 2 = 7. The first principal component alone accounts for 6 out of 7, or roughly 85.7 percent, of that total. Compressing two stats into one number here throws away only about 14 percent of the spread in the data. Rotate the axes, rank them by how much variance each one explains, keep only the ones that matter: that is PCA, done in miniature.

Verifying it in code

The same four steps, done by a computer instead of by hand, look like this:

import numpy as np

# Sixes and fifties for 4 hypothetical T20 players
data = np.array([
    [2, 1],   # Player A
    [4, 3],   # Player B
    [6, 5],   # Player C
    [8, 3],   # Player D
], dtype=float)

mean = data.mean(axis=0)
centered = data - mean
print("Mean:", mean)                  # [5. 3.]

cov = np.cov(centered.T, bias=True)   # population covariance
print("Covariance matrix:\n", cov)    # [[5. 2.]
                                       #  [2. 2.]]

eigvals, eigvecs = np.linalg.eigh(cov)   # returns ascending order
print("Eigenvalues:", eigvals)           # [1. 6.]

pc1 = eigvecs[:, 1]                      # eigenvector for eigenvalue 6
scores = centered @ pc1
print("PCA scores:", np.round(scores, 2))   # roughly [-3.58 -0.89 1.79 2.68]

np.linalg.eigh is built specifically for symmetric matrices, which every covariance matrix is, and it hands back real eigenvalues in ascending order along with matching unit length eigenvectors, the same (2/√5, 1/√5) direction found by hand. Signs can come out flipped: (-0.894, -0.447) works just as well as (0.894, 0.447), because both point along the same line, just in opposite directions. Nobody builds the covariance matrix and solves for eigenvalues by hand in practice; scikit-learn wraps the whole procedure into three lines:

from sklearn.decomposition import PCA

pca = PCA(n_components=1)
scores = pca.fit_transform(data)                    # centers internally
print(np.round(scores.ravel(), 2))                  # same 4 numbers, signs may flip
print(round(pca.explained_variance_ratio_[0], 3))    # 0.857

explained_variance_ratio_ reports the fraction of total variance each kept component accounts for, matching the 6/7 calculated by hand.

Before you run PCA: scaling and choosing how many components to keep

PCA maximizes variance, which makes it sensitive to the units each feature is measured in. If you added "runs scored" (ranging from single digits into the hundreds) alongside "sixes" (usually single digits) without adjustment, runs would dominate the variance purely because its numbers are larger, not because it is a more important stat. The standard fix is standardization: rescale every feature to have a mean of 0 and a standard deviation of 1 before running PCA, so every column starts on equal footing. scikit-learn's StandardScaler does this in one line and should almost always run before PCA when features are on different scales.

With more than two or three features, analysts often plot the eigenvalues in decreasing order on a bar chart called a scree plot, then look for the point where the bars stop dropping sharply and start leveling off, the "elbow." Components before the elbow are kept; components after it are usually mostly noise. On the classic Iris flower dataset, 150 flowers measured on 4 physical dimensions (sepal length, sepal width, petal length, petal width), the first principal component alone typically captures over 90 percent of the total variance, and the first two together capture more than 95 percent, which is why a 2D PCA plot of Iris flowers separates its three species almost perfectly even though the original data lived in four dimensions.

When straight lines are not enough: enter t-SNE

PCA has one hard limitation: every principal component is a straight line combination of the original features. That works well when the important structure in the data really is linear, stretched along some tilted axis. It works poorly when the structure is curved or made of separate clumps that no straight line can cleanly tell apart. Imagine two groups of players forming crescent shapes wrapped around each other; no straight axis separates them well, no matter how it is rotated, because a straight line cannot bend around a curve.

In 2008, researchers Laurens van der Maaten and Geoffrey Hinton, the same Hinton who later shared the 2024 Nobel Prize in Physics for his foundational work on neural networks, published a paper titled "Visualizing Data using t-SNE," introducing a technique built specifically for this situation: t-distributed Stochastic Neighbor Embedding, or t-SNE. Where PCA asks which straight direction captures the most overall spread, t-SNE asks a different question: which points are each other's neighbors, and how can a 2D map be arranged so that neighbors stay neighbors?

t-SNE works roughly like this. First, for every pair of points in the original high dimensional space, it computes a probability that one point would pick the other as its neighbor, based on distance: nearby points get a high probability, distant points get a low one, using a bell shaped Gaussian curve centered on each point. This turns the entire high dimensional distance structure into a web of neighbor probabilities. Second, t-SNE places the points randomly in a low dimensional space, usually 2D, and defines a similar set of neighbor probabilities there, but using a heavier tailed curve, a Student's t-distribution, which is where the "t" in the name comes from, instead of a Gaussian. Third, it nudges the 2D points, a small step at a time through gradient descent, so the 2D neighbor probabilities match the high dimensional ones as closely as possible, measuring the mismatch with a quantity called KL divergence. After enough of these small steps, points that were close together in the original space end up close together on the final 2D map.

The heavier tailed curve in the low dimensional step solves what the original paper calls the crowding problem. A high dimensional space has enormous volume, so a point can have many moderately distant neighbors without anything feeling cramped. A two dimensional map has far less space to work with, so if the low dimensional step used the same thin tailed Gaussian as the high dimensional step, all those moderately distant neighbors would be squeezed toward the center of the map. The heavy tailed t-distribution treats moderate distances as comparatively more probable, letting points spread further apart. That is why t-SNE plots tend to show clean, visually separated clusters rather than one dense blob.

Using t-SNE in practice

t-SNE has one important tuning knob: perplexity, which roughly controls how many nearby neighbors each point tries to stay close to. Small values focus on tight local structure; larger values consider a broader neighborhood. The original paper suggests values typically between 5 and 50, and most implementations, including scikit-learn, default to 30.

from sklearn.datasets import make_blobs
from sklearn.manifold import TSNE

# 90 synthetic players, 8 stats each, secretly grouped into
# 3 playing styles that make_blobs knows but TSNE is never told
X, true_style = make_blobs(
    n_samples=90, n_features=8, centers=3, random_state=42
)

embedding = TSNE(n_components=2, perplexity=30,
                  random_state=42).fit_transform(X)

print(embedding.shape)   # (90, 2), one (x, y) point per player

Plotting embedding would typically show three visually distinct clumps, even though t-SNE was never told the true groupings, because points from the same hidden group start out as each other's nearest neighbors in the original 8 dimensional space, and preserving exactly that is t-SNE's whole job. A famous real world version of this trick runs on the MNIST dataset of handwritten digit images, where every image is a 28 by 28 grid of pixels, 784 dimensions, per image. Run t-SNE on a few thousand of these images and it typically arranges them into ten neat clusters, one per digit, without ever being told which image shows which number.

t-SNE carries real costs that PCA does not. It is stochastic: running it twice on the same data, even with identical settings apart from the random seed, can produce differently shaped maps, though the clusters they contain usually stay similar. The distances between clusters and the sizes of clusters on a t-SNE plot are not reliable measurements; a tightly packed cluster is not necessarily more internally similar than a loose one, and two clusters drawn far apart are not necessarily less related than two drawn close together. Unlike PCA, which gives a reusable linear formula you can apply to brand new data points, t-SNE has no simple way to place a new point onto an existing map; adding new data usually means rerunning the whole algorithm. And t-SNE is computationally heavier than PCA, so with hundreds or thousands of raw features, a common workflow runs PCA first to cut the data down to perhaps 30 to 50 dimensions, then runs t-SNE on that smaller representation, both for speed and because PCA quietly removes some noise along the way.

PCA or t-SNE: choosing the right tool

Reach for PCA when:

• You need a fast, repeatable transformation you can apply to new data as it arrives.
• You are compressing features before feeding them into another model, to speed up training or reduce overfitting.
• You want to know exactly how much of the original information each new dimension keeps, through explained variance.

Reach for t-SNE when:

• Your only goal is a 2D or 3D picture that a human will look at to spot clusters or patterns.
• You suspect the structure in your data is curved or clumped rather than stretched along straight lines.
• You do not need to place new, unseen points onto the map later.

The two are not rivals. They are frequently used together: PCA first, to clean up and shrink a large feature set, t-SNE second, to turn that cleaned up set into a picture worth looking at.

Back to the scatter plot

Picture that IPL analyst again, now armed with both tools. If the goal is to feed player statistics into a model that predicts auction value, PCA is the right call: it compresses fifteen correlated stats into a handful of components, each with a known, reusable formula, ready to plug into new players as their stats update through the season. If the goal is a single chart for a team of scouts to glance at and immediately see which players play similar roles, t-SNE is the right call: it bends and folds the fifteen dimensional space so that similar players land near each other, even when their similarity has nothing to do with a straight line through the data.

Either way, the underlying job stays the same. A player, a flower, a handwritten digit, or a UPI transaction is really a point sitting in a space with far more dimensions than human eyes were built to read. Dimensionality reduction does not add information that was not there. It picks, honestly and mathematically, which parts of that information are worth keeping when you only have two dimensions of paper, or screen, to show it on.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind dimensionality reduction: pca and t-sne, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.