Building a Complete ML Project: End to End

The sixteenth over has just ended in a tense T20 run chase, the kind of situation IPL viewers see most nights during the season. The team batting second needs 38 runs off the last 24 balls, with six wickets in hand. Before the bowler even starts the run-up, the broadcast graphic updates: Win Probability — Batting Team 38%. Millions of viewers glance at that number and move on, but almost none of them ask the more interesting question: where did 38% actually come from? It was not typed in by a commentator's gut feeling. It came out of a machine learning system, built, tested, deployed, and quietly monitored by an engineering team long before a ball was bowled today. This chapter builds a simplified version of that system, stage by stage — not just training a model, but assembling the pipeline that carries raw data to a live number on a television screen.

From a Trained Model to a Working System

Most ML lessons start and end with the same move: load a dataset, call .fit(), print an accuracy score. That step is real and necessary, but it is a small fraction of what it takes to put a machine learning system in front of actual users, whether those users are broadcast viewers, a bank's fraud desk, or a farmer checking a crop-health app. The industry has a decades-old name for the fuller process: CRISP-DM (Cross-Industry Standard Process for Data Mining), a methodology from the 1990s that breaks a data project into phases — understanding the business problem, understanding the data, preparing it, modelling, evaluating, deploying — that loop back into each other rather than run once in a straight line. Modern ML teams have added a further habit on top of this loop: monitoring what happens after deployment, since a model accurate on launch day can quietly go wrong months later. The running example for this chapter builds one small piece of that loop: a system that predicts, ball by ball, the probability that the team chasing a target in a T20 match will win — the same kind of number that just flashed on screen.

Stage 1: Framing the Problem

Before any code is written, an ML project starts with a translation problem: turning a vague question ("who's going to win this?") into a precise, learnable task. This means answering two questions. First, what exactly are we predicting — a category or a number? Since "will the chasing team win" has exactly two outcomes, this is binary classification, a form of supervised learning where a model learns from historical examples that already carry the correct answer. Every historical match state becomes one training example, described by features — the input measurements, such as runs required, balls remaining, and wickets in hand — paired with a label, the known outcome: 1 if the chasing team won, 0 if they lost. Second, what information will genuinely be available at the exact moment the prediction is needed? During over 16 of a live chase, the required run rate and wickets in hand are known, but the final result is not. Getting this framing wrong, especially letting unavailable information sneak in, is one of the most common ways real ML projects fail before they even start. That mistake has a name, and it deserves a closer look before any data is touched.

Stage 2: Data and the Leakage Trap

A supervised model is only as good as the historical examples it learns from. For a win-probability system, that means ball-by-ball records of past matches: the score, wickets, and overs remaining at every delivery, paired with the eventual result. Data of exactly this shape is not exotic or hard to find — Cricsheet, a free and openly published dataset, has offered ball-by-ball records for thousands of international and franchise matches for years, used by student projects and professional analysts alike. Once such data is collected, the first job is not to touch the model at all — it is to ask a harder question of every column: would this actually be known at the moment the prediction is needed? Imagine including "final_team_score" as a feature to predict "did_the_chasing_team_won." A model will happily learn a near-perfect rule from it — a higher final score almost always means a win — and will report a dazzling training accuracy that is completely useless in production, because during over 16 of a live match the final score does not exist yet. This is data leakage: letting information from the future, or information too tightly correlated with the outcome, sneak into the training features. Leakage is dangerous precisely because it makes a model look better during development, not worse — the failure surfaces only after deployment, when it is expensive to fix. A disciplined ML engineer draws a mental line for every row of data — what did we actually know at exactly this moment? — and drops anything on the wrong side of it.

Stage 3: Cleaning, Splitting, and a Second Kind of Leakage

Real match data is rarely as tidy as a textbook table. A delivery might be missing a recorded venue; a data-entry error might list wickets in hand as 11, which is impossible since a team can only lose 10 wickets; a rain-affected match might have a null target. Data cleaning means finding and handling these systematically: dropping rows that cannot be recovered, filling gaps only where that is defensible, and flagging impossible values rather than silently trusting them. Categorical columns such as venue or "batting first / batting second" cannot go into most models as raw text; they need one-hot encoding, which turns one column of category names into several columns of 0s and 1s, one per category — a single call to pd.get_dummies() in pandas.

Once the data is clean, it must be split into pieces the model is never allowed to mix. The training set is used to fit the model; the test set is used exactly once, at the very end, to report honest performance. Here is match-state data moving through cleaning, feature engineering, and splitting:

import pandas as pd
from sklearn.model_selection import train_test_split

# Load ball-by-ball match state snapshots
df = pd.read_csv("chase_data.csv")

# Drop rows with missing or impossible values
df = df.dropna(subset=["runs_required", "balls_remaining", "wickets_in_hand"])
df = df[df["wickets_in_hand"].between(0, 10)]

# Feature engineering: required run rate (runs needed per over)
df["required_run_rate"] = (df["runs_required"] * 6) / df["balls_remaining"]

# Select features (X) and target (y)
X = df[["required_run_rate", "wickets_in_hand"]]
y = df["chasing_team_won"]

# Split BEFORE any scaling touches the data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"Training rows: {len(X_train)}, Test rows: {len(X_test)}")

Two details matter enormously here. First, stratify=y tells scikit-learn to preserve the ratio of wins to losses in both splits — without it, an unlucky random split could hand the test set mostly wins, making evaluation misleading. Second, the split happens before any scaling. This is a quieter, second kind of leakage: if the mean and standard deviation used for scaling were computed from the full dataset, train and test combined, the test set would secretly influence training, inflating the reported score. The rule is absolute: any statistic used to transform the data, whether a mean, a standard deviation, or a most-frequent category, is learned only from the training set, then applied to the test set unchanged.

Stage 4: Exploratory Data Analysis

Before training anything, a careful practitioner looks at the data. Exploratory Data Analysis (EDA) means checking distributions, checking for class imbalance — are wins and losses roughly balanced, or does one dominate? — and looking for relationships between features and the target. For this dataset, two checks matter most. Win rate plotted against wickets in hand should climb steadily: teams with more wickets standing win more often, matching cricketing common sense and confirming the data was recorded correctly. Win rate plotted against required run rate should fall steadily: the higher the rate needed, the fewer teams reach it. If either check came back flat or reversed, that would be a signal to inspect the data pipeline for bugs before training anything at all. In a real project, far more time is spent looking at data than looking at model output.

Stage 5: Choosing and Training a Model

Why not fit a straight line through the data, plain linear regression, and call anything above 0.5 a "win"? The problem is that linear regression has no ceiling or floor: depending on the feature values, it can output a "probability" of 1.4 or -0.2, both meaningless. What is needed is a function that takes any real number and squeezes it into the range between 0 and 1, so it can be read as a genuine probability. That function is the sigmoid function:

sigmoid(z) = 1 / (1 + e^(-z))

where e is Euler's number, approximately 2.71828. Feed the sigmoid a large positive number and it approaches 1; feed it a large negative number and it approaches 0; feed it exactly 0 and it returns exactly 0.5. Logistic regression is simply a weighted sum of the features, z = b + w1*x1 + w2*x2, passed through this sigmoid. The weights (w) and bias (b) are what training actually learns, by searching for values that make predicted probabilities match true outcomes as closely as possible across the training set. Technically, this means minimizing a loss function called log loss via gradient descent, a topic worth its own chapter but not one that needs hand-deriving to use the model correctly here. In code:

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Scale features: subtract the mean, divide by the standard deviation
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # learn AND apply on train
X_test_scaled = scaler.transform(X_test)          # apply only, using train's stats

# Train a logistic regression classifier
model = LogisticRegression(C=1.0, max_iter=1000)
model.fit(X_train_scaled, y_train)

# Evaluate on the held-out test set
y_pred = model.predict(X_test_scaled)
print("Test accuracy:", accuracy_score(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

Why scale the features at all? Required run rate typically ranges from around 4 to 15, while wickets in hand ranges from 0 to 10 — not wildly different here, but in general, features on very different scales can make training converge slowly or give one feature outsized influence simply because its raw numbers are bigger. Standardization — subtracting the mean and dividing by the standard deviation, exactly what StandardScaler does — puts every feature on comparable footing before training begins.

Worked Example: What the Model Actually Computes

Suppose training converges to these parameters (illustrative values, chosen to keep the arithmetic easy to follow by hand): an intercept b = 0.2, a weight w1 = -1.2 on the scaled required run rate, and a weight w2 = 0.4 on scaled wickets in hand. Suppose the scaler learned, from the training data, a mean required run rate of 8.0 with a standard deviation of 2.0, and a mean wickets-in-hand of 5.0 with a standard deviation of 2.0.

Now the exact situation from the opening of this chapter arrives: 38 runs needed off 24 balls, six wickets in hand. That is a required run rate of 38*6/24 = 9.5 runs per over. Here is exactly what model.predict_proba() computes, one step at a time.

Step 1 — scale the raw inputs, using the training set's stored mean and standard deviation, never statistics from the new point itself:

• Scaled required run rate: (9.5 - 8.0) / 2.0 = 1.5 / 2.0 = 0.75
• Scaled wickets in hand: (6 - 5.0) / 2.0 = 1.0 / 2.0 = 0.5

Step 2 — compute the weighted sum, z:

z = b + w1*x1 + w2*x2
z = 0.2 + (-1.2 * 0.75) + (0.4 * 0.5)
z = 0.2 - 0.9 + 0.2
z = -0.5

Step 3 — pass z through the sigmoid function:

sigmoid(-0.5) = 1 / (1 + e^0.5)
             = 1 / (1 + 1.6487)     [e^0.5, the square root of e, is approximately 1.6487]
             = 1 / 2.6487
             = approximately 0.378

Step 4 — interpret the result. The model outputs a win probability of approximately 0.378, or 37.8%, which a broadcast graphic would round to the 38% shown at the start of this chapter. Since scikit-learn's .predict() applies a default threshold of 0.5, this situation would be classified as class 0, "chasing team predicted to lose." But notice what would be lost if the system only displayed that binary label: a 37.8% chance is a real, meaningful, "tough but very much alive" chase, not a hopeless one. This is exactly why the broadcast shows a percentage rather than a flat "WIN" or "LOSE" — .predict_proba(), not .predict(), is the method that actually powers a live win-probability display. It also works as a sanity check on the model itself: a required rate of 9.5 an over with six wickets standing is genuinely difficult but very much gettable in T20 cricket, so a probability in the high 30s feels believable, not absurd. Hand-tracing a model's arithmetic on a single example like this is one of the most reliable ways to catch a bug, a flipped sign on a weight, a feature scaled with the wrong mean, before it ever reaches production.

Stage 6: Evaluation — Why Accuracy Alone Can Mislead

The classification_report call above prints several numbers at once, and they are worth computing by hand once so they stop being magic. Suppose the trained model is run against 10 held-out test situations:

• 5 situations where the chasing team actually won: the model correctly predicted "win" for 3 of them, and incorrectly predicted "lose" for the other 2.
• 5 situations where the chasing team actually lost: the model correctly predicted "lose" for 4 of them, and incorrectly predicted "win" for the other 1.

These four counts are arranged in what is called a confusion matrix: 3 true positives (predicted win, actually won), 2 false negatives (predicted lose, actually won), 1 false positive (predicted win, actually lost), and 4 true negatives (predicted lose, actually lost). Every classification metric that matters is built from just these four numbers.

Accuracy, the fraction of all predictions that were correct:

accuracy = (TP + TN) / total = (3 + 4) / 10 = 0.70

Precision, of every situation predicted as a "win," what fraction actually were wins:

precision = TP / (TP + FP) = 3 / (3 + 1) = 3 / 4 = 0.75

Recall, of every situation that was actually a win, what fraction the model caught:

recall = TP / (TP + FN) = 3 / (3 + 2) = 3 / 5 = 0.60

F1-score, the harmonic mean of precision and recall, a single number that punishes a model for being lopsided:

F1 = 2 * (precision * recall) / (precision + recall)
   = 2 * (0.75 * 0.60) / (0.75 + 0.60)
   = 0.90 / 1.35
   = approximately 0.667

Notice that all four numbers differ: 70% accuracy, 75% precision, 60% recall, 66.7% F1. This is exactly why accuracy alone is a dangerously incomplete report card, especially when the two classes are not perfectly balanced or when the two kinds of mistake carry different weight. A recall of 60% means the model misses 2 out of every 5 actual winning chases, calling a team out of contention when they are, in fact, about to pull it off. Whether that matters more than the false positive depends on the system's purpose: a broadcast graphic that is occasionally too pessimistic is a minor annoyance, but the same 60% recall in a system flagging suspicious UPI transactions means 2 out of every 5 real frauds slipping through uncaught — a very different order of consequence. Choosing which metric to optimize is a decision about the real world, not a technical detail to skip.

Stage 7: Tuning Without Cheating

Trying different settings for a hyperparameter like C by repeatedly checking against the test set is a subtle trap: the moment the test set starts influencing decisions, it stops being a fair, final judge, and whatever setting looks best on it will not hold up once genuinely new data arrives. The standard fix is k-fold cross-validation: the training data is split into k equal chunks (commonly 5), the model is trained k times, each time holding out a different chunk to validate on, and the k scores are averaged — a stable estimate using only training data, leaving the test set untouched until the project's very last step. scikit-learn automates this search:

from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV

# Bundle preprocessing and model together so they always travel as one unit
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000))
])

param_grid = {"classifier__C": [0.01, 0.1, 1, 10, 100]}
grid_search = GridSearchCV(pipeline, param_grid, cv=5, scoring="roc_auc")
grid_search.fit(X_train, y_train)

print("Best C:", grid_search.best_params_)
best_model = grid_search.best_estimator_

C controls regularization strength, as its inverse: a small C (like 0.01) keeps weights small and simple, guarding against overfitting — a model memorizing quirks of the training data instead of the general pattern — while a large C (like 100) fits the training data more aggressively, risking that same overfitting if pushed too far. Setting scoring="roc_auc" tunes toward the ROC-AUC metric, which measures how well the model ranks true wins above true losses across every possible threshold at once, rather than being locked to a single 0.5 cutoff: 0.5 means no better than a coin flip, and 1.0 means perfect separation. Notice too that scaler and classifier are now bundled into one Pipeline object, a production habit worth building early, since it guarantees the exact scaling statistics learned on training data travel automatically with the model, closing off a whole category of subtle train-serve mismatches.

Stage 8: Deployment — Leaving the Notebook

A trained model sitting in a notebook has helped nobody yet. To power a live broadcast graphic, it needs to be saved to disk and wrapped in a service that another program, such as the broadcaster's graphics engine updating after every ball, can call in real time. Saving the model uses serialization:

import joblib

joblib.dump(best_model, "win_predictor_pipeline.pkl")

Serving it commonly means wrapping the loaded model in a small web API. Using FastAPI, a popular Python framework for exactly this purpose, a minimal prediction service looks like this:

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI()
pipeline = joblib.load("win_predictor_pipeline.pkl")

class MatchState(BaseModel):
    required_run_rate: float
    wickets_in_hand: int

@app.post("/predict")
def predict(state: MatchState):
    features = np.array([[state.required_run_rate, state.wickets_in_hand]])
    probability = pipeline.predict_proba(features)[0][1]
    return {"win_probability": round(float(probability), 3)}

Every ball, the broadcaster's system would send a small request, such as {"required_run_rate": 9.5, "wickets_in_hand": 6}, to this /predict endpoint and receive back {"win_probability": 0.378} in well under a second, ready to render on screen. This is the moment an ML project stops being a data science exercise and becomes software engineering: the API has to handle malformed requests, respond fast enough for live television, and stay running through a three-hour match without crashing.

Stage 9: Monitoring — The Work After Launch

A model's accuracy on launch day is not a permanent property; it is a measurement that can decay. T20 batting has changed shape over the years — bigger bats, shorter boundaries at some grounds, and rule changes such as the IPL's Impact Player rule, introduced in the 2023 season, which lets a team field an extra specialist batter or bowler as a substitute and has pushed teams to bat more aggressively through the middle overs. A win-probability model trained only on matches from several seasons ago will quietly start making worse calls as the sport evolves, not because the code broke, but because the real-world relationship between required run rate and winning chances shifted underneath it. This is called concept drift. Its close cousin, data drift, is when incoming feature values themselves start looking statistically different from training even though the underlying relationship hasn't changed. Both are why serious ML systems log every live prediction, periodically compare recent accuracy against the launch-day baseline, and schedule retraining on fresh data rather than trusting a model trained once to stay correct forever. This ongoing discipline — versioning models, tracking live performance, retraining on schedule — is what the industry calls MLOps, and it is why a production ML system looks less like a single script and more like infrastructure that must be maintained the way any other running service is.

Building Responsibly: The Stakes Change, the Process Doesn't

A win-probability graphic that is occasionally wrong is low-stakes entertainment. But the same process, frame, collect, clean, engineer, train, evaluate, tune, deploy, monitor, sits underneath systems making far higher-stakes calls: a bank's model freezing suspected UPI fraud, an insurer's model pricing a policy, a hospital's model flagging a scan for urgent review. The same failure modes reappear here, with sharper consequences. Data leakage in a fraud model might mean it looks close to perfect in testing and then misses real fraud patterns once deployed. Training data that under-represents certain regions, income groups, or languages can leave a model less accurate for some groups than others — not from intended unfairness, but because nobody checked. This is why responsible ML practice treats evaluation as more than a single accuracy number — asking whether performance holds up across subgroups, not just on average — and means being honest with users about what a probability actually means. A 38% win chance is a genuine possibility, not a certain defeat, and a fraud model's flagged transaction is a signal for a human to review, not an automatic verdict. The pipeline built here is powerful precisely because it generalizes to almost any prediction problem, which is exactly why building it carefully, and questioning it at every stage, matters.

Back to over 16, and the number on the broadcast: 38%. That number is really a computed probability of 0.378, rounded by the graphic to a whole number, and it now stands for something specific rather than something mysterious: a carefully framed prediction problem, ball-by-ball data checked for leaks, features engineered and scaled with discipline, a logistic regression trained and cross-validated rather than eyeballed, a confusion matrix read for more than one number, a model wrapped in an API fast enough for live television, and a monitoring system watching for the day the sport changes enough to need retraining. None of that machinery is specific to cricket. Swap "required run rate" and "wickets in hand" for "transaction amount" and "device location," and the same nine stages build a fraud detector. Swap them for "soil moisture" and "days since sowing," and they build a crop-yield estimator for a farmer's phone. The model itself is the easy part; the pipeline around it, the part this chapter actually built, is the rest — and it is the same pipeline every time.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind building a complete ml project: end to end, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.