MLOps: From Notebook to Production

A 94% AUC Model, and a Company That Lost Money Anyway

A payments company building a UPI fraud-detection model trains a gradient-boosted classifier in a Jupyter notebook. The held-out test AUC is 0.94 — a strong number by any standard for transaction fraud, where positive examples are rare and noisy. The team exports the trained model as a pickle file, wraps it in a Flask endpoint, and ships it. Within the first week, two things go wrong simultaneously: fraud losses barely move, and the customer support queue fills up with users whose completely legitimate ₹200 grocery payments are being blocked.

The model file is untouched — it is the exact same weights that scored 0.94 in the notebook. The bug is not in the model. It is in the twelve lines of feature-computation code sitting around the model, and nobody wrote a test for those twelve lines, because in the notebook they didn't exist as a separate artifact at all — they were just cells that ran once, on a static CSV, in an order nobody recorded.

This is the gap MLOps exists to close: the distance between "a model that scores well on a fixed, offline dataset" and "a system that keeps scoring well while data, code, and the world keep changing underneath it." The rest of this chapter builds that system piece by piece, and returns to this exact fraud scenario for a fully worked numerical diagnosis.

Why a Notebook Cannot Be the Production System

Software engineering solved the problem of "code that works on my machine" with CI/CD: every change to code is automatically built, tested, and deployed through a repeatable pipeline. Machine learning has the same problem, plus two additional moving parts that classical CI/CD was never designed for: data and the model artifact it produces. A notebook conflates all three — code, data loading, and model training — into one mutable, order-dependent execution state, which breaks the three properties a production system needs.

Reproducibility. A notebook's output depends on which cells were run, in what order, with which in-memory variables left over from an earlier experiment. Two data scientists running "the same" notebook a week apart, against a data warehouse table that has since grown by ten thousand rows, will not get the same model. Production requires that a specific model artifact be traceable to an exact, immutable (code version, data snapshot, hyperparameter set) triple — this is what a training pipeline plus a data/model versioning system gives you and a notebook does not.

Isolation of the "small model, large system" problem. D. Sculley et al., in the widely cited paper "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015), make the observation that the actual machine-learning code in a production system is typically a small box surrounded by a much larger mass of glue code, data verification, feature extraction, serving infrastructure, and monitoring — and that this surrounding mass is where technical debt accumulates fastest. They name the CACE principle — "Changing Anything Changes Everything" — to describe how ML systems break the modularity assumptions ordinary software relies on: because a model implicitly learns correlations between features, adding, removing, or subtly altering the computation of any one feature can silently shift the meaning of every other feature the model was trained against. A notebook, with no test suite and no isolation between cells, is the worst possible environment to catch this.

Training–serving parity. The features a model sees at training time must be computed by the identical logic it will see at inference time. A notebook computes features once, in batch, against a static table. Production inference typically computes features in a completely different code path — often a different language, a different service, under millisecond latency constraints — and that second implementation is where the fraud-detection bug above actually lived. This single failure mode is common enough that it has a name, training–serving skew, and it is the misconception this chapter corrects explicitly below.

The MLOps Pipeline: CI, CD, and CT

MLOps takes the CI/CD discipline and adds a third loop that classical software does not need: CT, continuous training. The three loops are:

CI (continuous integration). Every change to feature code, training code, or a tracked data source triggers an automated pipeline: schema and statistical validation on the incoming data, unit tests on the feature and training code, a full retraining run, and an evaluation of the resulting model against a fixed validation slice and against the current production model's metrics on that same slice. A change only merges if it passes every gate — this is the ML analogue of "tests must pass before merge," except one of the "tests" is itself a multi-minute training job.

CD (continuous delivery/deployment). A model that passes CI produces a versioned artifact, pushed into a model registry along with its lineage — which code commit, which data snapshot, which hyperparameters, and its evaluation metrics. Deployment then follows a controlled rollout strategy rather than an instant full replacement:

Shadow deployment: the candidate model receives a copy of live production traffic and computes predictions, but those predictions are logged only — never shown to a user or acted on. This validates real-world latency and the shape of the prediction distribution with zero user-facing risk, and it is usually the first gate a new model passes after the registry.

Canary deployment: the candidate model is given a small, real slice of traffic — commonly 1–5% — whose users actually receive its decisions. Its metrics (business KPIs, not just offline accuracy) are compared statistically against the incumbent model serving the other 95–99%. Only if the canary's metrics hold up does traffic ramp further. This bounds the "blast radius" of a bad model to a small, monitored fraction of real users instead of everyone.

Blue-green deployment: two complete production environments exist simultaneously — the currently live one ("blue") and the new one ("green") — and traffic is switched atomically from one to the other. Its advantage is instant rollback (switch back to blue); its disadvantage is that it offers no partial, gradual validation on real traffic the way canary does, which is why the two are often combined: blue-green for the infrastructure cutover, canary-style traffic ramping for how much of that cutover is trusted at each moment.

CT (continuous training). Unlike ordinary software, a correctly deployed ML model degrades on its own, because the real world keeps moving while the model's learned parameters stay fixed. CT is the automated loop that retrains the model — on a schedule, or triggered by a detected drift signal from production monitoring — and feeds the result back into CI as a new candidate, without a human re-running a notebook by hand. This closes the pipeline into a loop rather than a one-way pipe, which is exactly what the diagram below depicts.

Feature Stores, and Why the Same Feature Name Can Mean Two Different Numbers

A feature store exists to solve one specific problem: the same named feature must be computed by the same logic wherever it is used, whether that is a batch training job reading a year of history or a live service answering in milliseconds. It has two serving surfaces built on one shared definition — an offline store, which serves large, point-in-time-correct historical batches for training, and an online store, which serves the current value of each feature with single-digit-millisecond latency for a live prediction request.

Here is exactly how "the same feature" diverges when that shared-definition contract is missing — this is the actual bug behind the fraud-detection scenario at the top of this chapter. Suppose the feature is named avg_txn_amount_7d. The training pipeline computes it as a 7-calendar-day rolling mean of each day's total transaction amount, including zero-transaction days:

def training_time_feature(daily_totals_last_7_days):
    # daily_totals_last_7_days: one entry per calendar day, 0 if no transaction that day
    return sum(daily_totals_last_7_days) / 7

daily_totals = [500, 1500, 2500, 0, 0, 0, 0]   # this user transacted on only 3 of the last 7 days
print(training_time_feature(daily_totals))
# -> 642.857142857...

The serving microservice, written independently by a different team under latency pressure, instead averages the last 7 transactions, regardless of how many calendar days they span:

def serving_time_feature(last_n_transaction_amounts):
    return sum(last_n_transaction_amounts) / len(last_n_transaction_amounts)

recent_txns = [500, 1500, 2500]   # this user has only 3 transactions on record, total
print(serving_time_feature(recent_txns))
# -> 1500.0

Both functions are correct code. Both are computing something a reasonable engineer would call "average transaction amount, last 7." But 642.86 ≠ 1500.0 — a factor of 1500.0 / 642.857142857 ≈ 2.33×. The model was trained to associate fraud risk with values near 642; at inference it is being handed values systematically 2.33 times larger for exactly the low-activity users most likely to look unusual, which pushes their risk score up and produces the false-positive blocks the support queue was flooded with. Nothing crashed. No exception was thrown. The model, the pickle file, the AUC of 0.94 — all of it was innocent. A feature store closes this gap by making the transformation logic itself the single artifact both the offline and online paths call, rather than two independent reimplementations of a name.

Common Misconception: "If the Model File Is the Same, the Predictions Will Match"

Students (and, empirically, many engineers on their first production ML system) assume that once a trained model's weights are loaded, identically, in both the offline evaluation script and the production server, offline accuracy is a guarantee of online accuracy. The worked example above is the direct counter-proof: the model artifact was byte-for-byte identical in both places, and the predictions still diverged, because the divergence lived entirely in the feature-computation code surrounding the model, not in the model itself. The correct mental model is that a deployed ML system's behavior is determined by the model plus its entire feature pipeline, and both halves need independent verification: offline accuracy tests the model against features computed one way; a separate, explicit online/offline feature parity test — replaying the same historical row through both the batch and the real-time feature code and asserting the outputs match within tolerance — is the only thing that actually catches training–serving skew before it reaches users.

Monitoring in Production: A Fully Worked Drift Calculation

Once deployed, a model cannot be monitored by "accuracy" directly in most real settings, because ground truth often arrives late (a fraud case may be confirmed only after a chargeback investigation weeks later) or never (undetected fraud has no label at all). Instead, production MLOps monitors data drift — has the distribution of an input feature shifted from what the model was trained on? — as an early-warning proxy for accuracy degradation, using a metric such as the Population Stability Index (PSI), borrowed from credit-risk modelling.

PSI compares a feature's distribution across the same bins in two datasets — the training ("expected") distribution and a recent production ("actual") window — using:

PSI = Σ (actual_i - expected_i) × ln(actual_i / expected_i)

summed over every bin i. Take the fraud model's transaction_amount feature, bucketed into five bins (₹0–500, ₹500–2,000, ₹2,000–10,000, ₹10,000–50,000, ₹50,000+). At training time the bin proportions were:

expected = [0.40, 0.30, 0.20, 0.08, 0.02]   # sums to 1.00

A week after deployment, the live traffic's bin proportions have shifted (larger transactions have become more common — plausibly a genuine shift in user behaviour, or a sign the fraud pattern itself has moved to bigger tickets):

actual   = [0.25, 0.25, 0.25, 0.15, 0.10]   # sums to 1.00

Computing each bin's contribution by hand, bin by bin:

bin 1: (0.25-0.40) × ln(0.25/0.40) = -0.15 × ln(0.625)  = -0.15 × (-0.470004) = 0.070501
bin 2: (0.25-0.30) × ln(0.25/0.30) = -0.05 × ln(0.8333) = -0.05 × (-0.182322) = 0.009116
bin 3: (0.25-0.20) × ln(0.25/0.20) =  0.05 × ln(1.25)   =  0.05 × ( 0.223144) = 0.011157
bin 4: (0.15-0.08) × ln(0.15/0.08) =  0.07 × ln(1.875)  =  0.07 × ( 0.628609) = 0.044003
bin 5: (0.10-0.02) × ln(0.10/0.02) =  0.08 × ln(5)      =  0.08 × ( 1.609438) = 0.128755

PSI = 0.070501 + 0.009116 + 0.011157 + 0.044003 + 0.128755 = 0.263531

This is confirmed by direct computation:

import math

def compute_psi(expected, actual):
    return sum((a - e) * math.log(a / e) for e, a in zip(expected, actual))

expected = [0.40, 0.30, 0.20, 0.08, 0.02]
actual   = [0.25, 0.25, 0.25, 0.15, 0.10]
print(compute_psi(expected, actual))
# -> 0.2635314389465628

The standard industry interpretation bands are PSI < 0.10 (no meaningful shift), 0.10–0.25 (moderate shift, investigate), and PSI > 0.25 (significant shift). At 0.2635, this feature has crossed into the "significant" band — this is precisely the alert the monitoring box in the diagram above raises, and precisely the signal that triggers continuous training (CT): the training pipeline is invoked automatically on fresh data before a human ever notices fraud losses moving.

Notice which bin dominated: bin 5 alone contributed 0.128755, roughly half the total PSI, despite representing only an 8-percentage-point absolute shift — smaller in absolute terms than bin 1's 15-point shift. This is the logarithmic term at work: a proportion moving from a small expected baseline (2%) to five times its own size (10%) produces a large ln(actual/expected) even though the raw percentage-point movement is modest. PSI is deliberately more sensitive to relative change in rare bins than to absolute change in common ones, because rare-bin behaviour (very large or very unusual transactions, in this case) is often exactly where fraud patterns first show up.

Active Recall

Attempt every question before reading its answer.

Q1. A team deploys a model, confirms the model file's checksum matches exactly what was evaluated offline, and still sees degraded production performance. Give the most likely explanation and name it.

Q2. Distinguish shadow deployment from canary deployment along one specific axis: does either one affect what a real user actually experiences?

Q3. What does the "CT" in "CI/CD/CT" stand for, and why does traditional (non-ML) software CI/CD have no equivalent of it?

Q4. A separate feature, device_country, shows expected = [0.90, 0.10] (India, other) and actual = [0.70, 0.30] in the latest production window. Compute its PSI by hand and state whether it crosses the 0.25 significant-shift threshold.

Q5. Referring to the worked transaction_amount PSI example: suppose an analyst discovers that the production data pipeline had been misclassifying some ₹10,000–50,000 transactions into the ₹50,000+ bin, and the corrected actual distribution is [0.25, 0.25, 0.25, 0.21, 0.04] instead of [0.25, 0.25, 0.25, 0.15, 0.10]. Recompute PSI with the corrected numbers. Does it still cross the 0.25 threshold? If the team had also tightened their alert threshold from 0.25 to 0.15, would that change which of the two actual distributions (original vs. corrected) triggers an alert?

Q6. Why is a "feature store" the correct fix for training–serving skew, rather than simply telling both engineering teams to "be more careful"?

A1. Training–serving skew. An identical model artifact can still receive systematically different feature values at inference time than it saw during training, because the feature-computation code is a separate artifact from the model and is frequently reimplemented independently for the low-latency serving path. Checksumming the model file only verifies half the system.

A2. Shadow deployment computes predictions on live traffic but never surfaces them to a user or lets them affect a decision — it is pure observation, zero user-facing risk. Canary deployment lets the new model's predictions actually drive real decisions, but only for a small percentage of traffic — real risk, but deliberately bounded ("blast-radius limited") rather than eliminated.

A3. Continuous training — automatically retraining the model (on a schedule or triggered by a drift signal) and feeding the result back into the CI evaluation gate without manual intervention. Traditional software has no analogue because ordinary code does not degrade on its own as the world changes; a sorting algorithm that was correct last year is still correct today. A model's learned statistical relationship to the world can become stale purely from the world changing, even with zero code changes — which is why the ML pipeline needs a loop that software CI/CD does not.

A4. (0.70-0.90)×ln(0.70/0.90) + (0.30-0.10)×ln(0.30/0.10) = (-0.20)×ln(0.7778) + (0.20)×ln(3.0) = (-0.20)×(-0.2513) + (0.20)×(1.0986) = 0.05027 + 0.21972 = 0.26999 ≈ 0.270. This exceeds 0.25, so it also crosses into the significant-shift band — a plausible real signal that fraud traffic is shifting toward non-Indian device origins, worth investigating alongside the transaction-amount drift.

A5. Recomputing with actual = [0.25, 0.25, 0.25, 0.21, 0.04]: bins 1–3 are unchanged from before (0.070501 + 0.009116 + 0.011157). Bin 4 becomes (0.21-0.08)×ln(0.21/0.08) = 0.13×ln(2.625) = 0.13×0.965081 = 0.125461. Bin 5 becomes (0.04-0.02)×ln(0.04/0.02) = 0.02×ln(2) = 0.02×0.693147 = 0.013863. Total: 0.070501+0.009116+0.011157+0.125461+0.013863 = 0.230098. This is below 0.25 — the corrected data no longer triggers the original "significant" alert, only the "moderate, investigate" band, even though bin 4's shift got larger, because bin 5's contribution shrank far more (it was the dominant term before, contributing 0.1288 alone). The ripple is not intuitive from eyeballing percentage-point changes: PSI is dominated by whichever bin has the largest relative change against a small expected baseline, so moving mass from an extreme bin (5) into a less extreme one (4) can lower total PSI even while individual absolute shifts get larger elsewhere. Under a tightened 0.15 threshold, both the original (0.2635) and corrected (0.2301) distributions still trigger an alert — tightening the threshold makes the correction's effect disappear from the pass/fail decision, illustrating why the threshold itself is a tuned operating parameter, not a fixed constant.

A6. "Be more careful" is not a mechanism — it does not survive a new engineer joining either team, a deadline, or a refactor six months later, and it leaves two independent implementations that can silently diverge with no test to catch it, exactly as in the worked avg_txn_amount_7d example. A feature store makes the transformation logic itself a single shared, versioned artifact that both the offline (training) and online (serving) paths call — so there is structurally only one implementation to keep correct, not two to keep synchronized by discipline alone, and skew becomes something a parity test can mechanically assert against rather than something that depends on nobody making a mistake.

Think About It

Think about this: How would you explain mlops: from notebook to production to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where mlops: from notebook to production is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting mlops: from notebook to production to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind mlops: from notebook to production, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.