MLOps: CI/CD for Machine Learning

In October 2023, a fraud-detection model at a large UPI payment processor had been live for six months. Every software metric said it was healthy: the deployment pipeline was green, unit tests passed on every commit, the API served predictions with sub-50ms latency, and nobody had touched the training code since launch. Yet fraud losses were creeping up, week over week, with no corresponding alert firing anywhere. The engineers eventually traced it to something no software test could have caught: the distribution of transaction amounts flowing through the app had shifted — UPI Lite had rolled out for small offline payments, a festival season had pulled in a burst of high-value transfers, and a new class of scam had emerged that the training data, frozen in April, had never seen. The code was unchanged. The model was stale. Nothing in a conventional CI/CD pipeline is built to notice that, because conventional CI/CD only reacts to code changes — and here, nothing in the code had changed at all.

That gap is exactly what MLOps CI/CD exists to close. This chapter builds the pipeline that would have caught the UPI fraud model's decay automatically, derives the statistics that decide when a new model is actually better than the old one, and traces, step by step, how a drift signal turns into an automatic retrain.

Why software CI/CD is necessary but not sufficient

Continuous Integration and Continuous Delivery, as taught for conventional software, automate two questions: does this code change break existing behaviour (CI, answered by running a test suite on every commit), and can this code change be shipped safely (CD, answered by packaging, staging, and progressively rolling out a build). Both questions are fully answered by the code. If the tests pass and the build deploys cleanly, the software is trusted to keep working until someone changes the code again.

A machine learning system breaks that assumption at its foundation. A trained model is not a function written by a programmer — it is a function fit to a dataset, and its correctness depends on three things simultaneously: the code (feature transforms, training loop, serving logic), the data (the exact snapshot of examples it was trained and validated on), and the resulting parameters (weights, thresholds, hyperparameters). Change any one of the three and you have a different system, even if the other two are byte-identical to yesterday's release. Software CI/CD only watches one of those three axes. MLOps extends the discipline to all three, and adds a fourth pillar that has no software analogue at all: Continuous Training (CT) — the automated re-fitting of the model, triggered not by a developer's push but by a monitored signal that the world has moved.

D. Sculley and colleagues at Google formalised why this matters in their widely cited 2015 NeurIPS paper, "Hidden Technical Debt in Machine Learning Systems." Their central argument is that ML systems have a form of technical debt beyond ordinary code complexity, which they call CACE — Changing Anything Changes Everything. Because a model's parameters are jointly fit across every input feature, a change to one upstream data source, one feature's null-handling rule, or even the sampling order of training examples can silently shift the model's behaviour on features that were never touched. Sculley et al. use this to argue that entangled ML systems need far more monitoring and far stricter versioning discipline than their code volume alone would suggest — which is precisely the justification for treating data and hyperparameters as first-class, versioned inputs to the pipeline, not incidental details left in a notebook.

The pipeline, end to end

The diagram below is the release train that would have caught the UPI fraud model's drift. Three inputs — the code repository, a versioned data snapshot, and a shared feature store — feed a CI stage that checks schemas and runs unit tests on every commit or on a nightly schedule. That triggers the training pipeline, which logs exactly which code commit, which data version, and which hyperparameters produced this run, so that a later engineer (or the pipeline itself) can reproduce it byte-for-byte. The resulting artifact — weights plus its full provenance — is written to a model registry. A validation gate then compares the candidate against the model currently in production, using a statistical test rather than a bare "did loss go down." Only a model that clears the gate proceeds to a canary rollout; one that fails is rejected and the old model keeps serving while an engineer is paged. Production traffic is logged, monitored continuously for drift, and if the drift crosses a threshold, that monitoring stage reaches back and triggers the training pipeline again — closing the loop without a human ever filing a ticket.

The validation gate cannot be a bare number comparison

Notice the gate in the middle of the diagram does not simply check "is the new accuracy higher than the old one." On a holdout set of any realistic size, two models trained on nearly the same data will differ in accuracy by a fraction of a percent purely from randomness in which examples ended up in the split. A gate that promotes on any positive delta will eventually promote a strictly worse model by chance, and — worse — it gives no protection against a genuinely harmful regression that happens to land within noise. The gate has to ask a sharper question: is the observed difference larger than we would expect from sampling noise alone? That is a hypothesis test, and the same one your statistics unit already gave you: a test of one proportion (or two proportions) against a null hypothesis of no real difference. The canary-rollout worked example later in this chapter derives exactly that test with real numbers.

Detecting drift: the Population Stability Index, fully derived

The signal that reaches back and re-triggers training in the diagram above has to be a number, computed automatically, that quantifies "how different does today's input data look from what the model was trained on." The most widely used such measure in production ML — borrowed originally from credit-risk scoring, where regulators required a standard way to certify that a scoring model's population hadn't shifted — is the Population Stability Index (PSI).

PSI compares two distributions of the same feature, binned identically: the expected distribution (the reference — usually the training set) and the actual distribution (a recent window of production traffic). For each bin i, with expected proportion E_i and actual proportion A_i:

PSI = Σᵢ (Aᵢ − Eᵢ) · ln(Aᵢ / Eᵢ)

Take the UPI fraud model's transaction-amount feature, bucketed into five bins at training time and re-measured on this week's traffic:

Summing the last column: 0.02043 + 0.02877 + 0.00912 + 0.05108 + 0.09267 = 0.2021. The code below computes exactly this, term by term, and prints the same value:

import math

def population_stability_index(expected, actual):
    """expected, actual: lists of bin proportions, each summing to 1.0."""
    psi = 0.0
    for e, a in zip(expected, actual):
        psi += (a - e) * math.log(a / e)
    return psi

# Reference (training-time) distribution of transaction-amount buckets
expected = [0.10, 0.40, 0.30, 0.15, 0.05]

# Distribution observed in production traffic this week
actual   = [0.06, 0.30, 0.25, 0.25, 0.14]

psi = population_stability_index(expected, actual)
print(round(psi, 4))
# 0.2021

Industry practice on PSI (used this way since its origins in credit scoring, and carried into ML monitoring largely unchanged) applies three bands: PSI < 0.10 means no meaningful shift; 0.10 ≤ PSI < 0.25 means a moderate shift worth watching but not yet acting on; PSI ≥ 0.25 means a significant shift that should trigger action. At 0.2021, the UPI fraud model's amount feature sits in the "watch" band — the monitoring dashboard would show an amber flag, not yet a red one. That is the exact state the model was actually in before the incident described at the top of this chapter: drifting, visibly, for weeks, with no automated gate wired to act on it. The active recall section below traces what happens to this same feature one more billing cycle later, when the shift keeps moving in the same direction.

Canary rollout as a live hypothesis test

Passing the offline validation gate only proves the new model beat the old one on a fixed, already-collected holdout set. It says nothing about live traffic, where the input distribution can differ subtly from any holdout snapshot and where a bug in the serving path (not the model) can also hurt performance. That is what the canary stage in the diagram is for: instead of switching 100% of traffic at once, the new model is given a small slice — commonly 5% — and its live metrics are compared against the 95% still served by the incumbent model, in real time, before ramping further.

Suppose the fraud model's canary is live at 5% of traffic, processing 2,000 transactions an hour, and the false-negative rate (frauds the model misses) on that slice reads 0.9%, against a control baseline of 0.4% established over the model's prior months in production. Is that a real regression, or noise from a small sample? This is a one-sample test of a proportion against a known baseline rate p₀ = 0.004, with observed rate p̂ = 0.009 and n = 2,000:

z = (p̂ − p₀) / √( p₀(1 − p₀) / n )

import math
p0, p1, n = 0.004, 0.009, 2000
se = math.sqrt(p0 * (1 - p0) / n)
z = (p1 - p0) / se
print(round(z, 2))
# 3.54

Step by step: p₀(1 − p₀) = 0.004 × 0.996 = 0.003984. Dividing by n = 2,000 gives 0.000001992, and its square root is the standard error, 0.001411. The gap between observed and baseline, 0.009 − 0.004 = 0.005, divided by that standard error gives z ≈ 3.54. As a cross-check by an independent route: the expected count of missed frauds in 2,000 transactions at the baseline rate is 2,000 × 0.004 = 8, with a Poisson standard deviation of √8 ≈ 2.83; the observed count is 2,000 × 0.009 = 18, giving (18 − 8)/2.83 ≈ 3.54 — the same answer by a different path. A z of 3.54 corresponds to a two-tailed p-value under 0.001: this is not noise. In business terms, the 0.5-percentage-point gap translates to 2,000 × 0.005 = 10 additional missed frauds every hour the canary keeps running at this rate, at roughly ₹50,000 average loss each, or about ₹5,00,000 an hour of extra expected loss. A gate wired to halt automatically at, say, z > 3 would abort this canary and roll back before it ever reaches 25% of traffic — exactly the kind of decision that must not wait for a human to notice a dashboard.

Canary, shadow, and blue-green are not interchangeable

A common mix-up worth separating precisely. In a canary deployment, the new model actually serves a small slice of real traffic and its decisions have real consequences — which is why the statistical gate above matters so much: a bad canary genuinely costs money before anyone notices. In a shadow deployment, the new model runs alongside the incumbent on every request, but only the incumbent's output is returned to the user; the new model's predictions are logged and compared offline, with zero live impact. Shadow deployment is the safer choice when the cost of a live mistake is very high (say, a change to how loan eligibility is scored) and you are willing to wait longer for a verdict, since you need enough logged shadow traffic to reach statistical power before promoting. Blue-green deployment, borrowed unchanged from ordinary software CD, switches all traffic from the old version to the new one at once, with the old version kept running for instant rollback; it gives none of the gradual-exposure protection of canary or shadow, so in ML it is used mainly for changes that are provably safe by construction (a serving-infrastructure upgrade with no change to model weights), never for a genuine new model version.

Correcting the central misconception

The mistake nearly every student makes on first meeting this topic is treating MLOps CI/CD as "the DevOps pipeline you already know, with a training script swapped in for the build step." That framing misses the two properties that actually define the discipline. First, per the CACE principle from Sculley et al., a model's behaviour on every feature is entangled with every other feature through joint parameter fitting — you cannot unit-test "did the model get better" the way you unit-test "does this function return the right value," because there is no fixed right answer, only a statistical comparison against a baseline on held-out data, which is why the validation gate above is a hypothesis test and not an assertion. Second, and more fundamentally, a software CI/CD pipeline is asleep whenever no one pushes code — it has no mechanism to notice the world changing under a system that has not been touched. MLOps CI/CD is not the same pipeline with ML swapped in; it is that pipeline plus an entirely new triggering mechanism, Continuous Training, that can fire with zero code changes at all, purely because a monitored statistic like PSI crossed a threshold. A release train with no CT stage will pass every one of its own tests forever while the model it ships quietly stops being correct — which is exactly what happened to the UPI fraud model at the start of this chapter.

Active recall

Attempt each question before reading its answer.

1. In one sentence each, what does CI check, what does CD automate, and what does CT trigger on, in an ML pipeline — and which of the three has no equivalent in ordinary software engineering?

2. Why is "roll back to the previous model weights" sometimes an unsafe rollback, even when the weights file itself is uncorrupted?

3. The transaction-amount PSI computed in this chapter was 0.2021, in the "moderate — watch" band. Suppose next month the drift continues in the same direction: the 20,000+ bin grows from 14% to 20% of traffic, and to keep the distribution summing to 100%, the 500–2,000 bin drops from 30% to 24% (all other bins unchanged). Recompute PSI bin by bin, and state whether the CT trigger now fires.

4. A shadow deployment and a canary deployment both "test a new model on live traffic before full rollout." What is the one structural difference between them, and why does it mean a bad shadow model costs nothing while a bad canary model costs real money?

5. The canary z-test in this chapter used n = 2,000 transactions and found z ≈ 3.54, safely past a gate threshold of 3. If the canary slice were cut from 5% to 1% of traffic (n = 400 instead of 2,000, same observed rates p₀ = 0.004, p̂ = 0.009), recompute z. Does the gate still fire, and what does that imply about choosing a canary's traffic percentage?

6. Explain "training-serving skew" and name the specific architectural component in the pipeline diagram that exists to prevent it.

Answers

1. CI checks that new code, data, and features pass deterministic tests (schema validity, transform unit tests, data quality) before anything downstream runs. CD automates packaging and the staged rollout of a validated model artifact into production, with an automatic abort path. CT triggers a fresh training run — on a schedule, or automatically when a monitored signal like PSI crosses a threshold, with no code change required at all. CT is the pillar with no software-engineering equivalent, because ordinary software has no notion of its own artifact silently going stale while the code stays fixed.

2. Because a model release is really a triple — (weights, data version, hyperparameters) — glued together with a specific git commit of the training and serving code. Rolling back only the weights file can leave the serving code expecting a feature schema or preprocessing step that the newer training pipeline introduced (say, a new normalisation range or an added feature), so the "restored" old weights are fed inputs shaped for the new pipeline and can produce worse predictions than before the rollback, or crash outright. A safe rollback restores the whole registry entry, code commit included, not just the weights.

3. New actual distribution: [0.06, 0.24, 0.25, 0.25, 0.20] (still summing to 1.00). Bin 1, 3, and 4 terms are unchanged from before (0.02043, 0.00912, 0.05108). Bin 2: diff = 0.24 − 0.40 = −0.16, ratio = 0.24/0.40 = 0.600, ln(0.600) = −0.5108, term = (−0.16)(−0.5108) = 0.08173. Bin 5: diff = 0.20 − 0.05 = 0.15, ratio = 0.20/0.05 = 4.000, ln(4.000) = 1.3863, term = (0.15)(1.3863) = 0.20794. Sum = 0.02043 + 0.08173 + 0.00912 + 0.05108 + 0.20794 = 0.3703. That exceeds the 0.25 "significant shift" threshold, so the CT trigger fires: the pipeline automatically launches a new training run on the latest data snapshot, which then has to pass the CI schema/unit checks, clear the validation gate against the current production model on a fresh holdout, and go through its own 5%→100% canary ramp before it can replace the live model — the retrain does not skip any stage of the pipeline just because it was triggered automatically rather than by a human commit.

4. In a shadow deployment, the new model's output is computed and logged but never returned to the user — the incumbent model's output is what actually reaches production, so a bad shadow model produces bad predictions that are discarded, costing nothing but compute. In a canary deployment, the new model's output is the one actually returned to whatever slice of users it's serving, so a bad canary model makes real decisions — denying real transactions, missing real frauds — for as long as it stays live, which is exactly why the canary needs a statistical abort gate and shadow does not.

5. se = √(0.004 × 0.996 / 400) = √(0.00000996) ≈ 0.003156. z = 0.005 / 0.003156 ≈ 1.58. That falls below a gate threshold of 3 (or even the conventional 1.96 for a 95% two-tailed test only barely, and well below a stricter production threshold), so the same underlying regression that was unmistakable at n = 2,000 would not clear a typical gate at n = 400 — it is statistically indistinguishable from noise at that sample size. This is the core tension in choosing canary size: a smaller slice limits the blast radius of a bad model (fewer real users affected) but also limits the statistical power to detect a real regression quickly, letting a genuinely broken model run undetected for longer. Production systems usually resolve this by starting small but holding the canary at that traffic level for longer — accumulating enough transactions to reach adequate power — rather than by starting large.

6. Training-serving skew is a discrepancy between how a feature is computed at training time (often in a batch job, over historical data, in Python/SQL) and how the same feature is computed at serving time (often in a low-latency production service, sometimes in a different language, over live data) — even a subtly different rounding rule, null-handling default, or time-window boundary between the two implementations means the model sees inputs at serving time that don't match what it learned during training, degrading accuracy without any error being thrown. The feature store in the pipeline diagram exists specifically to prevent this: both the training pipeline and the serving pipeline read the same feature-computation logic and the same point-in-time-correct values from one shared store, rather than each maintaining its own separate implementation of "how do I compute average transaction amount over the last 30 days."

Think About It

Think about this: How would you explain mlops: ci/cd for machine learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind mlops: ci/cd for machine learning, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.