Edge Deployment: ML on Devices

The Deccan Queen enters a tunnel in the Bhor Ghat outside Karjat, and every phone in the compartment loses its last bar of signal in the same second. The ticket examiner is two rows away, scanning QR codes off people's screens. When she reaches you, you glance at your phone. No PIN, no pattern — it unlocks in well under a second, exactly the way it always does, tunnel or no tunnel. You show her the ticket and think nothing more of it.

But sit with that half-second for a moment. Your phone just captured your face, ran it through a trained neural network, compared the result against the face data it has stored for you, and decided you are you — entirely inside the device, with zero bytes sent anywhere, because for the next ninety seconds there was nowhere to send them. Compare that to asking a chatbot a question, which needs a live connection to a data centre that might be a thousand kilometres away. Both are machine learning. Only one of them works in a tunnel.

This chapter is about that difference, and about the engineering that makes the tunnel case possible: how you take a model built and trained on powerful cloud hardware and squeeze it down until it can think for itself, at speed, on a device that fits in your pocket and runs on a battery.

Inference on the Device vs. Inference in the Cloud

Every machine learning system has two very different phases. Training is the process of learning: feeding a model thousands or millions of examples and adjusting its internal numbers — its weights — until it gets good at a task. Training is enormously expensive in compute, and it almost always happens on powerful, well-cooled hardware: clusters of GPUs or TPUs sitting in a data centre. Inference is what happens afterward — using an already-trained model to make one prediction on one new input: this photo, this sentence, this face. Inference is far cheaper than training, but it still has to happen somewhere, and that "somewhere" is a real design decision.

If inference happens on a server that the device talks to over the internet, that is cloud inference: your phone sends data out, a data centre runs the model, and a result comes back. If inference happens directly on the device holding the data — your phone, a smartwatch, a security camera, a car — that is edge deployment, and the device doing the work is called an edge device. Face unlock, the predictive text in your keyboard, and the voice model that recognises a wake word before your phone even finishes waking up are all edge inference. A large language model writing a detailed answer to an open-ended question is, almost always, cloud inference — the model is simply too large to fit on a phone.

Why Not Just Call the Cloud Every Time?

Cloud inference is the default for a good reason: the model can be as large and accurate as you want, since it is not limited by what fits in someone's pocket. So why does an entire branch of MLOps exist just to push models onto devices instead? Five practical reasons keep showing up.

• Latency. Every cloud call is a round trip — data goes out, gets processed, and an answer comes back — and each leg takes real time, commonly tens to a few hundred milliseconds depending on network quality. A well-optimised model running locally can respond in single-digit milliseconds, because there is no network hop to wait for at all. For a camera filter that has to keep up with 30 frames of video every second, that difference is the whole game: each frame must be processed in well under 1000 ÷ 30 ≈ 33 milliseconds to look smooth, and a round trip to a server rarely fits inside that budget.
• Reliability. A cloud model simply does not work without a network connection. Face unlock has to work in a Ghat-section tunnel, in a lift, in a village with patchy 4G, and in flight mode at 35,000 feet — situations that describe a very large fraction of India at any given moment.
• Privacy. Your face, your voice, and your typing patterns are sensitive. Keeping the raw data on the device and only ever producing a yes/no decision locally means biometric data never has to leave your phone at all.
• Cost and bandwidth. A cloud model has to serve every request from every user, forever, on rented server time — a real, recurring bill that scales with the size of your user base. A model shipped inside the app is a one-time engineering cost; after that, each prediction is effectively free, since it runs on hardware the user already owns.
• Battery. Waking a phone's radio, sending data out, and waiting for a reply is not free of power cost either. For small, frequent tasks — checking a wake word thousands of times a day — a tiny model running on efficient local hardware can use less energy overall than repeatedly waking up the network connection.

A Phone Is Not a Data Centre

To see why edge deployment is hard, it helps to see just how different the two environments are. A data-centre GPU built for training, such as an NVIDIA A100, ships with 40 or 80 gigabytes of dedicated memory, draws several hundred watts of power, and sits in a rack with dedicated cooling and a stable power supply the size of a room. A typical smartphone sold in India has somewhere between 4 and 8 gigabytes of RAM, shared with the operating system, the camera app, WhatsApp, and everything else running at the same time — and its entire power budget, screen and radio included, is a handful of watts, drawn from a battery holding roughly 15 to 20 watt-hours that is expected to last a full day. There is no fan. If a chip runs hot for too long, the operating system throttles it to protect the hardware and the user's hand.

Because of this squeeze, most modern phone chipsets include a small, specialised piece of hardware built to do one thing extremely efficiently: run neural networks. This is usually called a Neural Processing Unit (NPU), and it sits on the same chip alongside the regular CPU and GPU. Apple calls its version the Neural Engine; Qualcomm, whose Snapdragon chips power a large share of Android phones sold in India, calls its version the Hexagon processor; MediaTek, whose Dimensity chips are common in budget and mid-range Indian phones, calls its version the APU. An NPU is built almost entirely out of circuits that do one operation — multiply two numbers and add the result to a running total — over and over, in parallel, using far less energy per operation than a general-purpose CPU would. The catch is that an NPU works best on models shaped to fit its constraints: predictable, fixed-size operations, and, critically, low-precision numbers instead of the high-precision numbers used during training.

Shrinking a Model to Fit in Your Pocket

A model trained on a cloud GPU is usually stored using 32-bit floating-point numbers, or FP32, for every weight, because that precision helps training converge accurately. FP32 is also wasteful: every single number takes 4 bytes, and a modestly sized image classifier can easily have several million such numbers. Before a model can live comfortably on a phone, engineers typically apply one or more compression techniques.

• Quantization reduces the precision used to store each weight — commonly from 32-bit floats down to 8-bit integers, or INT8. Instead of storing 0.734 as a float, you store a small integer that approximates it, along with one scaling number that tells you how to convert back. This is the single most widely used compression technique in edge deployment, and we will trace exactly how it works below.
• Pruning removes weights and connections that contribute almost nothing to the output. Many trained networks turn out to have far more connections than they actually need, and setting the smallest ones to exactly zero — or removing whole neurons and filters — shrinks the model with only a small accuracy cost.
• Knowledge distillation trains a small "student" model to imitate the outputs of a large, accurate "teacher" model, rather than training the student from scratch on raw labels alone. The student ends up smaller and faster than the teacher, while inheriting much of what the teacher learned.
• Efficient architecture design builds the model to be small from day one. MobileNet, a well-known image-classification architecture from Google designed specifically for phones, replaces the standard convolution operation with a cheaper two-step version called a depthwise separable convolution, cutting both computation and parameter count dramatically compared to older, larger architectures built for data-centre GPUs.

These techniques are often combined: start from an architecture designed to be lean, distil it from a larger teacher if one is available, prune the smallest remaining weights, and quantize whatever is left. Each step trades away a little accuracy for a real, measurable reduction in size and latency, which is why the next step is always the same question: how much did we actually save, and what did it cost?

Worked Example: Quantizing a Weight, Byte by Byte

Quantization sounds abstract until you do it once by hand. Suppose that after training, the weights in one layer of a network range from −1.27 to +1.27. We want to store each of those numbers using an 8-bit signed integer instead of a 32-bit float, using the symmetric int8 range from −127 to +127 — keeping it symmetric around zero avoids needing an extra offset, since this weight distribution is already centred near zero.

The first thing quantization needs is a scale: a single number that maps the real-valued range onto the integer range.

scale = max(abs(min_weight), abs(max_weight)) / 127
scale = max(abs(-1.27), abs(1.27)) / 127
scale = 1.27 / 127
scale = 0.01

Now take one specific weight from that layer, say w = 0.734, and quantize it:

• Step 1 — divide by the scale: 0.734 ÷ 0.01 = 73.4
• Step 2 — round to the nearest integer: round(73.4) = 73
• Step 3 — store 73. Instead of 4 bytes for the float 0.734, the model now stores the integer 73 in a single byte.

To actually use this weight during inference, the interpreter reverses the process, multiplying the stored integer back by the scale:

• Dequantize: 73 × 0.01 = 0.73
• Error introduced: |0.734 − 0.73| = 0.004

A single weight is off by 0.004 — about half a percent. Spread this small, mostly random rounding error across millions of weights, and a network's overall accuracy typically drops by only a fraction of a percentage point, which is a trade most applications are happy to make. Here is the same arithmetic as runnable code:

def quantize(weight, scale):
    return round(weight / scale)

def dequantize(q_value, scale):
    return q_value * scale

scale = 1.27 / 127          # 0.01
w = 0.734

q = quantize(w, scale)           # round(73.4) = 73
w_approx = dequantize(q, scale)  # 73 * 0.01 = 0.73

print(f"Original: {w}, Stored as: {q}, Recovered: {w_approx}")
print(f"Error: {abs(w - w_approx):.3f}")
# Original: 0.734, Stored as: 73, Recovered: 0.73
# Error: 0.004

Now scale this up to an entire model. MobileNet, the architecture mentioned above, has roughly 4.2 million parameters in its standard configuration. Stored in FP32, at 4 bytes each:

4,200,000 parameters × 4 bytes = 16,800,000 bytes ≈ 16.8 MB

Quantized to INT8, at 1 byte each:

4,200,000 parameters × 1 byte = 4,200,000 bytes ≈ 4.2 MB

That is a model going from 16.8 MB down to 4.2 MB — exactly a 4× reduction, because it is exactly a 4-byte-to-1-byte reduction — for a typical accuracy cost of well under one percentage point on standard image benchmarks. On top of the size win, INT8 arithmetic runs faster on NPUs than FP32 arithmetic does, so quantization usually buys lower latency as well as a smaller file.

From Training Ground to Pocket: The Deployment Pipeline

Getting a model from a training script to a running app follows a fairly standard sequence in a real MLOps workflow.

• Train the full-precision model on powerful hardware — a cloud GPU or TPU cluster — using as much data and compute as the budget allows, and measure its baseline accuracy on held-out test data.
• Compress the trained model using quantization, pruning, or distillation, checking accuracy again after every change so you know exactly what each step costs.
• Convert the model into a format built for on-device runtimes — commonly .tflite for Google's TensorFlow Lite (recently rebranded LiteRT), .onnx for the cross-framework ONNX Runtime, or Core ML's .mlpackage format for Apple devices.
• Bundle that converted file inside the mobile app or device firmware, the same way you would bundle an image or a font.
• Run the model at inference time using a lightweight on-device interpreter, which loads the file and routes each operation to whichever piece of hardware — CPU, GPU, or NPU — will execute it fastest.
• Monitor real-world performance after release, because a model that scored well in the lab can behave differently across thousands of phone models, camera qualities, and lighting conditions in the field. Deployment is not the finish line; it is where the real test data starts arriving.

In practice, the compress and convert steps are usually done with the same tool. Here is a Python conversion script using TensorFlow, turning an already-trained Keras model into a quantized .tflite file:

import tensorflow as tf

# `model` was already trained to classify handwritten digits (0-9)
model = tf.keras.models.load_model("digit_classifier.h5")

# Convert to TensorFlow Lite format, applying default
# optimizations (this enables post-training quantization)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quantized_model = converter.convert()

with open("digit_classifier.tflite", "wb") as f:
    f.write(tflite_quantized_model)

print("Saved a compressed model ready for on-device deployment.")

This .tflite file is what actually ships inside the app. Before handing it to an Android or iOS engineer, it is good practice to check that it still behaves correctly by running it through Python's own TFLite interpreter — the same engine that will eventually run inside the app, just invoked from a laptop first:

import numpy as np
import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="digit_classifier.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# One 28x28 grayscale digit image, pixel values scaled to [0, 1]
digit_image = np.random.rand(1, 28, 28, 1).astype(np.float32)

interpreter.set_tensor(input_details[0]["index"], digit_image)
interpreter.invoke()

prediction = interpreter.get_tensor(output_details[0]["index"])
predicted_digit = np.argmax(prediction)
print(f"Predicted digit: {predicted_digit}")

On the actual phone, an Android app would load this same .tflite file using TensorFlow Lite's Kotlin or Java API, and an iOS app would typically convert it to Core ML instead — but the file produced by the conversion step above is the artefact that makes either one possible.

What Compression Costs You

None of this is free, and a large part of MLOps work on the edge is measuring the trade-off honestly rather than assuming it will be fine.

• Accuracy. Quantization and pruning almost always cost some accuracy — often a small fraction of a percentage point for careful INT8 quantization, but potentially much more for aggressive compression, a poorly chosen calibration dataset, or a model that was already operating close to its limits. The only way to know is to re-run the full evaluation after every compression step, not just once at the end.
• Latency budgets depend on the use case. A live camera filter needs each frame processed in well under 33 milliseconds to hold 30 frames per second; a wake-word detector needs to run constantly in the background on almost no power at all; a fraud check running after a UPI QR scan can usually afford to take a little longer. There is no single "fast enough" — the target comes from the product, not the model.
• Hardware diversity. A model tested only on a flagship phone or a laptop's GPU can behave very differently on the wide range of budget and mid-range devices actually sold across India, some of which lack a strong NPU and fall back to a slower CPU path. Just as training data has to represent the users a model will serve, test devices have to represent the hardware it will actually run on.

Edge ML You've Already Used Today

Once you know what to look for, on-device inference turns out to be everywhere in a typical Indian smartphone user's day.

• Face unlock, as in the opening scene — a small neural network compares your face against locally stored data and returns a yes/no decision, with no network round trip involved at all.
• Google Translate's camera mode can translate the text in a photo — a shop sign, a menu, a form — the instant you point your camera at it, and once you have downloaded a language pack for offline use, this works with no internet connection whatsoever, which matters enormously outside strong-network metro areas.
• Gboard and other Android keyboards predict your next word, including mixed Hindi-English typing, using a small language model that runs locally on every keystroke. Sending every keystroke to a server for prediction would be both far too slow and a serious privacy problem.
• UPI QR payments show a genuinely hybrid design worth noticing: reading and decoding the QR code from the camera feed is on-device computer vision, fast and fully offline, but authorising the actual transfer of money still requires a live connection to your bank and the UPI network. Edge and cloud are not rivals here; the system uses each one for what it is good at.

Back on the Train

The tunnel outside Karjat eventually ends, and every phone in the compartment finds its signal again within a few seconds, right on schedule. But your phone's face-matching model was never waiting for that signal. It did its job in the dark, using a model that started life as a large, FP32, cloud-trained network and was deliberately shrunk — quantized down to 8-bit integers and converted into a compact runtime file — specifically so it could live inside a chip the size of a fingernail and answer in under a second, on battery power, with nothing sent anywhere.

That shrinking is not a shortcut taken because engineers were careless about accuracy. It is a deliberate, measured trade, made the way you traced it above: compute a scale, quantize each weight, check exactly how much accuracy you gave up, and decide whether the size and speed you gained were worth it. In a country where connectivity ranges from gigabit fibre in a metro apartment to a single flickering bar in a Ghat-section tunnel, a model that only works when the network cooperates is not really a finished product at all. Getting the model right is half the job. Getting it to run reliably in someone's pocket — in a tunnel, on a five-year-old phone with 4 GB of RAM — is the other half, and it is the half this chapter has been about.

Think About It

Think about this: How would you explain edge deployment: ml on devices to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind edge deployment: ml on devices, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.