Model Serving: TorchServe, Triton, and vLLM

The problem training never prepares you for

A Bengaluru startup fine-tunes a 7-billion-parameter Llama-2 model on three months of customer chat logs to answer order-status questions for an online retailer. The offline evaluation looks great: low perplexity, sensible answers, a happy product team. Someone wraps the checkpoint in a Flask route — model.generate(tokenize(request.json["text"])) — deploys it on a single A100 GPU, and ships it two days before a festive-season sale. On sale day, traffic goes from 20 requests a minute to 4,000. The P50 latency, a comfortable 300 ms in testing, becomes a P99 of 41 seconds. Support tickets pile up faster than the bot can answer them, and nvidia-smi shows the GPU sitting at 6% utilization the entire time — a $30,000 accelerator mostly idle while customers wait.

Nothing about the model changed. What broke is a discipline distinct from model-building: serving — the systems layer that turns a trained checkpoint into a service with predictable latency under concurrent, bursty, variable-length load. A Flask loop processes one request at a time: while GPU core 0 tokenizes request 2, cores 1 through 107 sit empty, because nothing else has been scheduled onto them. Training pipelines never have to solve this — a training job owns the whole GPU for hours and processes one enormous, pre-shuffled batch. Serving must pack unpredictable, arriving-whenever requests onto the same GPU while keeping tail latency low. TorchServe, Triton Inference Server, and vLLM are three answers to that packing problem, each shaped by a different assumption about what kind of model is being served.

TorchServe: turning a checkpoint into a managed service

TorchServe, PyTorch's own serving framework, treats a model as a pluggable unit called a model archive (a .mar file bundling weights, code, and metadata) and runs each loaded model inside a pool of backend worker processes. Requests hit an HTTP or gRPC front end, get routed to an available worker, and a Python object called a handler defines exactly how that worker turns raw bytes into a prediction, through four lifecycle methods: initialize (load the model once), preprocess, inference, and postprocess.

from ts.torch_handler.base_handler import BaseHandler
import torch

class SentimentHandler(BaseHandler):
    # self.model and self.tokenizer are loaded inside initialize(),
    # which BaseHandler already calls with the model artifacts from
    # the .mar file (assumed helper, not shown here).

    def preprocess(self, data):
        text = data[0].get("data") or data[0].get("body")
        if isinstance(text, (bytes, bytearray)):
            text = text.decode("utf-8")
        return self.tokenizer(text, return_tensors="pt", truncation=True)

    def inference(self, tokens):
        with torch.no_grad():
            logits = self.model(**tokens).logits
        return logits

    def postprocess(self, logits):
        probs = torch.softmax(logits, dim=-1)[0]
        return [{"positive": float(probs[1]), "negative": float(probs[0])}]

torch-model-archiver --model-name sentiment --version 1.0 \
  --serialized-file model.pt --handler sentiment_handler.py

torchserve --start --model-store model_store --models sentiment=sentiment.mar

curl http://127.0.0.1:8080/predictions/sentiment -T review.txt

Why a pool of processes rather than a single multi-threaded server? Python's Global Interpreter Lock means one process can only execute one thread of Python bytecode at a time, so threads can't truly parallelize the tokenization and tensor-shuffling work that preprocess/postprocess do on the CPU. TorchServe sidesteps this by spawning several independent OS processes per model, each with its own Python interpreter and its own copy of the model weights, so N workers really do run N requests' CPU-side work concurrently. The cost is memory: four workers for a 2 GB model means roughly 8 GB of GPU or host memory just for redundant weight copies, since nothing is shared across processes. TorchServe also supports dynamic batching (accumulate requests up to a max batch size or a timeout, then run one forward pass) and can host many different models behind one server, with a management API to load, scale, and version them without redeploying the whole service. It is the natural choice when you have one or a handful of ordinary PyTorch models — a classifier, a small vision model — and want REST/gRPC hosting with minimal new infrastructure to learn.

Triton Inference Server: multi-framework orchestration at data-center scale

NVIDIA's Triton Inference Server generalizes the same idea across frameworks. A model repository is a directory of models, each with a config.pbtxt describing its inputs, outputs, and backend — which can be ONNX Runtime, TensorRT, PyTorch TorchScript, TensorFlow, or arbitrary Python code. Triton's distinguishing engineering feature is fine control over how a single GPU is shared among concurrent requests.

name: "resnet50_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 32
input [
  { name: "input" data_type: TYPE_FP32 dims: [3, 224, 224] }
]
output [
  { name: "output" data_type: TYPE_FP32 dims: [1000] }
]
dynamic_batching {
  preferred_batch_size: [8, 16, 32]
  max_queue_delay_microseconds: 5000
}
instance_group [
  { count: 2 kind: KIND_GPU }
]

The dynamic_batching block tells Triton to hold arriving requests for up to max_queue_delay_microseconds (5 ms here), hoping enough arrive to reach a preferred_batch_size, then run one padded forward pass over whatever it collected — if the timer expires first, it runs an under-sized batch rather than blocking indefinitely. The instance_group block is the second lever: it launches multiple concurrent execution streams of the same model on the GPU. This matters because a single instance's pipeline typically has dead time — the CPU-to-GPU copy for the next batch's inputs, or postprocessing of the previous batch's outputs — during which the GPU's compute units (SMs) are unused even though a batch is "running." A second instance can use exactly that dead time to run its own forward pass, so raising instance_group.count is the standard fix when profiling shows GPU compute utilization plateauing well below 100% even though batches are forming at the preferred size. Triton also supports ensembles — DAGs that chain a preprocessing model, an inference model, and a postprocessing model as one served pipeline — which is why it dominates in production vision and recommendation systems that mix several frameworks and stages behind one endpoint.

Why autoregressive generation breaks both of these

Both TorchServe's dynamic batching and Triton's dynamic_batching share an assumption baked into their design: a model does one forward pass per request, all requests in a batch finish at the same time, and results are returned together. That is true for a classifier or a ResNet. It is badly false for a large language model doing autoregressive generation, which produces its output one token at a time, feeding each new token back in as the next input, until it emits an end-of-sequence token or hits a length limit. Two requests submitted together might need 20 tokens and 800 tokens respectively — nobody knows in advance which.

Suppose four requests are batched together this way: three ask short questions needing 50 generated tokens each, one asks for a long essay needing 500 tokens. A GPU can process a batch of concurrent sequences at roughly the same per-step latency regardless of how many active sequences occupy the batch slots (attention and the feed-forward layers dominate the cost, and both scale with the number of tokens processed per step far more than with the number of distinct sequences). Static/dynamic batching requires the whole batch to finish before any results return and before any new request can be admitted, so all four slots stay reserved for the full 500 steps even though three of them go idle after step 50. Total slot-steps consumed = 4 slots × 500 steps = 2,000. Slot-steps that actually produced a token = 50 + 50 + 50 + 500 = 650. Utilization = 650 / 2,000 = 32.5%. Two-thirds of the reserved GPU capacity is wasted holding empty seats for the longest passenger. Under sustained, saturating load — queued requests always waiting — an ideal scheduler could instead refill a freed slot the instant a short request finishes, keeping essentially every slot-step productive. That is close to a 3× throughput gain purely from scheduling, with no change to the model or the hardware. This gap between naive batch scheduling and continuous scheduling is exactly the throughput problem the original PagedAttention work (Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023) set out to close, reporting throughput improvements of roughly 2–4× over prior state-of-the-art LLM-serving systems, and much larger gains over naive request-at-a-time serving like the Flask loop above.

vLLM: continuous batching and paged KV-cache memory

vLLM solves the scheduling half of the problem with continuous (also called iteration-level) batching: instead of batching at the request level, it batches at the level of a single decoding step. After every token generated, the scheduler checks which sequences finished and admits queued requests into those now-free slots immediately, so the GPU's active batch composition can change every step. No request holds up any other request's completion, and no slot sits idle waiting for the batch's slowest member.

Continuous batching alone still runs into a second, purely memory-side problem: every active sequence needs its own KV cache — the stored key and value projections from every previous token, at every transformer layer, so the model doesn't have to recompute attention over the whole prefix at each new step. A naive implementation reserves a contiguous block of GPU memory per sequence sized for the maximum possible sequence length, since it doesn't know in advance how long a given request will run. If the reserved maximum is 2,048 tokens and the sequence actually only needs 1,024, half of that reservation sits wasted (internal fragmentation) — and because different sequences finish at different lengths, memory also fragments externally as chunks free up in different places. vLLM's PagedAttention borrows the operating-system idea of virtual memory paging: each sequence's KV cache is split into small, fixed-size blocks (a typical block holds 16 tokens' worth of keys and values), and a per-sequence block table maps logical block indices to wherever those blocks physically sit in GPU memory — not necessarily contiguous, and not necessarily reserved in advance. A sequence only requests a new physical block when it actually needs one. Waste is now bounded to at most one partially-filled block per sequence, not an entire worst-case-length reservation — the paper reports this brings fragmentation down to under 4% in typical workloads.

Worked example — sizing the KV cache. Each transformer layer stores one key vector and one value vector per token, each of length hidden_size. In fp16 (2 bytes per element), the memory per token per layer is 2 (K and V) × hidden_size × 2 bytes, and across all layers:

KV_bytes_per_token = 2 × num_layers × hidden_size × bytes_per_element

For Llama-2-7B (num_layers = 32, hidden_size = 4096, fp16):

KV_bytes_per_token = 2 × 32 × 4096 × 2 = 524,288 bytes ≈ 512 KB

As a sanity check against a real published figure: OPT-13B has 40 layers and a hidden size of 5,120. The same formula gives 2 × 40 × 5120 × 2 = 819,200 bytes ≈ 800 KB per token — close to the ~800 KB per token figure the PagedAttention paper cites for that exact model, which confirms the formula is being applied correctly rather than just producing a plausible-looking number.

Now size a realistic deployment. Serve a batch of 16 concurrent Llama-2-7B sequences, each averaging 1,024 tokens of prompt-plus-generation:

total_tokens = 16 × 1,024 = 16,384
KV_memory   = 16,384 × 524,288 bytes = 8,589,934,592 bytes ≈ 8.59 GB
weight_memory = 7×10^9 params × 2 bytes = 14×10^9 bytes = 14 GB
total = 8.59 GB + 14 GB ≈ 22.59 GB

On a 40 GB A100, 22.6 GB leaves about 17.4 GB of headroom for activations and CUDA overhead — comfortable. On a 24 GB consumer card (an RTX 4090), 22.6 GB leaves only ~1.4 GB, which is too tight once activation memory and the CUDA context are added; the deployment would not reliably fit.

Now trace what happens if this same deployment switches both the model weights and the KV cache to int8 (1 byte per element instead of 2) — a common move to shrink memory footprint:

KV_bytes_per_token(int8) = 2 × 32 × 4096 × 1 = 262,144 bytes = 256 KB
KV_memory(int8) = 16,384 × 262,144 bytes = 4,294,967,296 bytes ≈ 4.29 GB
weight_memory(int8) = 7×10^9 × 1 byte = 7×10^9 bytes = 7 GB
total(int8) = 4.29 GB + 7 GB ≈ 11.3 GB

Both the KV cache and the weights halve, so total memory drops from 22.6 GB to 11.3 GB — not just the KV-cache term someone might reach for first. On the 24 GB RTX 4090 this now fits comfortably, with roughly 12.7 GB of headroom left over for activations and even a larger batch, whereas the fp16 version did not fit at all. The ripple runs through every term in the budget, not only the one most directly tied to "cache."

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-hf", gpu_memory_utilization=0.90)
params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=200)

outputs = llm.generate(
    ["Explain UPI settlement in one paragraph."], params
)
for out in outputs:
    print(out.outputs[0].text)

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-hf --port 8000

That second command starts an OpenAI-API-compatible endpoint, so existing client code written against the OpenAI chat-completions API can point at a self-hosted vLLM server with only a base-URL change — a large part of why vLLM has become the default execution engine behind many self-hosted LLM deployments, including as a pluggable backend inside Triton itself.

Common misconception: is batching just batching?

Students who learn TorchServe's or Triton's dynamic batching first often assume vLLM's continuous batching is the same mechanism under a different name — "wait for a bunch of requests, then run them together." It is not, and the distinction is exactly the one worked through above. Dynamic batching groups at the request level: a batch is fixed at admission time and every member is stuck together until the whole batch completes, which is fine when every request costs the same one-shot forward pass but catastrophic when request lengths vary by 10× or more, because the batch is only as fast as its slowest member (the 32.5%-utilization example). Continuous batching groups at the decoding-step level: the batch's membership is re-decided after every single token, so a short request's completion immediately frees capacity for a queued one. The two are solving genuinely different problems — one amortizes fixed per-batch overhead for uniform-cost models, the other maximizes tokens produced per second under a fluctuating, per-request cost that isn't known until the request finishes.

How the two batching strategies differ

Choosing among the three in practice

These are not mutually exclusive tools competing for the same job. TorchServe is the right default for a shop running a small number of ordinary PyTorch models — a classifier, an embedding model — that wants REST/gRPC hosting, versioning, and multi-model management without adopting new infrastructure. Triton is the right choice once a deployment needs to mix frameworks (an ONNX preprocessing model feeding a TensorRT-optimized vision model feeding a Python postprocessing step, say, in a recommendation or vision pipeline) or needs the fine-grained instance-group control that squeezes maximum throughput out of a fixed GPU fleet. vLLM (or its close cousin TensorRT-LLM) is purpose-built for one thing — serving autoregressive text generation — and should be the execution engine wherever the workload is an LLM producing variable-length output, because continuous batching and PagedAttention directly attack the two failure modes (request-level scheduling and static memory reservation) that hurt LLM throughput specifically. In practice these compose: Triton can host a vLLM backend, using Triton's request routing, multi-model management, and ensembling around vLLM's decoding-specific scheduler and memory manager, so a large deployment might use Triton as the front door and vLLM as the engine plugged in behind it for every model that happens to be an LLM.

Active recall

Attempt each question before reading its answer.

1. A Triton model has dynamic_batching { preferred_batch_size: [8, 16, 32] max_queue_delay_microseconds: 5000 }. Five requests arrive at once, then nothing for 10 ms. What batch size does Triton run, and when?
2. Using KV_bytes_per_token = 2 × num_layers × hidden_size × bytes_per_element, compute the per-token KV-cache size for a 40-layer model with hidden size 5,120 running in fp16, and name the real model this configuration matches.
3. The Llama-2-7B deployment above (16 sequences × 1,024 tokens, fp16) switches to int8 weights and int8 KV cache. Recompute: (a) new KV-cache bytes per token, (b) new total KV-cache memory for the batch, (c) new weight memory, (d) new grand total, and (e) whether it now fits on a 24 GB RTX 4090.
4. Why does TorchServe run each model's workers as separate OS processes instead of separate threads within one process?
5. A Triton vision model shows GPU compute utilization plateaued at ~40% even though dynamic_batching is successfully forming batches at the preferred size. What single config change is most likely to help, and why?
6. True or false: PagedAttention eliminates KV-cache memory waste entirely.

Worked answers.

1. Triton waits up to the 5,000-microsecond (5 ms) queue delay hoping to reach a preferred size. Since only 5 requests have arrived and none of 8/16/32 is reached, the timer expires at the 5 ms mark and Triton runs the batch with exactly the 5 requests it has — it does not pad up to 8, and it does not keep waiting past the configured delay.

2. 2 × 40 × 5,120 × 2 = 819,200 bytes ≈ 800 KB per token. This matches OPT-13B, the model configuration used as the reference example in the original PagedAttention paper.

3. (a) 2 × 32 × 4,096 × 1 byte = 262,144 bytes = 256 KB per token — half the fp16 value, since only the byte width changed. (b) 16,384 tokens × 262,144 bytes = 4,294,967,296 bytes ≈ 4.29 GB, half the previous ≈8.59 GB. (c) 7×10^9 × 1 byte = 7×10^9 bytes = 7 GB, half the previous 14 GB, because the weight dtype changed too, not just the KV cache. (d) 4.29 GB + 7 GB ≈ 11.3 GB, half the previous ≈22.6 GB total. (e) Yes — 11.3 GB fits comfortably inside 24 GB, leaving roughly 12.7 GB of headroom, whereas the original ≈22.6 GB fp16 footprint did not fit with any safety margin at all. The full ripple touches every term in the budget: KV cache, weights, and the final total, not just the KV-cache line someone might update first.

4. Python's Global Interpreter Lock (GIL) prevents two threads in one process from executing Python bytecode simultaneously, so a single multi-threaded process could not truly parallelize the CPU-side preprocessing/postprocessing across workers. Separate OS processes, each with its own interpreter, sidestep the GIL and let multiple requests' CPU-side work run concurrently — at the cost of N redundant copies of the model in memory for N workers.

5. Increase instance_group.count so more concurrent execution streams of the model run on the same GPU. A single instance typically has idle gaps — waiting on host-to-device data transfer or postprocessing — during which no compute happens even though a batch is "running"; a second instance can use exactly that idle time to run its own forward pass, raising overall SM utilization without changing the batching configuration at all.

6. False. PagedAttention eliminates external fragmentation and bounds internal fragmentation to at most one partially-filled block per sequence (with a 16-token block size, at most 15 wasted tokens' worth of memory per sequence), rather than an entire worst-case-length reservation. It reduces waste to a small, bounded amount — it does not reduce it to zero.

Think About It

Think about this: How would you explain model serving: torchserve, triton, and vllm to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind model serving: torchserve, triton, and vllm, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.