1. A model that answers one question is not a system that answers a million
IRCTC's AskDISHA chatbot exists to answer exactly the kind of question that floods in every morning at 10:00 AM, the instant the Tatkal booking window opens: "PNR status," "is there a waitlist on 12951," "refund not credited." Now imagine rebuilding it around a modern large language model instead of the rule-based system it actually runs on. In a notebook, this is a five-minute job: load a 7-billion-parameter checkpoint, call model.generate() on one query, get a fluent answer in under a second. The demo is convincing. It is also almost useless as evidence about whether the system will survive 10:00 AM, when tens of thousands of passengers ask a question in the same sixty seconds.
The reason is that a notebook and a production system are being measured on different axes entirely. A notebook is scored on accuracy, fluency, whether the answer is right. A production system is additionally scored on throughput (queries served per second per GPU), tail latency (not the average response time, but the 99th-percentile one, since that's the passenger who gives up and calls the helpline), and cost per query (because someone is paying for every GPU-second, whether or not it produced anything useful). None of these three numbers show up when you call generate() once in a notebook. All three are decided by the same underlying resource, and understanding that resource is the actual subject of this chapter: not "how do transformers work" — you already know that — but "what physically limits how many of them a GPU can run for how many users, at what cost."
2. The roofline: why serving is usually a memory problem, not a compute problem
Every processor has two independent ceilings: how many floating-point operations it can do per second (its peak FLOPS), and how many bytes it can move between its memory (HBM) and its compute units per second (its memory bandwidth). The roofline model (Williams, Waterman & Patterson, Communications of the ACM, 2009) plots achievable performance against a single ratio, arithmetic intensity — how many FLOPs a piece of work does per byte it reads from memory. Below a hardware-specific threshold called the ridge point (peak FLOPS ÷ peak bandwidth), a workload is memory-bound: the compute units sit idle waiting for bytes to arrive, no matter how fast they are. Above the ridge point, it's compute-bound: bytes arrive faster than the arithmetic can consume them.
Now put a single autoregressive decode step through this lens. Generating one token means one forward pass through every parameter of the model. A standard approximation used throughout the scaling-law literature (Kaplan et al., 2020, "Scaling Laws for Neural Language Models") treats each parameter as costing about two FLOPs — one multiply, one add — per token. For a P-parameter model, that's ≈2P FLOPs. To do that arithmetic, every one of those P parameters has to be streamed out of HBM at least once, because a 7-billion-parameter model's weights (14 GB in fp16) dwarf the ~40 MB of on-chip cache a GPU has to reuse data from. So the bytes moved are ≈2P as well (2 bytes per parameter at fp16). Arithmetic intensity at batch size 1:
AI = 2P FLOPs ÷ 2P bytes = 1 FLOP per byte moved.
An NVIDIA A100 40GB (PCIe) delivers roughly 312 TFLOPS of fp16 tensor-core throughput and roughly 1,555 GB/s of HBM bandwidth, per its published spec sheet — a ridge point of about 312e12 ÷ 1.555e12 ≈ 200 FLOP/byte. A single decoding request runs at an arithmetic intensity of 1, two hundred times below the ridge point. The GPU's math units are, quite literally, spending almost the entire step idle, waiting for weight bytes to arrive over the memory bus. You can verify this by timing the bound directly: streaming 14 GB of fp16 weights at 1,555 GB/s takes 14e9 ÷ 1.555e12 ≈ 9.0 milliseconds — and that's the floor on a single decode step regardless of how much raw compute the chip has, because the compute finishes long before the bytes do.
Common misconception — "a faster GPU (more FLOPS) means faster chatbot replies." This is the claim most people intuitively make, and the roofline shows exactly why it's usually wrong for single-request decoding: at AI ≈ 1, you are nowhere near the compute ceiling, so raising the compute ceiling further changes nothing. What actually shortens that 9.0 ms floor is either more memory bandwidth, or fewer bytes to move per step. The second lever — quantization — is one we'll return to with real numbers in Section 6, and it works precisely because it attacks the bottleneck that's actually binding, not the one that isn't.
3. Batching is how you buy back the wasted compute
If a single request leaves 199 out of every 200 FLOP-slots unused, the fix is obvious: don't serve one request at a time. When B independent sequences are batched together, the GPU still only has to stream each of the P parameters out of HBM once per step — the weights are read into on-chip memory and then reused across all B rows of the batched matrix multiply — but now it performs ≈2P·B FLOPs against those same bytes. So:
AI(B) = 2P·B ÷ 2P = B FLOP/byte (at fp16).
Batching converts a memory-bound problem into a progressively more compute-bound one, for free, just by serving more people at once — this single fact is the entire economic reason production LLM systems batch aggressively rather than handling requests one at a time. But it isn't free in every sense: every one of those B concurrent sequences needs its own working memory on the GPU, in the form of a KV cache — the per-token key and value vectors that self-attention needs to keep around for every previous token in that sequence, so it doesn't recompute them at every step. That memory has to come from the same 40 GB the weights already live in, and it is what actually caps how large B can get in practice.
4. Sizing the KV cache: a full worked derivation
For a transformer with L layers and hidden dimension H (so that the combined width of all attention heads is H), each token requires storing one key vector and one value vector of size H per layer. At d bytes per stored number:
bytes per token = 2 (K and V) × L × H × d
For a Llama-2-7B-shaped model (L = 32, H = 4,096) stored at fp16 (d = 2 bytes): 2 × 32 × 4,096 × 2 = 524,288 bytes ≈ 512 KB per token. A single 2,048-token conversation therefore costs 512 KB × 2,048 ≈ 1.07 GB of GPU memory just to hold its KV cache — before the model has generated a single new word, that memory is already spoken for by the prompt. Here is the full budget, computed rather than eyeballed:
def kv_cache_bytes_per_token(num_layers, hidden_dim, dtype_bytes=2):
# one K vector and one V vector, per layer, each of size hidden_dim
return 2 * num_layers * hidden_dim * dtype_bytes
def max_concurrent_sequences(gpu_mem_gb, model_params_b, seq_len,
num_layers, hidden_dim,
weight_dtype_bytes=2, kv_dtype_bytes=2):
gpu_bytes = gpu_mem_gb * (1024 ** 3)
weight_bytes = model_params_b * 1e9 * weight_dtype_bytes
free_bytes = gpu_bytes - weight_bytes
per_seq_bytes = kv_cache_bytes_per_token(
num_layers, hidden_dim, kv_dtype_bytes) * seq_len
return int(free_bytes // per_seq_bytes), free_bytes, per_seq_bytes
n, free, per_seq = max_concurrent_sequences(
gpu_mem_gb=40, model_params_b=7, seq_len=2048,
num_layers=32, hidden_dim=4096)
print(f"{n} sequences | {free/1e9:.2f} GB free | {per_seq/1e9:.2f} GB/sequence")This prints 26 sequences | 28.95 GB free | 1.07 GB/sequence. Walking the arithmetic: the model's 7 billion fp16 weights consume 14 GB, leaving 40 × 1024³ − 14e9 ≈ 28.95 GB free on the card; each 2,048-token sequence's KV cache costs ≈1.07 GB (matching the hand derivation above exactly); 28.95 ÷ 1.07 rounds down to 26 sequences the GPU can hold in flight simultaneously. (This ignores the 10–20% of memory that real serving frameworks reserve for activation buffers and fragmentation headroom, so a production deployment would see a slightly lower cap than the raw arithmetic suggests — a real, and important, engineering margin.) Twenty-six is a hard ceiling: request number 27 cannot be admitted no matter how much spare compute the GPU has, because there is nowhere left to store its attention history.
5. Static batching wastes the capacity you just paid for
Knowing the ceiling is only half the problem — the other half is how requests are scheduled into it. The naive approach, static (request-level) batching, forms a batch of B requests, runs the entire batch forward step by step, and only starts a new batch once every request in the current one has finished. If four requests in a batch need 4, 10, 14, and 16 decoding steps respectively, the framework must either compute on meaningless padding tokens for the three that finished early or simply leave those GPU rows idle — either way, the resource sits unused — until the sixteenth step, when all four retire together and a new batch of four can begin.
The diagram's own numbers make the cost concrete. Over the 16-step window, static batching leaves row A idle for 12 steps, row B for 6, row C for 2, row D for 0 — 20 idle slot-steps out of 4 rows × 16 steps = 64 total, or 31.25% of paid-for GPU capacity thrown away, and worse, a fifth user arriving at step 5 has to wait until step 16 to be admitted even though three of the four rows had been sitting empty for a while. Continuous batching, described next, removes both problems at once.
6. Continuous batching and PagedAttention
The fix, called iteration-level scheduling, was introduced by Orca (Yu et al., OSDI 2022): instead of committing to a fixed batch until every member finishes, the scheduler re-evaluates who's active after every single decode step. The instant request A finishes at step 4, its row is free, and a newly queued request can be admitted into it on step 5 — not step 16. Every row stays busy on every step; the diagram's row A goes on to serve request E and then request I within the same 16-step window, while under static batching it would have sat hatched-out and empty.
This scheduling idea by itself doesn't solve the memory problem from Section 4 — as sequences of wildly different lengths come and go, naively allocated contiguous KV-cache buffers fragment the GPU's memory the way a badly managed heap does. PagedAttention (Kwon et al., SOSP 2023 — the system behind the widely used vLLM serving engine) solved this by borrowing the idea of OS virtual memory paging: a sequence's KV cache is stored in small, fixed-size, non-contiguous memory blocks (analogous to memory pages), with a lightweight block table mapping each sequence to its blocks. Blocks free up and get reassigned to new sequences immediately and without fragmentation, which is what actually makes the 26-sequence capacity ceiling computed in Section 4 achievable in practice rather than lost to memory waste.
The scheduling loop itself, stripped to its control flow (the model's actual forward pass is a separate, heavyweight operation, marked here rather than implemented):
EOS_TOKEN = -1 # sentinel; a real tokenizer defines this
class Request:
def __init__(self, req_id, prompt_tokens, max_new_tokens):
self.req_id = req_id
self.tokens = list(prompt_tokens)
self.max_new_tokens = max_new_tokens
self.generated = 0
self.finished = False
def run_decode_step(active_requests):
# (assumed helper, not shown) one batched forward pass:
# takes the last token of every active request, returns
# one next-token id per request.
...
def continuous_batching_scheduler(incoming_queue, max_batch_size):
active = []
steps = 0
while incoming_queue or active:
while len(active) < max_batch_size and incoming_queue:
active.append(incoming_queue.pop(0))
next_tokens = run_decode_step(active)
for req, tok in zip(active, next_tokens):
req.tokens.append(tok)
req.generated += 1
if req.generated >= req.max_new_tokens or tok == EOS_TOKEN:
req.finished = True
active = [r for r in active if not r.finished]
steps += 1
return stepsThe line doing the actual work is the inner while loop: it tops up active from the queue on every iteration of the outer loop, not once per batch. That one difference from a static implementation — refilling continuously instead of at batch boundaries — is the entire mechanism behind the throughput gain shown in the diagram.
7. Quantization: paying precision to buy back memory and bandwidth
Section 4's 26-sequence ceiling came from a fixed 40 GB budget split between 14 GB of fp16 weights and however much is left for KV caches. Weight-only quantization schemes such as GPTQ and AWQ compress those weights to 4 bits (0.5 bytes) per parameter while leaving activations and KV cache in a higher-precision format, cutting weight storage to 3.5 GB. Re-running the same function from Section 4 with weight_dtype_bytes=0.5:
n2, free2, _ = max_concurrent_sequences(
gpu_mem_gb=40, model_params_b=7, seq_len=2048,
num_layers=32, hidden_dim=4096, weight_dtype_bytes=0.5)
print(f"{n2} sequences | {free2/1e9:.2f} GB free")This prints 36 sequences | 39.45 GB free — freeing 10.5 GB of weight memory buys room for ten more concurrent 2,048-token conversations, a 38% capacity gain for a single deployment. Quantization pays off a second time on the roofline from Section 2: with d bytes per parameter, arithmetic intensity generalizes to AI(B) = 2B/d. At fp16 (d = 2), that's exactly B, as derived earlier; at int4 (d = 0.5), it's 4B — the same 26-or-36-sequence batch now moves four times fewer bytes per unit of useful compute, pushing the workload substantially closer to the ridge point. (If the GPU also computes in int4 rather than just storing weights that way, its own ridge point shifts too — but the qualitative result survives either way: quantization attacks both halves of the memory bottleneck, capacity and bandwidth, at once.) It does not, however, touch the KV-cache side of the memory budget at all — a longer conversation still costs exactly as much whether the weights are fp16 or int4, which is the crux of one of the active-recall questions below.
8. Beyond the serving loop: versioning, drift, and rollback
Everything above concerns a single, static, already-trained model. A production system also has to survive the model changing under it. Teams typically run a canary or shadow deployment before a full rollout — routing a small slice of live traffic (say 1%) to a new model version, comparing its behavior against the incumbent on the same real inputs, and only promoting it once it's not silently worse on some subgroup of queries the offline evaluation set didn't cover. Monitoring in production tracks not the average latency but the p99 — the worst 1% of requests — because that's the tail that generates complaints and churn even when the mean looks fine. It also tracks drift: input distribution drift (the mix of questions users actually ask shifts over time — Tatkal-week traffic looks nothing like an ordinary Tuesday) and output drift (the model itself starts producing subtly different answers to the same question after a routine dependency upgrade, without anyone touching the weights). Because a model's failure mode is rarely a crash — it's a confident, fluent, wrong or unsafe answer — this monitoring is inseparable from the ethics side of "AI Applications & Ethics": a drift-detection pipeline that flags a rising rate of a particular harmful or biased response category is doing safety work, not just an SRE's job. Finally, every production model deployment needs an immediate rollback path — the previous version kept warm and ready — because the alternative to a fast rollback is leaving a bad model serving real users while a fix is being debugged.
Active recall
Attempt each of these before reading the answer beneath it.
Q1. A colleague argues that upgrading from a 300-TFLOPS-class GPU to a 900-TFLOPS-class one should roughly triple a chatbot's single-user response speed. Using the roofline idea, explain why this is usually wrong for autoregressive decoding at batch size 1, and say what actually would speed it up.
A1. At batch 1, arithmetic intensity is ≈1 FLOP/byte (Section 2), far below any realistic ridge point (~200 for A100-class hardware) — the GPU is memory-bound, spending its step waiting on HBM bandwidth, not doing math, so tripling peak FLOPS changes nothing about a bottleneck that was never the compute units. Step time is set by weight_bytes ÷ bandwidth (9.0 ms for the 14 GB fp16 example). What does help: more memory bandwidth, or fewer bytes to move — quantization cut the same step to 2.25 ms, a 4× speedup that exactly tracks the 4× drop in bytes per parameter (2 bytes → 0.5 bytes), confirming the workload really was bandwidth-bound the whole time.
Q2. Using max_concurrent_sequences, find the batch-size cap for the same 7B model on the same 40 GB A100 under: (a) fp16 weights, 4,096-token sequences; (b) int4 weights, 4,096-token sequences. Compare all four numbers you now have (fp16/2048, fp16/4096, int4/2048, int4/4096) and explain the direction of each change.
A2. Baseline (fp16, 2,048) = 26. Doubling sequence length alone (fp16, 4,096): per-sequence KV cost doubles to ≈2.15 GB while free memory is unchanged (weights are still fp16), so the cap roughly halves to 13. Quantizing alone (int4, 2,048) = 36, since free memory rises from 28.95 GB to 39.45 GB while per-sequence cost is untouched. Combining both (int4, 4,096) = 18: quantization still grows the numerator (more free memory), but the doubled sequence length still doubles the denominator (per-sequence cost) — the two effects don't cancel, they multiply, landing close to half of the quantize-only figure and well below simply adding the two individual gains. The general lesson: sequence length taxes the KV-cache side of the budget no matter what precision the weights are stored in, while quantization only ever relieves the weight-storage side — the two levers are independent and must both be reasoned about, not substituted for each other.
Q3. Using the request lifecycle in the diagram, explain why continuous batching raises throughput without materially hurting the latency of a request already in flight.
A3. A request already generating tokens is unaffected by what happens in a neighbouring row — its own row advances exactly one step per iteration regardless of whether other rows are being refilled, because the scheduler inserts newly admitted requests only into slots that are already free on that same iteration. What continuous batching removes is queuing delay for new arrivals and wasted capacity for the operator: static batching wastes 20 of 64 slot-steps (31.25%, computed directly from the diagram) and forces a fifth user to wait until step 16 even though three of the four rows had been empty for a while.
Q4. Under the fp16/2,048-token configuration, a 27th concurrent request arrives at a GPU already serving its computed cap of 26. Name three legitimate production responses and one response that is illegitimate because it silently breaks a correctness guarantee.
A4. Legitimate: (i) queue the request and admit it as soon as a slot frees, which is exactly what the continuous-batching scheduler already does; (ii) return explicit backpressure (an HTTP 429 or equivalent "system busy" signal) so the caller can retry or fail visibly; (iii) route it to a second GPU or replica — serving, unlike training, is embarrassingly parallel across independent requests, so unlike the single-GPU compute ceiling from Q1, adding GPUs here scales aggregate throughput close to linearly; (iv) swap a lower-priority sequence's KV cache out to CPU RAM temporarily and resume it later (what vLLM calls preemption). Illegitimate: silently truncating the 27th request's context, or silently serving it a cached/stale answer without telling the caller — both avoid a crash but violate the correctness contract the caller is relying on, which is worse than a visible failure.
Q5. Derive, from first principles, arithmetic intensity as a function of batch size B and bytes-per-parameter d for one decode step of a P-parameter transformer. Then state in one sentence why quantization moves a memory-bound decode workload toward compute-bound.
A5. FLOPs for the batched step ≈ 2P·B (each of the B sequences performs its own ≈2P FLOPs against the same loaded weights). Bytes moved from HBM ≈ P·d (the P parameters are streamed once per step, at d bytes each, and reused across all B rows of the batched matmul while resident on-chip). So AI = 2PB ÷ (Pd) = 2B/d. Quantization shrinks d, which sits in the denominator, so AI rises even at a fixed B — fewer bytes are paid per unit of useful compute, sliding the operating point rightward along the roofline toward (and potentially past) the ridge point, where the GPU's arithmetic, not its memory bus, becomes the limiting resource.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind building production ai systems, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.