India's eSanjeevani teleconsultation network has crossed 300 million consultations since 2020, run out of district hospitals with patchy bandwidth and, under the Digital Personal Data Protection Act, 2023, a hard rule that patient symptom data cannot be shipped to a foreign API for processing. Suppose your job is to give every Primary Health Centre doctor an AI assistant that drafts a triage summary from a patient's spoken symptoms. You cannot call GPT-4 over the internet — the data residency clause forbids it, and the per-token billing would bankrupt a state health budget serving thousands of consultations a day. The only option is to run an open-weight language model on hardware you control. That single constraint — self-hosting — is what pulls a student from "here is a cool AI model on the internet" into an entire engineering stack: where the weights come from, how you shrink a 14-gigabyte model to fit on hardware a district hospital can actually buy, and how you turn a loaded model into something that answers one doctor's query or ten thousand doctors' queries without falling over. That stack has a name, and it has four distinct, independently swappable layers: Hugging Face Hub (distribution), GGUF quantization (compression), and two competing serving engines — Ollama and vLLM — built for opposite ends of the concurrency spectrum.
Layer 1: The Hub Is a File Format Problem, Not Just a Website
Hugging Face Hub is usually introduced as "GitHub for models" — a place to browse model cards and click download. The engineering detail that matters for production is the file format the weights are stored in, because it determines whether loading a model is safe and how fast it starts. Early Hub checkpoints shipped as PyTorch .bin files, which are Python pickle archives. Unpickling a file executes arbitrary Python bytecode embedded in it — downloading a stranger's .bin file and loading it is equivalent to running their code on your machine. Hugging Face's response, released in 2022, is safetensors: a file that starts with a JSON header mapping each tensor name to its dtype, shape, and byte offset, followed by a single flat buffer of raw tensor bytes. There is no executable content in the format at all — a parser just reads the header and memory-maps the buffer, so loading is both faster (no deserialization, the OS pages tensors in on demand) and cannot run arbitrary code. Today, a model's config.json on the Hub — the file listing hidden_size, num_hidden_layers, num_attention_heads — is the single source of truth that every downstream tool reads to know the model's shape. Both the quantizer that shrinks the weights and the serving engine that allocates memory for them read those same four numbers. That is the thread this chapter follows: one architecture, two very different journeys.
Layer 2: Why FP16 Doesn't Fit, and What GGUF Actually Does About It
A 7-billion-parameter model stored in the standard 16-bit floating point format Hugging Face ships by default costs 2 bytes per parameter: 7,000,000,000 × 2 = 14,000,000,000 bytes, 14 GB. That alone exceeds a consumer 12 GB GPU before you have loaded a single activation or KV cache entry. GGUF — introduced by Georgi Gerganov's llama.cpp project in August 2023 as the successor to the earlier GGML format — solves this by quantizing: replacing each 16-bit float with a much shorter code, block by block.
The simplest scheme, Q4_0, works on blocks of 32 consecutive weights. Within a block, every weight is represented as a 4-bit integer code (0–15), and the whole block shares one 16-bit floating-point scale factor (delta, Δ). To reconstruct a weight you compute code × Δ (after re-centering the 4-bit range around zero) — the scale absorbs the actual magnitude of that neighborhood of weights, so a 4-bit code only has to represent relative position within the block, not the true numeric range of the whole tensor. This is exactly the layer of granularity that matters: quantizing the whole tensor with one global scale would waste bits on outliers, while quantizing every 32 weights separately lets each block's scale match its own local range.
Do the bit accounting for one block: 32 codes × 4 bits = 128 bits, plus one 16-bit scale = 144 bits, for 32 weights. That is 144 ÷ 32 = 4.5 bits per weight — not the "4-bit" the format's name suggests, because the scale overhead has to be amortized in.
# Q4_0 block layout: 32 weights per block
weights_per_block = 32
bits_per_code = 4 # one 4-bit code per weight
scale_bits = 16 # one fp16 scale (delta) per block
block_bits = weights_per_block * bits_per_code + scale_bits # 128 + 16 = 144
bits_per_weight = block_bits / weights_per_block # 144 / 32
n_params = 7_000_000_000
fp16_bytes = n_params * 2
q4_0_bytes = n_params * bits_per_weight / 8
print(bits_per_weight) # 4.5
print(fp16_bytes / 1e9, "GB") # 14.0 GB
print(round(q4_0_bytes / 1e9, 4), "GB")# 3.9375 GB
Trace it: bits_per_weight is exactly 144/32 = 4.5. fp16_bytes is 7e9 × 2 = 14000000000, printed as 14.0 GB. q4_0_bytes is 7e9 × 4.5 / 8 = 3937500000.0, printed as 3.9375 GB. The compression ratio is 16 bits ÷ 4.5 bits ≈ 3.56×, and it must equal the size ratio regardless of parameter count (14 ÷ 3.9375 = 3.5556…) — a useful sanity check whenever you audit a quantized file size, since the ratio depends only on the bit widths, not on how many parameters the model has.
Real GGUF files mostly don't use plain Q4_0 anymore. The "k-quants" (Q4_K_M, Q5_K_M, and similar) apply mixed precision within a model: tensors that empirical testing shows are more sensitive to quantization error — certain attention-output and feed-forward projection matrices — are stored at 5- or 6-bit precision, while the bulk of the weights stay at 4-bit. This pushes the effective average up from the naive 4.5 to roughly 4.7–5.0 bits per weight for the "M" (medium) variants, trading a modest size increase for a measurable drop in perplexity (a model's average per-token surprise, the standard quality metric for a quantized checkpoint) compared to uniform Q4_0.
Two Downstream Paths From One Checkpoint
GGUF is not the only quantization path, and this is where the ecosystem branches in a way worth being precise about. GGUF is specific to the llama.cpp/Ollama world. The vLLM serving engine, covered next, natively consumes Hugging Face's own safetensors checkpoints — either full fp16/bf16 weights, or weights quantized with schemes native to that ecosystem such as AWQ (Activation-aware Weight Quantization, Lin et al., 2023) or GPTQ, which quantize per-channel using calibration data rather than llama.cpp's fixed 32-weight blocks. Treating GGUF as "the" universal quantization format for open models is a common error — it is one format tied to one inference engine family, not a Hub-wide standard.
Layer 3a: Ollama — a Single-Stream Local Daemon
Ollama wraps llama.cpp behind a small HTTP daemon and a Docker-like CLI. You point it at a GGUF file with a Modelfile:
FROM ./openmodel-7b-q4_k_m.gguf
PARAMETER num_ctx 4096
PARAMETER temperature 0.3
SYSTEM """You are a clinical triage assistant. Ask one focused
follow-up question at a time and never suggest a diagnosis."""
ollama create triage-7b -f Modelfile registers it; ollama run triage-7b starts an interactive session. Internally, Ollama loads one GGUF file into memory and processes one generation stream at a time per loaded model instance — there is no batching of concurrent requests inside a single model instance, and no shared paging of the KV cache across requests. That is not a limitation so much as a design choice: a district hospital's single on-prem GPU box serving one doctor at a time on a slow connection is exactly the workload Ollama is built for — quick to set up, low operational overhead, and the GGUF format's aggressive quantization means it can run on a GPU (or even CPU) that would never hold an fp16 checkpoint.
Layer 3b: vLLM — PagedAttention and the KV-Cache Memory Wall
Now scale the scenario up: a state health department wants one central GPU cluster serving triage sessions for every PHC in the state simultaneously — hundreds of concurrent conversations, not one. This is where the KV cache, not the model weights, becomes the memory bottleneck.
Every token a transformer generates requires storing that token's key and value vectors, for every layer, so future tokens can attend back to it — the KV cache. Using the Llama-2-7B architecture's published dimensions (Touvron et al., 2023: 32 layers, hidden size 4096), and fp16 storage for cache entries:
layers, hidden, bytes_per_val = 32, 4096, 2 # Llama-2-7B, fp16 KV cache
kv_per_token = 2 * layers * hidden * bytes_per_val # 2 for K and V
print(kv_per_token, "bytes") # 524288 (512 KiB)
seq_len, batch = 2048, 32
kv_per_seq = kv_per_token * seq_len
kv_total = kv_per_seq * batch
print(kv_per_seq / 1024**3, "GiB per sequence") # 1.0
print(kv_total / 1024**3, "GiB for the batch") # 32.0
Trace it: kv_per_token = 2 × 32 × 4096 × 2 = 524288 bytes, exactly 512 KiB — every generated token costs half a megabyte of GPU memory just to remember its own key/value pair, independent of the 4.5-bit-per-weight trick from the previous section, because the KV cache is computed at inference time in full precision, not stored on disk. One conversation with a 2048-token context therefore needs 524288 × 2048 = 1,073,741,824 bytes, exactly 1 GiB (2^30, since 512 × 2048 = 2^9 × 2^11 = 2^20 × 2^10). Thirty-two simultaneous conversations at that context length need 32 GiB of KV cache alone — before a single weight is loaded, and already past what a 24 GB consumer GPU can hold.
The standard fix before 2023 was to over-allocate a contiguous memory block for the maximum possible sequence length per request, which wastes enormous amounts of memory to internal fragmentation whenever a sequence finishes early or a batch mixes short and long prompts. Kwon et al. (SOSP 2023) introduced PagedAttention, the mechanism vLLM is built around: split the KV cache into small fixed-size blocks (analogous to OS virtual-memory pages), maintain a per-sequence block table mapping logical token positions to physical blocks, and allocate blocks on demand rather than reserving a worst-case contiguous span. A sequence's blocks need not be physically contiguous — this is the exact same trick that lets an operating system run more processes than it has contiguous free RAM for. It also lets vLLM share blocks across requests that have an identical prompt prefix (a common system prompt, like the triage assistant's instructions above) via copy-on-write, so the shared prefix's KV cache is computed and stored once, not once per conversation.
The second half of vLLM's throughput advantage is continuous batching (also called iteration-level scheduling), an idea from Orca (Yu et al., OSDI 2022) that vLLM adopted and combined with PagedAttention. Static batching waits for every sequence in a batch to finish before admitting new requests — if 31 short replies finish while one long one is still generating, the GPU sits mostly idle waiting on that one sequence, or padding wastes compute on already-finished slots. Continuous batching instead makes the scheduling decision every decoding step: the instant a sequence finishes, its slot (and its freed KV blocks) is handed to a new incoming request on the very next iteration. The GPU is never waiting on the slowest sequence in an arbitrary batch boundary.
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf", dtype="float16")
params = SamplingParams(temperature=0.3, max_tokens=200)
prompts = [
"Patient reports fever and cough for three days.",
"Patient reports abdominal pain since this morning.",
]
outputs = llm.generate(prompts, params) # vLLM schedules and pages internally
for out in outputs:
print(out.outputs[0].text)
Everything that made PagedAttention necessary — the scheduling, the block table, the copy-on-write prefix sharing — happens inside llm.generate(); the caller just submits prompts and gets results back, whether it submits 2 prompts or 2,000.
Misconception: "4-Bit Quantization Is Basically Free"
The Q4_0 derivation above makes a 3.56× size reduction look like a strictly better deal with no downside — smaller file, same model, why wouldn't you always quantize as aggressively as possible? The correction: quantization is lossy compression, and the loss shows up as measurably higher perplexity, i.e., the model becomes less confident (and, past a threshold, less accurate) at predicting the next token. Rounding every weight to the nearest of 16 possible 4-bit levels within a block discards real information, and that loss compounds through 32 transformer layers of matrix multiplication. This is precisely why the k-quants exist: llama.cpp's own documentation and community perplexity benchmarks consistently show plain Q4_0 sitting further from the fp16 baseline than Q4_K_M, because Q4_K_M spends a few extra tenths of a bit per weight on the specific tensors most responsible for that gap. There is a real accuracy-versus-memory tradeoff curve here, not a free lunch — for a clinical triage assistant, where a subtly wrong follow-up question has real consequences, that curve is exactly what you'd want to benchmark (against a held-out set of real triage transcripts) before picking a quantization level, rather than defaulting to the smallest file that fits on the GPU you happen to have.
Diagram: From Hub Checkpoint to Two Serving Engines
Active Recall
Q1. A repository on the Hub offers both a pytorch_model.bin and a model.safetensors for the same checkpoint. You don't fully trust the uploader. Which do you load, and why, precisely?
Q2. A 3-billion-parameter model is quantized with Q8_0: 32-weight blocks, each weight an 8-bit code, one fp16 scale per block. Derive the bits per weight and the resulting file size.
Q3. A hospital IT team wants to serve exactly one doctor's triage sessions on a single GPU workstation with no other load on it. Would you recommend Ollama or vLLM, and what specific property of vLLM's design goes unused in this deployment?
Q4. Starting from the worked KV-cache example (Llama-2-7B, 2048-token sequences, batch of 32 → 32 GiB), the state health department now wants to double the context window to 4096 tokens (for longer patient histories) and raise the target concurrency to 64 simultaneous sessions. Trace the full effect on KV-cache memory, and name two distinct engineering responses available if the resulting number exceeds your GPU budget.
Q5. Explain, using the Q4_0 bit-accounting from this chapter, why "Q4" quantization does not mean the file is exactly one quarter the size of the fp16 original.
Q6. A benchmark shows a Q4_0-quantized model scoring visibly worse than a Q4_K_M-quantized version of the same model on a held-out perplexity test, despite Q4_0 being smaller. Explain the mechanism responsible, referencing what k-quants do differently.
A1. Load model.safetensors. A .bin checkpoint is a Python pickle archive, and unpickling executes arbitrary code embedded in the file — loading an untrusted .bin is equivalent to running the uploader's code on your machine. Safetensors stores a plain JSON header (tensor name → dtype/shape/offset) followed by a raw byte buffer with no executable content, so a malicious uploader has no code-execution path through the format itself, regardless of how much you trust them.
A2. Block bits = 32 weights × 8 bits + 16-bit scale = 256 + 16 = 272 bits per 32 weights → 272 ÷ 32 = 8.5 bits/weight. File size = 3,000,000,000 × 8.5 ÷ 8 = 3,187,500,000 bytes ≈ 3.19 GB. Sanity check: compression ratio vs fp16 is 16 ÷ 8.5 ≈ 1.88×, much less aggressive than Q4_0's 3.56× — consistent with Q8_0 being the "high fidelity, modest compression" option in the GGUF family.
A3. Ollama. Its single-stream design is a non-issue when there is only ever one conversation active, and its simpler setup (one GGUF file, one Modelfile, no cluster scheduler) minimizes operational burden for a single-workstation deployment. vLLM's unused property here is continuous batching (and, by extension, the KV-block sharing across concurrent sequences) — there is only one sequence, so there is nothing to batch or share blocks between; PagedAttention's memory-efficiency advantage only pays off once multiple sequences are competing for GPU memory simultaneously.
A4. Per-token KV cost is unchanged (524,288 bytes) since it depends only on the architecture, not on sequence length or batch size. Per-sequence cost doubles with context length: 524,288 × 4096 = 2,147,483,648 bytes = 2 GiB (double the earlier 1 GiB, as expected since KV cost scales linearly with token count). Total cost then also scales with the doubled batch (32→64): 2 GiB × 64 = 128 GiB — a 4× increase overall (2× from context, 2× from batch), not the 2× a student might guess from changing "only" the context window; the ripple hits both multiplicands in kv_total = kv_per_token × seq_len × batch. 128 GiB exceeds a single A100-80GB's 80 GiB, and leaves only 32 GiB of headroom on a dual-A100-80GB node's 160 GiB combined VRAM — not enough once model weights and activation memory are added. Two responses: (1) store the KV cache in a lower-precision format (e.g., fp8 instead of fp16), halving every number above, or (2) shard the KV cache and the model itself across more GPUs using tensor parallelism, which is what vLLM's multi-GPU deployment mode is for — neither is available to a single-instance Ollama deployment.
A5. "Q4" names the width of the per-weight code (4 bits), not the average bits-per-weight of the file, because every block also carries a 16-bit scale shared across only 32 weights. That scale overhead — 16 bits spread over 32 weights, i.e. 0.5 bits/weight — pushes the true average to 4.5 bits, not 4. The smaller the block size, the larger this relative overhead becomes (more scales per weight); the larger the block, the closer the average creeps toward the nominal 4 bits, at the cost of each scale having to fit a wider range of weight magnitudes.
A6. Uniform 4-bit rounding in Q4_0 applies the same precision to every tensor, including the specific attention-output and feed-forward projection matrices that turn out to be disproportionately sensitive to rounding error — small perturbations there propagate and amplify through the remaining transformer layers. Q4_K_M's k-quant scheme identifies those sensitive tensors and stores them at 5- or 6-bit precision instead, spending a modest amount of extra file size (pushing the average from 4.5 toward roughly 4.8 bits/weight) specifically where it reduces the compounding error the most, rather than spending it uniformly across all weights where most of it would be wasted on tensors that were already tolerant of 4-bit rounding.
Think About It
Think about this: How would you explain open-source ai ecosystem: huggingface, ollama, vllm, and gguf quantization to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where open-source ai ecosystem: huggingface, ollama, vllm, and gguf quantization is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting open-source ai ecosystem: huggingface, ollama, vllm, and gguf quantization to at least 3 other topics you have studied.