Suppose a Bengaluru-based platform decides to self-host a DeepSeek-V3-class open-weight model to handle customer support across a dozen Indian languages for a user base the size of a national telecom operator. The model card says 671 billion total parameters, 37 billion activated per token. The infra lead's first instinct — "we only need enough GPU memory for the 37 billion active ones" — turns out to be wrong, and figuring out why is the actual engineering problem this chapter is about. A second problem follows immediately: language-specific traffic is not uniform. A regional-language product launch sends a burst of Hindi and Tamil queries through the router, and because experts in a trained MoE model empirically specialize (some skew toward code, some toward a language family, some toward formatting), one or two experts get hammered while others sit idle. Somewhere in that cluster, a buffer overflows, and the system has to decide, in microseconds, what to do with a token it cannot process on time.
A companion chapter on sparse gating networks covers how a router picks which experts a token goes to — the softmax-over-experts, top-k selection, and the auxiliary load-balancing loss that nudges training toward even usage. That machinery answers "which expert does this token want?" It says nothing about what happens when four thousand tokens all want the same GPU at once, or how you physically move activations between devices that don't share memory. That is the systems layer this chapter teaches: expert parallelism, capacity-limited dispatch, communication cost, and the load-balancing mechanism DeepSeek-V3 uses instead of an auxiliary loss.
Why routing becomes a placement problem at scale
A dense transformer of 37B parameters fits, in bf16, in about 74 GB — one or two modern GPUs, replicated across a cluster with ordinary data parallelism. A 671B-parameter MoE model does not fit that way, because the router can send any token to any expert, and different tokens in the same batch typically activate different experts. Over a full batch, nearly every expert sees traffic. That means every GPU in the serving cluster must have some route — direct or indirect — to every expert's weights, even though any single token only touches a handful of them. This is a placement problem, not a compute problem, and it is the reason large MoE models need a parallelism strategy the dense-model world does not: expert parallelism (EP).
Three parallelism strategies coexist in a large MoE deployment, and they solve different bottlenecks:
Data parallelism (DP) replicates the entire model on each device group and splits the batch across replicas — it doesn't touch the memory problem at all, since every replica still needs every expert.
Tensor parallelism (TP) splits an individual weight matrix (say, one expert's up-projection) across GPUs and reassembles the result with an all-reduce after every layer. TP shrinks the matrix a single GPU must hold, but it shrinks it for every expert on that GPU equally — if a GPU is still assigned all 256 experts, TP doesn't reduce the total memory footprint per device the way sparsity promises.
Expert parallelism (EP) instead assigns whole experts to specific GPUs — GPU 0 physically holds experts 0 through 31, GPU 1 holds 32 through 63, and so on. Per-GPU memory now shrinks in proportion to the EP degree, which is the actual payoff of sparsity. The cost is that a token's activation has to physically travel, over the interconnect, to whichever GPU owns its chosen expert, get processed, and travel back. That round trip is implemented as two collective communication operations per MoE layer: an all-to-all dispatch (every GPU sends its tokens to the GPUs holding their assigned experts) and an all-to-all combine (results travel back to the tokens' home GPUs to be merged with the rest of that token's forward pass). Production systems (GShard: Lepikhin et al., 2021; Switch Transformer: Fedus, Zoph & Shazeer, 2022) run TP inside an expert-parallel group when a single expert is still too large for one GPU, and EP across groups — the two are complementary, not competing.
Capacity factor: the buffer that decides who gets dropped
Dispatch only works if each expert's inbox has a fixed, pre-allocated size — you cannot allocate GPU memory dynamically mid-forward-pass. That fixed size is the expert's capacity, and it is set relative to the average load, not the worst case:
capacity = (tokens_per_batch × top_k / num_experts) × capacity_factor
The first factor, tokens_per_batch × top_k / num_experts, is what each expert would receive under perfectly uniform routing. The capacity_factor (Switch Transformer trains with values around 1.0–1.25) is slack above that average, bought deliberately, because routing is never perfectly uniform. When an expert's actual demand exceeds its capacity, the excess tokens are not queued or retried — they are dropped for that layer. A dropped token's assignment to that expert is simply zeroed out before the all-to-all transfer even happens (both GShard and Switch Transformer build the dispatch mask this way, so a dropped token costs no bandwidth and no compute, not just no compute). If a token's every assigned expert drops it — both experts, in a top-2 scheme — the token receives no expert contribution at all for that layer and passes through purely via the transformer's residual connection, identical to what would happen if the entire MoE block were skipped.
Worked example: sizing a dispatch under real imbalance
Take a single MoE layer processing a global batch of T = 16,384 tokens, routed with top_k = 2 across E = 8 experts, one expert per GPU, with capacity_factor = 1.25. (These are round numbers chosen to keep the arithmetic exact and traceable — not DeepSeek-V3's or Mixtral's actual configuration, which are noted separately below.)
def expert_capacity(tokens_per_batch, num_experts, capacity_factor, top_k=1):
"""Per-expert buffer size (Switch Transformer / GShard formula)."""
return int((tokens_per_batch * top_k / num_experts) * capacity_factor)
def dispatch_report(expert_token_counts, capacity):
"""Given how many tokens *want* each expert, report accepted/dropped."""
report = {}
for expert_id, wanted in enumerate(expert_token_counts):
accepted = min(wanted, capacity)
dropped = wanted - accepted
report[expert_id] = {"wanted": wanted, "accepted": accepted, "dropped": dropped}
return report
T = 16384
E = 8
TOP_K = 2
CAPACITY_FACTOR = 1.25
cap = expert_capacity(T, E, CAPACITY_FACTOR, TOP_K)
print(cap)
# Demand skewed by a regional-language traffic spike hitting expert 3.
# The eight values must sum to T * TOP_K = 32768 — a useful sanity check.
wanted_per_expert = [4096, 4096, 4096, 5600, 4096, 4096, 4096, 2592]
report = dispatch_report(wanted_per_expert, cap)
print(report[3])
Trace it by hand first. Uniform demand per expert would be 16384 × 2 / 8 = 4096 tokens; with 25% slack, cap = 4096 × 1.25 = 5120. So print(cap) prints 5120. Expert index 3 is overloaded to 5,600 tokens (the language-spike scenario), while expert 7 is correspondingly underloaded to 2,592 so the eight demand values still sum to 32,768. Inside dispatch_report, expert 3's accepted count is min(5600, 5120) = 5120, and dropped is 5600 - 5120 = 480. So print(report[3]) prints {'wanted': 5600, 'accepted': 5120, 'dropped': 480} — 480 tokens, out of 5,600 that wanted expert 3, get no contribution from it this layer.
Now price the communication. Each token's hidden vector has d_model = 4096 entries; in bf16 that is 2 bytes each, so 8,192 bytes per token per hop. Only successfully dispatched tokens are transferred (dropped ones are masked out before the all-to-all, costing nothing). Total dispatch attempts are T × top_k = 32,768; with 480 dropped and every other expert under capacity, 32,288 actually cross the network: 32,288 × 8,192 bytes ≈ 252.2 MiB. The combine step ships the same 32,288 results back, another ≈252.2 MiB, for roughly 504 MiB per MoE layer, per forward step, moving over NVLink or InfiniBand rather than through any FLOP unit. DeepSeek-V3's report states the first 3 of its 61 transformer layers are kept dense and the remaining 58 use MoE; scaling this per-layer estimate by 58 layers — using this section's toy T/E/top_k/d_model, not DeepSeek-V3's actual routing config (real d_model = 7,168, 256 routed experts, top-8) — gives ≈28.5 GiB of illustrative all-to-all traffic for a single forward step, before a single matrix multiply inside an expert has run. This is why, at frontier scale, all-to-all bandwidth and topology-aware EP-group placement are first-order engineering concerns, entirely separate from FLOP accounting.
The dispatch–compute–combine pipeline
The diagram below works a smaller, fully-traceable version of the same mechanism: 6 tokens, top-2 routing, 4 experts (one per GPU), capacity 2 tokens per expert. Solid lines are accepted dispatches; dashed lines are dropped (blocked before transfer, by the FIFO-ordered capacity check — whichever tokens arrive first at an expert fill its two slots, later ones are cut). Token t6 is the instructive case: both of its assigned experts (E1 and E3) are already full when it arrives, so it is dropped twice and receives zero expert computation for this layer — the dashed red arc shows it skipping straight to the output via the residual connection, exactly as a token dropped at every assignment does in a real deployment.
DeepSeek-V3's fix: auxiliary-loss-free load balancing
The classic remedy for imbalance, used since GShard and Switch Transformer, is an auxiliary loss added during training: L_aux = α · N · ∑_i f_i P_i, where N is the number of experts, f_i is the fraction of tokens actually routed to expert i, and P_i is the router's average softmax probability for expert i. The N factor keeps the loss's magnitude invariant to the number of experts — under uniform routing, both f_i and P_i scale roughly as 1/N, so without the N multiplier the summed product would shrink toward 1/N and α would need re-tuning every time the expert count changed. Minimizing this term pushes routing toward uniformity. It works, but α is a genuine trade-off knob: set it too low and imbalance (and dropping) persists; set it too high and the balancing term starts fighting the language-modeling gradient, measurably hurting model quality — because the same backward pass that is trying to make the model predict the next token correctly is also being pulled toward routing more evenly, and those two objectives are not always aligned for a given token.
DeepSeek-V3 (DeepSeek-AI, 2024) removes that trade-off by not touching the loss at all. Building on the fine-grained-expert and shared-expert design introduced in DeepSeekMoE (Dai et al., 2024), each routed expert i gets a bias term b_i that is added directly to its routing score before the top-k selection is made — so the bias changes which experts get picked, not how the model is scored on next-token prediction. After every training step, the system checks each expert's load against its target and nudges the bias: overloaded experts get b_i decreased by a small fixed step, underloaded experts get it increased, which makes them relatively more attractive to the router on the next step. Crucially, the bias only affects the discrete top-k selection; the weight used to combine each selected expert's output back into the token's representation still comes from the original, unbiased affinity score. Load balancing becomes a control loop acting purely on routing decisions, fully decoupled from the gradient that trains the model to predict text — no α to tune, and no quality trade-off purchased for the sake of even GPU utilization.
A common misconception
The natural reading of "671B total, 37B active" is that the model only needs 37B parameters' worth of GPU memory to serve — after all, only a handful of experts fire per token. This is wrong, and the reason is worth stating precisely: active parameters set compute (FLOPs) per token; total parameters set memory. Because different tokens in the same batch — let alone different batches over time — route to different experts, essentially every expert sees traffic across a serving workload. Every expert's weights therefore have to be resident somewhere in the cluster's GPU memory at all times, which means the aggregate memory requirement across all devices is driven by the full 671B parameters (roughly 1.3 TB in bf16, before KV cache and activations), not the 37B a single token touches. Expert parallelism reduces the memory each individual GPU has to hold — split 671B parameters' worth of experts across 32 GPUs and each holds roughly 21B parameters' worth — but it does not reduce the total footprint the cluster needs, and it introduces the all-to-all communication cost worked out above as the price for that split. The actual latency and throughput win from sparsity is real, but it shows up in FLOPs per token (37B-active compute instead of 671B-dense compute), not in a smaller total memory bill. Mixtral 8x7B (Jiang et al., 2024) makes the same point at a smaller scale: roughly 46.7B total parameters but only about 12.9B active per token — a serving cluster for it still has to hold all 46.7B, distributed across however many GPUs its expert-parallel group uses.
Active recall
Q1. A layer routes T = 8,192 tokens with top_k = 2 across E = 16 experts, capacity_factor = 1.5. Compute the per-expert capacity.
Q2. Under the setup in Q1, one expert actually receives 1,800 tokens. How many are dropped, and — if a dropped token was that token's only surviving assignment — what does that token's output for the layer become?
Q3. Using this chapter's larger worked example (T = 16,384, E = 8, top_k = 2, capacity_factor = 1.25, d_model = 4,096, bf16, expert index 3 wanting 5,600 tokens, expert index 7 wanting 2,592), the systems team raises capacity_factor to 2.0 to eliminate drops during a festive-season traffic surge, changing nothing else. Trace the full effect on: (a) the new capacity, (b) whether expert 3 still drops tokens, (c) each expert's reserved buffer memory in MiB, (d) total all-to-all communication volume for the layer, and (e) expert 7's buffer utilization.
Q4. Why doesn't tensor parallelism alone solve the same memory problem that expert parallelism solves, even though both "split" the model across GPUs?
Q5. A classmate says: "DeepSeek-V3 has 37B active parameters, so it only needs about 37B parameters of GPU memory to serve." What is wrong with this claim, and what actually determines the cluster's memory requirement?
Q6. How does DeepSeek-V3's bias-adjustment load balancing differ from the classic auxiliary-loss approach in what it directly modifies, and why does that avoid the auxiliary loss's quality trade-off?
Answers.
A1. capacity = (T × top_k / E) × capacity_factor = (8192 × 2 / 16) × 1.5 = 1024 × 1.5 = 1536 tokens.
A2. Dropped = 1800 - 1536 = 264 tokens. If a dropped assignment was that token's only surviving one (its other top-2 assignment was also dropped or it was single-routed), the token receives zero expert contribution for the layer and its output is simply the residual-stream value passed through unchanged — mathematically identical to skipping the MoE block for that token.
A3. (a) New capacity = (16384 × 2 / 8) × 2.0 = 4096 × 2.0 = 8192. (b) Expert 3's demand of 5,600 is now below 8,192, so zero tokens are dropped there (or anywhere else, since no other expert exceeded even the old capacity of 5,120). (c) Buffer memory per expert = capacity × d_model × 2 bytes: at capacity 5,120 that was 5120 × 4096 × 2 = 41,943,040 bytes = 40 MiB; at capacity 8,192 it becomes 8192 × 4096 × 2 = 67,108,864 bytes = 64 MiB — a 1.6× increase, exactly matching the capacity_factor ratio (2.0/1.25 = 1.6), on every one of the 8 experts, not just expert 3. (d) Communication volume barely moves: previously 32,288 of 32,768 dispatch attempts crossed the network (≈252.2 MiB dispatch + ≈252.2 MiB combine ≈ 504.5 MiB/layer); now all 32,768 succeed (≈256 MiB + ≈256 MiB = 512 MiB/layer) — a rise of under 1.5%, because communication volume is set by T × top_k × d_model, which capacity_factor barely touches once drops were already a small fraction. Across 58 MoE layers this moves total traffic from ≈28.5 GiB to ≈29.0 GiB per forward step — a small absolute change. (e) Expert 7's demand (2,592) never changed — only the buffer ceiling did — so its utilization actually falls, from 2592/5120 ≈ 50.6% to 2592/8192 ≈ 31.6%. Raising capacity_factor didn't fix the underlying imbalance between expert 3 and expert 7; it just paid for expert 3's overload with more reserved (and increasingly idle, for expert 7) buffer memory instead of paying for it with dropped tokens. The imbalance itself is only fixed by better routing — which is exactly what DeepSeek-V3's bias mechanism targets.
A4. Tensor parallelism shards a matrix multiply that every GPU in the TP group still participates in for every expert assigned to that group — it reduces the size of each shard, but every GPU in the group still needs a slice of every expert's weights, so the number of experts a GPU is "responsible for" doesn't shrink. Expert parallelism instead assigns entire experts to specific GPUs, so a GPU holding 1/32 of the experts only stores 1/32 of the total expert weight, genuinely shrinking its memory footprint in proportion to how many experts it's skipped. TP reduces per-matrix memory; EP reduces which matrices you have to hold at all. Production systems combine them: TP within a group when even one expert doesn't fit a single GPU, EP across groups to spread the full set of experts.
A5. The claim conflates active parameters (which determine FLOPs per token) with total parameters (which determine memory footprint). Because different tokens route to different experts, effectively all 671B parameters must be resident somewhere across the serving cluster's GPUs at all times, not just the 37B a given token happens to use — the cluster's aggregate memory requirement is set by the 671B total, roughly 1.3 TB in bf16 before KV cache and activations. Expert parallelism can shrink the memory any single GPU carries (by holding only a subset of experts), but it doesn't shrink the total the cluster needs, and that per-GPU split is precisely what creates the all-to-all communication cost.
A6. The classic auxiliary loss adds a balancing term directly to the training objective, so its strength (α) trades off against the primary language-modeling gradient — too strong, and the model is pulled away from predicting text correctly for the sake of even routing. DeepSeek-V3's bias term instead modifies only the routing score used for top-k selection, adjusted step-by-step from observed expert load, while the weight used to combine a selected expert's output is still computed from the original, unbiased affinity score. Because the bias never enters the loss the model is trained against, load balancing becomes a separate control loop over routing decisions rather than a competing training objective — removing the need to tune a trade-off coefficient and avoiding the quality cost that a strong auxiliary loss can impose.
Think About It
Think about this: How would you explain mixture of experts at scale to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where mixture of experts at scale is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting mixture of experts at scale to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind mixture of experts at scale, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.