In December 2024, DeepSeek-AI released a technical report for DeepSeek-V3: 671 billion total parameters, but only 37 billion of them touched for any single token. That gap — 634 billion parameters sitting in the model but idle on a given forward pass — is not a rounding curiosity. It is the entire economic argument for Mixture of Experts (MoE) at frontier scale, and it is also where most of the engineering difficulty lives. Knowing how a gating network picks which experts to activate (the subject of the sibling chapter on sparse gating) tells you almost nothing about why serving a 671B-parameter model is hard, or why a naive MoE implementation can lose most of its theoretical speedup the moment it leaves a single GPU. This chapter is about that second problem: what "efficient scaling" actually costs in memory, communication, and lost tokens once you put a Mixture-of-Experts model on real hardware — and the specific architectural moves (fine-grained experts, expert parallelism, capacity-limited routing, auxiliary-loss-free balancing) that production systems use to claw the efficiency back.
The distinction MoE is built on: compute vs. memory
A dense transformer has one number that describes its size: parameter count. Double the parameters, and — holding everything else fixed — you roughly double both the memory needed to hold the model and the FLOPs needed to run a forward pass, because every parameter participates in every token's computation. MoE breaks that equivalence on purpose. In an MoE layer, the feed-forward block is replicated E times ("experts"), and a router sends each token to only k of them (commonly k=1 or k=2 for older designs, k=8 in DeepSeek-V3's 256-expert layer). The FLOPs a token costs are governed by k, not E. The memory the model occupies is governed by E, not k.
This is where the first and most consequential misconception sits, and it is worth naming explicitly before going further: a Mixture-of-Experts model does not use less GPU memory than a dense model with the same total parameter count, even though only a fraction of its parameters run on any given token. Students (and, historically, some early production teams) reason: "only 2 of 8 experts fire per token, so surely I only need to keep 2 of 8 experts loaded." This fails because routing is data-dependent and computed per token, per layer, at runtime. Token 1 in a batch might route to experts {2, 5}; token 2 in the same batch might route to {1, 7}. Across a batch of any realistic size, essentially every expert gets hit by something. There is no way to know in advance which experts a given batch will need, so all E experts — all 671B parameters, in DeepSeek-V3's case — must be resident in the cluster's aggregate GPU memory at all times. MoE buys you a discount on FLOPs (compute, and therefore training and inference latency for a fixed batch), not on the memory footprint of the model. This is precisely why DeepSeek-V3 needed hundreds of GPUs' worth of high-bandwidth memory to serve, despite activating "only" 37B parameters per token — 37B governs the arithmetic; 671B governs the storage.
Expert parallelism: where the model actually lives
Once you accept that all experts must be resident somewhere, the systems question becomes: resident where? The answer used in every large-scale MoE deployment since Lepikhin et al.'s 2020 GShard paper (Google) is expert parallelism: shard the experts themselves across devices, so GPU 0 holds experts {0, 1}, GPU 1 holds experts {2, 3}, and so on, rather than replicating every expert on every GPU. This is a different axis from the data parallelism and tensor parallelism a Grade 11/12 student has likely already seen — it doesn't split a single matrix multiply across devices, and it doesn't just replicate the whole model and split the batch. It splits the *set of experts*.
Expert parallelism creates a routing problem that has no analogue in dense-model training: a token living on GPU 0, after the router decides it belongs to expert 3 (which lives on GPU 1), has to physically travel to GPU 1, get processed by that expert's feed-forward network, and travel back. This round trip is implemented as a collective communication operation called all-to-all: every GPU simultaneously sends some of its tokens' hidden states to every other GPU and receives tokens routed to the experts it holds. Two all-to-all passes happen per MoE layer — one "dispatch" (send tokens to their assigned experts) and one "combine" (send the expert outputs back to each token's original GPU so the model can continue to the next layer). This is the part sparse-gating-network coverage typically skips, and it is the part that determines whether a real deployment hits its theoretical FLOPs savings or not: all-to-all is a network-bandwidth-bound operation, not a compute-bound one, and on a cluster with poor interconnect it can dominate wall-clock time even though the actual expert FFN math is cheap.
Capacity factor: the second cost of scaling
All-to-all also forces a decision that dense models never have to make: how much buffer space to allocate per expert, before you even know how many tokens will route to it. If routing were perfectly uniform, an MoE layer with 8 experts, top-2 routing, and 4,096 tokens per step would send exactly 1,024 tokens to each expert (4,096 tokens × 2 routes ÷ 8 experts). But routing is learned and data-dependent, not uniform — some experts specialize and get overloaded, a phenomenon documented as early as the original Switch Transformer paper (Fedus, Zoph & Shazeer, 2021/2022), which found individual experts could receive dramatically more than their "fair share" of tokens.
Because the GPU buffer holding each expert's input must be a fixed size (allocated before the forward pass runs), every production MoE defines a capacity factor: a multiplier on the expected uniform load that sets how much slack each expert's buffer gets. Capacity = (tokens per step × k ÷ E) × capacity_factor. Tokens routed to an expert beyond its capacity don't get silently queued — they overflow, and the standard behavior (used since GShard and Switch Transformer) is to drop them: that token skips this expert for this layer, typically passed through via a residual/skip connection instead of a real expert transformation. A higher capacity factor means fewer drops but more wasted compute and memory on padding (buffers sit partially empty on the common case); a lower capacity factor means tighter buffers but more dropped tokens and worse model quality.
Worked example: how many tokens actually get dropped
Take a training step with sequence length 512, batch size 8 (so 4,096 tokens total), routed with top-2 gating across 8 experts, expert-parallelized one-expert-per-GPU, with a capacity factor of 1.25 — a realistic industry-standard value.
tokens_per_step = 512 * 8 # 4096
k_top = 2 # top-2 routing
E = 8 # experts (= GPUs, one expert each)
capacity_factor = 1.25
total_assignments = tokens_per_step * k_top # 4096 * 2 = 8192
expected_per_expert = total_assignments / E # 8192 / 8 = 1024.0
capacity = expected_per_expert * capacity_factor # 1024 * 1.25 = 1280.0
Under perfectly uniform routing, every expert would get exactly 1,024 tokens against a capacity of 1,280 — comfortable headroom. But suppose (as Switch Transformer's authors observed happens in practice) one expert specializes heavily and draws 1,500 tokens in this step:
tokens_per_step = 4096 # from above
total_assignments = 8192 # from above
capacity = 1280.0 # from above
overloaded_expert_load = 1500
dropped = overloaded_expert_load - capacity # 1500 - 1280.0 = 220.0
drop_rate_of_that_expert = dropped / overloaded_expert_load # 220/1500 = 14.67%
drop_rate_of_total_routing = dropped / total_assignments # 220/8192 = 2.69%
drop_rate_of_all_tokens = dropped / tokens_per_step # 220/4096 = 5.37%
Running this gives 220 dropped tokens on that one expert — about 14.7% of what it was asked to process, or 2.7% of all top-2 routing decisions in the step. Those 220 tokens pass through this layer via the residual connection instead of getting a real expert transformation. This is not a bug being tolerated; it is a deliberate, tunable trade — the alternative to dropping is either a much larger (and therefore memory-wasting) capacity factor, or a slower, non-fixed-size buffer that breaks the parallel, batched execution GPUs need to be fast. The capacity factor is, in a real sense, the "how much scaling efficiency am I willing to trade for how much quality" knob of the whole architecture.
Fine-grained experts and shared experts: DeepSeekMoE's answer
Dai et al. (DeepSeek-AI, 2024), in the DeepSeekMoE paper, identified a subtler inefficiency in the original coarse-grained MoE design (few, large experts): when an expert is large and general-purpose, tokens that need only a narrow slice of its knowledge still pay for the whole expert's parameters and FLOPs, and knowledge that is genuinely common across all tokens (basic syntax, for instance) gets redundantly re-learned inside every expert instead of being shared. DeepSeekMoE's fix has two parts. First, fine-grained expert segmentation: instead of, say, 8 experts of a given total FFN width, split into many more, much smaller experts (DeepSeek-V3 uses 256 routed experts per MoE layer) while activating a larger absolute number of them per token (top-8 of 256, rather than top-2 of 8). This gives the router combinatorially many more ways to compose a token's processing — C(256,8) possible expert subsets versus C(8,2) — letting experts specialize more sharply without wasting capacity on tokens that only need part of what a coarse expert offers. Second, shared expert isolation: a small number of experts (DeepSeek-V3 uses 1) are always active for every token, regardless of routing, to absorb the common, redundant knowledge that would otherwise be duplicated across many routed experts. This is a direct efficiency move: it lets the routed experts specialize further, because they no longer need to each independently re-encode general-purpose patterns.
Balancing load without fighting the loss function
Getting routing to spread load evenly (so the capacity-factor drop problem above stays small) has historically required an auxiliary load-balancing loss — an extra term added to the training objective (introduced in Shazeer et al., 2017, and used in Switch Transformer) that penalizes the router for sending too many tokens to the same expert. The problem: this auxiliary loss competes with the actual language-modeling loss. Weight it too heavily and the model spends capacity balancing instead of predicting text well; weight it too lightly and load stays skewed, and the capacity-factor drop problem gets worse. DeepSeek-V3's technical report describes an auxiliary-loss-free alternative: instead of an extra loss term, each expert carries a learned bias added to its routing score, and that bias is nudged up or down after each training step based purely on whether the expert was over- or under-loaded — a control-loop adjustment that steers routing toward balance without ever touching the gradient of the main training objective. It is a small change with a real payoff: balance is maintained as a side effect of a simple bookkeeping rule rather than as a tug-of-war baked into backpropagation.
Reading the diagram
The figure below traces one MoE layer's forward pass under expert parallelism, using a small, self-contained example (4 GPUs, one expert each, top-1 routing, capacity = 4 tokens) so the mechanism stays legible — the numbers here are illustrative, not the 4,096-token example above. A batch of 15 tokens is routed by the gate; the all-to-all dispatch physically moves each token's hidden state to the GPU holding its assigned expert; each expert processes what fits in its capacity buffer and drops the rest; the all-to-all combine sends the processed results back. GPU 1 draws 6 tokens against a capacity of 4, so 2 are dropped and fall back to a residual pass-through — this is the same mechanism as the 220-token drop in the worked example above, just at a scale you can count by eye.
Active recall
Attempt each question before reading its answer.
- A model has 16 experts per MoE layer with top-2 routing. If you double the number of experts to 32 while keeping top-2 routing and batch size fixed, what happens to (a) FLOPs per token and (b) total model memory?
- Why can't a serving system simply keep only the "popular" experts loaded in GPU memory and evict the rest, the way a cache might evict cold data?
- Using the formula capacity = (tokens_per_step × k ÷ E) × capacity_factor, compute the capacity per expert for tokens_per_step = 8192, k = 2, E = 16, capacity_factor = 1.5.
- Why is all-to-all communication, not the expert FFN matrix multiplies, usually the bottleneck when an MoE model is spread across many GPUs?
- Take the worked example in this chapter (4,096 tokens/step, E = 8, top-2, capacity_factor = 1.25, one overloaded expert receiving 1,500 tokens). If top-k is changed from 2 to 4 with everything else held fixed, recompute: total token-expert assignments, expected tokens per expert, capacity per expert, and the multiplier on both FLOPs per token and all-to-all communication volume relative to the original top-2 setup.
- What problem does DeepSeekMoE's shared-expert isolation solve, and why does auxiliary-loss-free load balancing avoid a weakness of the older auxiliary-loss approach?
Answers
1. FLOPs per token stay essentially unchanged — a token is still processed by exactly 2 experts, and expert FFN width is unchanged, so the arithmetic work per token doesn't depend on how many experts exist in total, only on k. Total model memory roughly doubles, because doubling E means doubling the number of expert FFN parameter sets that must be stored somewhere in the cluster. This is the compute/memory split from the "first principles" section: E controls memory, k controls compute.
2. Because routing is computed per token, per layer, at inference time, based on that token's actual content — which experts a given batch will need is not known until the router runs. There is no stable "popular vs. unpopular" split the way there is with, say, cached web content: over any reasonably sized and diverse batch, most or all experts get hit by something. Evicting an expert risks a cache miss on essentially every batch, so production systems keep all experts resident across the cluster's GPUs (via expert parallelism) rather than trying to cache a subset.
3. Expected tokens/expert = (8192 × 2) ÷ 16 = 16384 ÷ 16 = 1024. Capacity = 1024 × 1.5 = 1536 tokens.
4. The expert FFN computation for a token is a small, fixed-size matrix multiply — cheap, and something GPUs are extremely fast at. All-to-all, by contrast, must physically move every routed token's hidden-state vector across the interconnect (often across whole nodes, not just within one) twice per MoE layer (dispatch and combine), and that data movement is limited by network bandwidth, which grows far more slowly than GPU compute throughput does. As you add more GPUs to hold more experts, the volume of data crossing the network per layer grows too, so communication time can dominate even though the actual FLOPs are small — this is why interconnect quality, not raw GPU count, is often the ceiling on how far MoE scaling helps in practice.
5. With k = 4: total assignments = 4096 × 4 = 16384 (double the original 8192). Expected tokens/expert = 16384 ÷ 8 = 2048 (double the original 1024). Capacity at the same capacity_factor of 1.25 = 2048 × 1.25 = 2560 (double the original 1280) — note the overloaded expert's 1,500-token draw from the original example no longer even exceeds capacity at k=4's more generous buffer, though a proportionally larger overload would still spill over. FLOPs per token double, because each token is now processed by twice as many experts (k appears linearly in both the assignment count and the compute cost). All-to-all communication volume also roughly doubles, since twice as many token-copies must be dispatched to (and combined back from) remote experts — every one of these quantities moves together because they all scale with k, which is exactly the ripple a change to routing width produces throughout the system, not just at the gate.
6. Shared-expert isolation gives a small number of experts (always active, regardless of routing) the job of absorbing knowledge that is genuinely common across most tokens — basic syntax, frequent patterns — so that the routed, specialized experts don't each have to redundantly relearn it. Without this, coarse or fine-grained routed experts waste some of their capacity duplicating the same general-purpose knowledge, cutting into how sharply they can specialize. Auxiliary-loss-free balancing avoids a real weakness of the older approach — an extra loss term added to the main training objective — because that extra term competes with the actual next-token-prediction loss for gradient "attention": weighted too high, the model over-prioritizes balance over quality; too low, and load stays skewed. Adjusting a per-expert routing bias after each step based on observed over/under-load steers the router toward balance directly, without ever perturbing the gradient of the main loss, so balance and prediction quality stop competing.
Think About It
Think about this: How would you explain mixture of experts: scaling models efficiently to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind mixture of experts: scaling models efficiently, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.