Mixture of Experts: How Modern LLMs Scale

Suppose an engineering team at an Indian AI lab has been allocated a slice of GPU compute under the IndiaAI Mission's shared-cluster scheme — say a few dozen H100 nodes, provisioned in advance because H100s are scarce and every GPU-hour is accounted for. They plan to serve an open-weight model built like DeepSeek-V3: 671 billion total parameters, but only 37 billion "activated" for any single token (DeepSeek-AI, 2024). Someone on the team reads that headline ratio and reasons: bf16 weights are 2 bytes each, so 37 billion activated parameters need roughly 37 × 10⁹ × 2 bytes ≈ 74 GB — that almost fits on one 80 GB H100. They provision accordingly. The deployment fails immediately, not with a slow response but with an out-of-memory crash before the first token is generated. The real number is 671 × 10⁹ × 2 bytes ≈ 1,342 GB, roughly 18 times more than they budgeted, needing at minimum 1,342 ÷ 80 ≈ 17 GPUs just to hold the weights — before a single byte of KV cache. That gap between "parameters activated per token" and "parameters that must physically sit in memory" is not a rounding error. It is the central engineering fact of how mixture-of-experts (MoE) models actually scale in production, and it is the subject of this chapter.

If you have already met the softmax router and top-k expert selection in the companion chapter on sparse gating networks, treat that as solved. This chapter starts one layer below the surface: at the point where a training or inference cluster has to physically move activations between GPUs, reserve fixed-size buffers for unpredictable traffic, and decide what to do when a router sends more tokens to an expert than that expert can hold. These are the mechanics that determine whether "sparsity" actually saves you anything, and they are where most of the real engineering decisions in frontier LLM systems — DeepSeek-V3, Mixtral, Switch Transformer, GShard — were made.

Active parameters versus total parameters: the lever MoE actually pulls

Start with a small, concrete MoE feed-forward layer so every number can be checked by hand. Let the model dimension be d_model = 512, and let each expert be a standard two-layer FFN with hidden expansion d_ff = 2048 (an up-projection 512 → 2048 followed by a down-projection 2048 → 512). Ignoring biases, one expert's parameter count is:

• Params per expert = 2 × d_model × d_ff = 2 × 512 × 2048 = 2,097,152 (≈ 2.10M)
• With E = 8 experts total: total expert params = 8 × 2,097,152 = 16,777,216 (≈ 16.78M)
• With top-k = 2 routing: active expert params per token = 2 × 2,097,152 = 4,194,304 (≈ 4.19M)
• Ratio of total to active = 16,777,216 ÷ 4,194,304 = 4.0, which equals E ÷ k = 8 ÷ 2 exactly

That last line is not a coincidence to memorize — it is a structural identity. Because every expert in this design is the same size S, total parameters are E·S and active parameters are k·S, so the ratio is always exactly E/k, independent of how large you make d_model or d_ff. This is the lever: for a fixed compute budget per token (which top-k fixes), you can keep growing E — adding more experts, each still only S parameters — and total model capacity grows linearly with E while per-token FLOPs stay flat. A dense model matching this layer's per-token compute could only ever own 4.19M FFN parameters at this layer; the MoE layer owns 16.78M, four times more "knowledge," for the same forward-pass arithmetic. Scale this identity up to DeepSeek-V3's real numbers — 671B total, 37B active — and the effective ratio is 671 ÷ 37 ≈ 18×: eighteen times the parameter capacity of a dense model that costs the same to run per token (DeepSeek-AI, 2024). That is the entire economic case for MoE in one division.

Expert parallelism: what "activated" costs you when experts live on different GPUs

The parameter-count identity above is a static, single-GPU story. Real frontier models spread their experts across many GPUs — a strategy called expert parallelism, distinct from the data-, tensor-, and pipeline-parallelism you may already know, and introduced at scale by GShard (Lepikhin et al., 2020). Each GPU stores only a shard of the total experts; when a token's router decision doesn't match a locally-hosted expert, the token's hidden-state vector has to physically travel over the network to whichever GPU owns that expert, get processed, and travel back. This exchange is called an all-to-all operation because, in the worst case, every GPU may need to send data to every other GPU.

Extend the earlier example to a real cluster layout: keep E = 8 experts and top-k = 2, but now distribute the 8 experts across G = 4 GPUs (2 experts per GPU), with each GPU also holding T = 4,096 local tokens for this training step, in bf16 (2 bytes per value). Work through the communication volume for the dispatch phase — sending each token's hidden state to its chosen experts' GPUs:

• Global batch across all GPUs: N = T × G = 4,096 × 4 = 16,384 tokens
• Each token dispatches to k = 2 experts, so total dispatch events = N × k = 16,384 × 2 = 32,768
• Bytes per dispatch = d_model × 2 bytes = 512 × 2 = 1,024 bytes (one token's hidden-state vector)
• Total dispatch volume = 32,768 × 1,024 bytes = 33,554,432 bytes = 32 MiB

Once each expert finishes processing its assigned tokens, the results have to travel back to the GPU each token originated from — the combine phase. It moves exactly as many vectors of the same size in the reverse direction, so it costs another 32 MiB. One MoE layer, one training microbatch, forward pass only: 64 MiB of cluster-interconnect traffic that a dense model never has to pay, because a dense model's FFN never needs to ask "which GPU has the right weights for this token." Backward pass needs a comparable exchange to route gradients back through the same paths, so a full training step on this one layer costs roughly double again. Multiply by every MoE layer in the network and this communication becomes a first-order cost that a systems team sizes network interconnect (NVLink, InfiniBand) around — not an afterthought.

Capacity factor: why a router's decision can be overruled by hardware

All-to-all collectives run efficiently only when every GPU knows in advance exactly how many bytes it will send and receive — the underlying communication primitives want fixed, statically-known buffer shapes. But a router's decisions are made per-token, dynamically, and there is no way to guarantee in advance that exactly the same number of tokens will pick each expert. Real batches are lumpy: some experts, especially in early training or on skewed input distributions, get more traffic than others. Systems solve this with a capacity factor — a hyperparameter C that fixes each expert's buffer to hold C times its "fair share" of tokens, and simply drops whatever doesn't fit (Fedus, Zoph & Shazeer, 2021; Lepikhin et al., 2020).

Work a small, self-contained version of this. Suppose in some microbatch, 4 experts share a fair-share target of 100 tokens each (400 dispatch decisions total), and the actual routing came out imbalanced: E1 = 130, E2 = 90, E3 = 95, E4 = 85 (these sum to 400, confirming no tokens are unaccounted for). With capacity factor C = 1.2, each expert's hard buffer limit is:

• cap = C × target = 1.2 × 100 = 120 tokens per expert
• E1 received 130 > 120, so 130 − 120 = 10 tokens are dropped — for those 10 tokens, the expert's contribution is skipped and the token passes through this layer via the residual connection instead, receiving no expert transformation at all
• E2 (90), E3 (95), E4 (85) are all under the 120 cap, so nothing is dropped for them — but the buffer is still allocated at fixed size 120 and processed as one dense matmul, so E2 wastes 120 − 90 = 30 buffer slots, E3 wastes 25, and E4 wastes 35, all as zero-padded compute that still costs FLOPs and never produces useful output

This is the second hidden cost of MoE that the "37B active parameters" headline hides: raising C reduces dropped tokens (better quality) but increases wasted, zero-padded compute on every under-loaded expert (worse efficiency); lowering C does the reverse. Production teams tune C as a genuine quality-versus-throughput knob, and the diagram above shows exactly this trade-off: Expert 1's bar breaks through the dashed capacity line and sheds its red overflow segment, while Experts 2 through 4 sit below the line with visible unused headroom.

Auxiliary-loss-free load balancing: fixing the router without punishing the model

The obvious fix to token dropping is to make the router balance its own traffic. Classic approaches (Shazeer et al., 2017; Fedus, Zoph & Shazeer, 2021) add an auxiliary loss during training — an extra term added to the language-modeling loss that penalizes uneven dispatch across experts. It works, but it creates a genuine tension: its gradient pulls the router toward uniformity for its own sake, competing with the gradient that's actually trying to make the model predict text well. Weight the auxiliary loss too heavily and you get well-balanced experts that are mediocre at their job; weight it too lightly and balance degrades, load-shedding kicks in, and quality drops from dropped tokens instead.

DeepSeek-V3 replaces this with an auxiliary-loss-free strategy (Wang et al., 2024; adopted in DeepSeek-AI, 2024): instead of an extra loss term, each expert i gets a learned bias b_i that is added to its routing affinity score only for the purpose of deciding the top-k selection — it never touches the weight used to combine the expert's output, so it never distorts the actual gradient signal the router receives from the language-modeling loss. After each training step, the bias is nudged by a small fixed step γ based purely on observed load: overloaded experts get their bias lowered (making them less likely to be picked next step); underloaded experts get it raised. No loss term, no competing gradient — just a simple control-loop correction sitting outside the differentiable path.

Trace this on the same imbalanced loads from before, with target = 100, γ = 0.02, and bias starting at zero for all four experts:

def update_bias(loads, target, bias, gamma=0.02):
    new_bias = []
    for load, b in zip(loads, bias):
        if load > target:
            b = b - gamma
        elif load < target:
            b = b + gamma
        new_bias.append(round(b, 4))
    return new_bias

bias = [0.0, 0.0, 0.0, 0.0]
target = 100
iterations = [
    [130, 90, 95, 85],   # iteration 1: same imbalance as the capacity example
    [122, 93, 97, 88],   # iteration 2: E1 easing down, others rising
    [112, 96, 99, 93],   # iteration 3: continuing to converge
]
for loads in iterations:
    bias = update_bias(loads, target, bias)
    print(bias)

Tracing it by hand: iteration 1 has E1 = 130 > 100 so its bias drops by 0.02 to −0.02, while E2, E3, E4 are all under 100 so each rises by 0.02 to +0.02 — printing [-0.02, 0.02, 0.02, 0.02]. Iteration 2 repeats the same rule on the new loads (E1 = 122 still over, the rest still under), stacking another ±0.02 onto each bias to give [-0.04, 0.04, 0.04, 0.04]. Iteration 3 does it again, producing [-0.06, 0.06, 0.06, 0.06] (the code rounds to 4 decimals as an explicit precision guard against binary floating-point drift accumulating over many iterations, though for these particular values Python's default float printing would already show the same clean digits without it). Notice the pattern this is steering: E1's load is falling step over step (130 → 122 → 112) while the others climb toward 100, exactly the correction a suppressed bias should produce, achieved without a single change to the actual training loss.

Fine-grained experts: more specialization at the same compute and memory budget

A separate, complementary idea from the DeepSeekMoE line of work (Dai et al., 2024) attacks a different inefficiency: early MoE designs like Mixtral used a small number of large experts — 8 experts, top-2 routing, each a full-size FFN (Jiang et al., 2024) — which means a token's final representation is always some combination of exactly one of only C(8,2) = 28 possible expert pairs. DeepSeekMoE's fine-grained segmentation instead splits each expert into m smaller pieces (each 1/m the size) while proportionally scaling up how many are selected, so both the total and active parameter budgets are unchanged. Take m = 4 on our earlier 8-expert example: E' = 32 experts, each S/4 in size, with k' = 8 selected per token.

• Total capacity, coarse design: E × S = 8S. Fine-grained: E' × (S/4) = 32 × S/4 = 8S — identical.
• Active compute, coarse design: k × S = 2S. Fine-grained: k' × (S/4) = 8 × S/4 = 2S — identical.
• Combinatorial routing choices, coarse: C(8,2) = 28. Fine-grained: C(32,8) = 10,518,300 (32!/(8!·24!), computed directly).

Same memory footprint, same per-token FLOPs, but roughly 376,000 times more distinct expert combinations available to the router. DeepSeek-V3's actual production configuration takes this further: 256 fine-grained routed experts with 8 selected per token, plus one always-on shared expert that every token passes through regardless of routing (DeepSeek-AI, 2024). The shared expert captures whatever knowledge is genuinely universal — common syntax, frequent patterns — so the routed experts are freed to specialize on the token-specific residue rather than each having to re-learn generic behavior. Fine granularity buys combinatorial flexibility without touching the memory-versus-compute trade-off that governs everything else in this chapter.

The misconception: "fewer active parameters means a smaller memory footprint"

The hook that opened this chapter is worth stating as a rule, because the mistake it describes is the single most common misreading of MoE scaling claims. Students (and, per the opening scenario, engineers under real budget pressure) see "37B active out of 671B total" and conclude that an inference server only needs enough GPU memory for the active subset — after all, isn't that the whole point of sparsity? It is false, and the reason is structural, not a matter of degree: which 37B parameters are "active" is decided independently, per token, by the router, and in any batch containing more than a handful of tokens, nearly every expert gets touched by some token in that batch. There is no way to know in advance which 37B-parameter subset a given request will need, so the only safe strategy is to keep the entire 671B resident (sharded across the cluster via expert parallelism) and let each token find its own path through it at runtime. Sparsity in MoE buys you a discount on FLOPs per token — the arithmetic actually performed scales with active parameters. It buys you nothing on memory — the storage required scales with total parameters, full stop. Confusing the two is exactly the arithmetic error that turned a routine deployment into an out-of-memory crash in the opening scenario.

Active recall

Attempt each question before reading the worked answer beneath it.

1. In the 8-expert, top-2, d_model = 512, d_ff = 2048 example, why does the total-to-active parameter ratio equal exactly E/k regardless of the values of d_model or d_ff?
2. Using the distributed example (E = 8, G = 4, k = 2, d_model = 512, bf16, T = 4,096 tokens/GPU), what is the all-to-all dispatch volume, in MiB, for just the dispatch phase (not combine)?
3. Now suppose top-k is raised from 2 to 4 in that same distributed setup, with E, G, d_model, precision, and T all unchanged. Recompute: (a) the active-to-total parameter ratio, (b) the total dispatch events and dispatch byte volume, and (c) the average (balanced) tokens routed to each expert.
4. True or false: since only 37B of DeepSeek-V3's 671B parameters activate per token, an inference server only needs enough GPU memory to hold 37B parameters worth of weights. Justify your answer.
5. In the capacity-factor example (target = 100 tokens/expert, observed E1 load = 130), if the capacity factor C is raised from 1.2 to 1.5, does E1 still drop tokens? Compute the new cap and the new dropped count, and state one cost of raising C this way.
6. If the bias step size γ in the auxiliary-loss-free update were doubled from 0.02 to 0.04, what would the bias vector be after iteration 1 (loads [130, 90, 95, 85], target 100, starting bias all zero)? What is the qualitative risk of setting γ too high?

Worked answers

1. Every expert in that design has identical size S, so total parameters are E·S and active parameters are k·S. The ratio (E·S)/(k·S) = E/k always cancels S, so it never depends on how large each individual expert is — only on how many experts exist versus how many are selected per token. Here that gives 8/2 = 4.
2. Global batch N = T × G = 4,096 × 4 = 16,384 tokens. Dispatch events = N × k = 16,384 × 2 = 32,768. Bytes per dispatch = 512 × 2 = 1,024 bytes. Total = 32,768 × 1,024 = 33,554,432 bytes = 32 MiB.
3. (a) Ratio = E/k = 8/4 = 2 (down from 4 — doubling k halves the capacity-to-compute advantage). (b) N is unchanged at 16,384 (T and G didn't change), but dispatch events = N × k = 16,384 × 4 = 65,536 — double the original 32,768 — so dispatch volume = 65,536 × 1,024 bytes = 67,108,864 bytes = 64 MiB, double the original 32 MiB (and combine volume doubles identically, so the forward-pass all-to-all total doubles from 64 MiB to 128 MiB). (c) Average tokens/expert = N × k / E = 16,384 × 4 / 8 = 8,192, double the original 4,096, because twice as many dispatch decisions are now spread across the same 8 experts. Every one of these figures ripples from the single change to k — none stayed fixed.
4. False. Memory footprint scales with total parameters, not active parameters, because different tokens within the same batch activate different experts, and which experts a given request will need cannot be known before routing happens. Any batch of realistic size touches nearly every expert, so the entire 671B parameters (sharded across the serving cluster) must stay resident. Only the FLOPs performed per token scale with the 37B active figure — compute is cheap relative to a dense 671B model, but memory is not.
5. New cap = 1.5 × 100 = 150. E1's load of 130 is now below 150, so zero tokens are dropped (down from 10). The cost: buffer slots reserved for balanced experts grow too — E2 (load 90) now wastes 150 − 90 = 60 slots instead of 30 — so more GPU memory is reserved per expert and more zero-padded matmul compute is wasted on every step. Raising C trades dropped-token quality loss for wasted compute and memory headroom; it is not a free improvement.
6. With γ = 0.04: E1 (130 > 100) drops by 0.04 to −0.04; E2, E3, E4 (all < 100) each rise by 0.04 to +0.04, giving [-0.04, 0.04, 0.04, 0.04] — exactly double the γ = 0.02 result after one step. The risk of setting γ too high is overshoot: a large correction can push a previously-overloaded expert's bias so low that it becomes underloaded the very next step (and vice versa for underloaded experts), causing the routing distribution to oscillate around the target instead of settling into it — the same step-size-versus-stability trade-off that governs any control loop or learning-rate choice.

Think About It

Think about this: How would you explain mixture of experts: how modern llms scale to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where mixture of experts: how modern llms scale is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting mixture of experts: how modern llms scale to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind mixture of experts: how modern llms scale, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.