Mixture Of Experts

Mixture of Experts (MoE) is a neural network architecture where each input token is routed to only a few specialized sub-networks, reducing inference cost while preserving the knowledge capacity of a much larger model.

What It Is

When you’re evaluating pretrained models for a decoder-only transformer project, one architectural detail changes the math more than anything else: whether the model uses Mixture of Experts. MoE is the reason some of the largest models today can run faster and cheaper than their parameter counts suggest. It decouples total model size from the computation required per token, which directly affects your hardware requirements, latency, and serving costs.

Think of it like a hospital with dozens of specialist doctors. When a patient walks in, a triage nurse evaluates the symptoms and sends them to two or three relevant specialists — not the entire medical staff. The hospital holds enormous collective expertise, but each patient only consumes a small fraction of it. In MoE, the triage nurse is called a gating network (or router), and the specialists are called experts.

In a standard dense transformer, every single token passes through every parameter in every layer. An MoE transformer replaces some of these dense layers with MoE layers, each containing multiple smaller feed-forward networks — the experts. For each token, the gating network produces a score for every expert, then selects the top one or two. Only those chosen experts process the token. The others stay idle for that particular input.

This design means a model can store hundreds of billions of parameters worth of learned knowledge while only activating a fraction of them per token. According to Meta AI Blog, LLaMA 4 Scout uses 16 experts and activates 17 billion parameters out of 109 billion total. According to DeepSeek API Docs, DeepSeek V3.2 takes this further with 256 experts, activating 37 billion parameters from a pool of 685 billion. The active parameter count, not the total, determines your actual inference speed and memory needs.

Training an MoE model introduces a challenge called load balancing. If the gating network sends most tokens to the same few experts, those experts become bottlenecks while others go underused. Training procedures include auxiliary losses (extra penalty terms that discourage the router from overloading a few experts) or specialized routing strategies to spread the workload evenly. Modern MoE models handle this well in practice, but the gating network itself requires careful design and tuning during pretraining.

How It’s Used in Practice

Most practitioners encounter MoE when comparing pretrained models for deployment. You’ll notice that some models report two parameter counts: total and active. That gap between total and active parameters is MoE at work. When benchmarking, an MoE model often matches or beats a dense model with similar total parameters while running significantly faster at inference.

Cloud API providers frequently serve MoE models behind their endpoints. The lower per-token compute cost translates directly to lower serving costs and faster response times, which is why many of the most capable commercial APIs run on MoE architectures. If you’ve called a frontier model through an API recently, there’s a good chance it was an MoE model.

For teams building on decoder-only transformers, MoE also shapes your fine-tuning strategy. According to Fireworks Blog, fine-tuning MoE models with LoRA requires separate adapter matrices for each expert in each MoE layer. With models that have over a hundred experts, this complexity adds up. Dense models remain simpler to fine-tune, which is a real trade-off when choosing your base model.

Pro Tip: When comparing pretrained models, always check the active parameter count, not just the total. The active count determines your real memory footprint and latency at inference. A model reporting hundreds of billions of total parameters might only need the GPU resources of a much smaller dense model.

When to Use / When Not

Common Misconception

Myth: A model with hundreds of billions of parameters uses all of them for every response, making it inherently slow and expensive. Reality: MoE models activate only a small subset of their total parameters for each token. The gating network routes inputs to one or two experts per layer, so the actual computation scales with the active parameter count. A model with hundreds of billions of total parameters can run at the speed and cost of a much smaller dense model.

One Sentence to Remember

Mixture of Experts gives you big-model knowledge at small-model cost by routing each token through just a few specialist sub-networks — and understanding this trade-off is the first step to picking the right pretrained model for your decoder-only transformer project.

FAQ

Q: What is the difference between a dense transformer and a Mixture of Experts transformer? A: A dense transformer runs every token through all parameters. An MoE transformer routes each token to a small subset of expert sub-networks, reducing per-token computation while keeping total model capacity high.

Q: Does Mixture of Experts make fine-tuning harder? A: Yes. LoRA-based fine-tuning on MoE models requires separate adapter matrices for each expert in each MoE layer, which increases memory and complexity compared to fine-tuning a dense model of similar active size.

Q: How does the gating network decide which experts to use? A: The gating network takes each token’s hidden state as input and scores every available expert. The top-scoring experts — usually one or two — process that token while the rest stay idle.

Sources

• Meta AI Blog: The Llama 4 herd: natively multimodal AI - LLaMA 4 Scout and Maverick MoE architecture details and expert configurations
• DeepSeek API Docs: DeepSeek-V3.2 Release - DeepSeek V3.2 MoE specifications including expert count and active parameters