What is a mixture-of-experts (MoE) model in simple terms?

It's a model split into many specialized sub-networks called experts, with a router that sends each token to only a few of them. So the model can be huge in total size — holding lots of knowledge — while only doing a small amount of computation per token. It separates how much the model knows from how much it costs to run.

Why does an MoE model need so much VRAM if it only uses a few experts?

Because the router can send any token to any expert, every expert must be resident in memory even though only a few fire per token. So an MoE model needs VRAM proportional to its total parameter count, not its active count — Mixtral 8x7B needs roughly the memory of a dense 47B model despite 13B-equivalent compute.

Is an MoE model faster than a dense model of the same total size?

Yes, per token. Because only a fraction of parameters activate, an MoE does far fewer FLOPs per token than a dense model with the same total parameter count, so it's cheaper and faster to run — as long as you can fit all the experts in memory. It trades memory cost for compute efficiency.

How does the router decide which experts to use?

The router (gating network) computes a score for each expert given the token, usually via a small learned projection followed by a softmax, then selects the top-k highest-scoring experts — often the top 2. Only those experts process the token. During training, load-balancing techniques keep tokens from collapsing onto a few favorite experts.

Does "8x7B" mean 56 billion parameters?

No. Mixtral 8x7B is about 46.7B total, not 56B, because the experts share the attention and embedding layers — only the feed-forward layers are split into eight experts. And only ~12.9B are active per token. Total never equals experts times expert-size; it's a naming convention, not arithmetic.

Are MoE models better or worse than dense models for reasoning?

It's mixed. At matched pretraining quality, MoE models tend to do well on knowledge-heavy tasks and respond strongly to instruction tuning, but can lag dense models on some reasoning-heavy benchmarks. The reasoning-focused MoE models like DeepSeek-R1 close that gap with reinforcement learning rather than architecture alone.

← Resources

By Ren Sugaya — Performance Engineer, GPU & Inference · 2026-05-12 · INSIGHT

Mixture of Experts Explained: Total vs Active Parameters

A mixture-of-experts model has a huge total parameter count but activates only a fraction per token, using a router to pick a few expert sub-networks. This explains how MoE works, the all-important total-vs-active distinction, why it's compute-cheap but memory-hungry, and what that means for running models like DeepSeek and Mixtral.

What a mixture of experts is

A mixture-of-experts (MoE) LLM is a sparse transformer in which some feed-forward layers contain multiple parallel sub-networks called "experts," and a learned router selects only a small subset of them to process each token. The result is a model with a large total parameter count that activates only a fraction of those parameters per token.

Contrast that with a dense model, where every token passes through the same feed-forward network in every block, activating all parameters every time. MoE swaps those dense layers for sparse expert layers, so capacity (how much the model knows) is decoupled from cost (how much compute each token takes).

Experts and the router

Two components define an MoE layer. The experts are each a small neural network, typically a feed-forward block. The router (or gating network) scores the incoming token against each expert and routes it to the top-k highest scorers — commonly the top 2 of 8. Only those experts contribute to that token's output; the rest sit idle for that token.

This top-k sparsity is the whole trick: it's what cuts the compute per token. Some designs add a "shared expert" that is always active alongside the routed ones, to capture common patterns every token needs.

Total vs active parameters — the key idea

The single most important concept — and the most mangled one online — is the difference between total and active parameters. Total parameters are all the experts that must be loaded; active parameters are only those used per token. Headline size is not compute, so the follow-up to any MoE parameter count is always: how many are active per token?

The "8x7B" naming is the classic trap. Mixtral 8x7B is not 8 × 7 = 56B parameters. Because the experts share the attention and embedding layers — only the feed-forward layers are split into experts — the real total is about 46.7B, with roughly 12.9B active per token. Its compute resembles a ~13B dense model even though it holds ~47B parameters.

Real models: Mixtral and DeepSeek

Two concrete examples anchor the idea. Mixtral 8x7B has ~46.7B total parameters and ~12.9B active per token via top-2-of-8 routing. DeepSeek-V3 scales the pattern far higher: 671B total parameters with just 37B active per token, using 256 routed experts plus one always-on shared expert, with the router selecting 8 routed experts per token. DeepSeek-R1, the reasoning model, is built on the V3 base — same architecture, same 671B total / 37B active, with reinforcement learning added for math and code.

By 2025, MoE had become the dominant frontier architecture, with Llama 4, Qwen3, and others all adopting it — precisely because it lets labs scale total capacity without scaling per-token compute.

The VRAM caveat: fast compute, big memory

Here's the catch that surprises people: sparse compute does not reduce memory. Every expert must be resident in VRAM (or RAM) even though only a few fire per token, because the router might send the next token to any of them. So Mixtral 8x7B needs roughly the memory of a dense ~47B model — on the order of 96 GB unquantized — despite having the compute footprint of a 13B model. You cannot fit it on a single consumer GPU just because "only 13B is active."

The trade-off summary: MoE gives you better tokens-per-dollar (fewer FLOPs per token) but worse memory-per-dollar (all experts resident). Quantization helps the memory side, which is why MoE models are usually run quantized locally or on multi-GPU endpoints.

Running MoE models in osFoundry

osFoundry runs MoE models like DeepSeek-V3/R1 and Mixtral either locally or on a dedicated cloud endpoint, so you get the per-token efficiency of sparse activation without managing the architecture yourself. Because the real bottleneck is memory — every expert must sit in VRAM — the platform makes that a deployment choice: run local when your hardware can hold all experts (quantized), or use a dedicated endpoint when a 671B-class model won't fit on your machine. And because all model calls are BYOK, you can switch freely between an MoE model for cheap high-capacity inference and a dense model for a smaller memory footprint, per task. The total-vs-active trade-off becomes a setting, not a hardware headache.

Frequently asked questions

What is a mixture-of-experts (MoE) model in simple terms?: It's a model split into many specialized sub-networks called experts, with a router that sends each token to only a few of them. So the model can be huge in total size — holding lots of knowledge — while only doing a small amount of computation per token. It separates how much the model knows from how much it costs to run.
What's the difference between total and active parameters?: Total parameters are all the experts loaded into memory; active parameters are the few actually used to process a given token. DeepSeek-V3, for example, is 671B total but only 37B active per token. Compute tracks the active count, while memory tracks the total — which is why the two numbers always need to be quoted together.
Why does an MoE model need so much VRAM if it only uses a few experts?: Because the router can send any token to any expert, every expert must be resident in memory even though only a few fire per token. So an MoE model needs VRAM proportional to its total parameter count, not its active count — Mixtral 8x7B needs roughly the memory of a dense 47B model despite 13B-equivalent compute.
Is an MoE model faster than a dense model of the same total size?: Yes, per token. Because only a fraction of parameters activate, an MoE does far fewer FLOPs per token than a dense model with the same total parameter count, so it's cheaper and faster to run — as long as you can fit all the experts in memory. It trades memory cost for compute efficiency.
How does the router decide which experts to use?: The router (gating network) computes a score for each expert given the token, usually via a small learned projection followed by a softmax, then selects the top-k highest-scoring experts — often the top 2. Only those experts process the token. During training, load-balancing techniques keep tokens from collapsing onto a few favorite experts.
Does "8x7B" mean 56 billion parameters?: No. Mixtral 8x7B is about 46.7B total, not 56B, because the experts share the attention and embedding layers — only the feed-forward layers are split into eight experts. And only ~12.9B are active per token. Total never equals experts times expert-size; it's a naming convention, not arithmetic.
Are MoE models better or worse than dense models for reasoning?: It's mixed. At matched pretraining quality, MoE models tend to do well on knowledge-heavy tasks and respond strongly to instruction tuning, but can lag dense models on some reasoning-heavy benchmarks. The reasoning-focused MoE models like DeepSeek-R1 close that gap with reinforcement learning rather than architecture alone.