11 min read

Dense models vs mixture of experts

Senior LLMs Scaling3 November 2026

DeepSeek-V3, released at the end of 2024, has 671 billion parameters. For each token it generates, it uses about 37 billion of them. That is the idea behind a mixture of experts (MoE) model: a very large model in which each token only passes through a small, chosen part. Mixtral 8x7B made the design popular in open models at the end of 2023, and many of the largest open models since, including DeepSeek-V3 and Meta's Llama 4, are built this way.

A dense model uses every parameter for every token. An MoE model has many more parameters in total, but only a few are active per token. That one difference splits the usual cost of a model in two: computation follows the active parameters, while memory follows the total. Most of what is surprising about MoE models, good and bad, follows from that split.

Your answer

A 400B parameter MoE model has 17B active parameters per token. A dense 70B model is the alternative. Which needs more GPU memory to serve, which does less computation per token, and which would you run on a single 80 GB GPU?

Experts and a router#

A transformer layer has two main parts: attention, which mixes information between tokens, and a feed-forward network (FFN), which processes each token on its own. The FFN holds most of the parameters. An MoE layer replaces that one FFN with many FFNs, the experts, and adds a small router that decides which experts each token goes to.

1/4 The router scores all experts for this token, say 8 or 256 of them.

The router is learned along with everything else. Experts do not end up as neat specialists in "law" or "cricket". Studies of trained MoE models find them specialising in finer, stranger things, such as punctuation, certain token types or parts of a syntax, and the same topic uses different experts in different layers. The names "expert" and "mixture" oversell it. It is better thought of as a very large model in which each token takes a sparse path.

Typical shapes:

Mixtral 8x7B: 8 experts per layer, 2 chosen per token. About 47B parameters in total, about 13B active. (Not 56B, because attention and embeddings are shared, only the FFNs are multiplied.)
DeepSeek-V3: 256 small experts per layer plus one shared expert every token uses, 8 routed experts chosen per token. 671B total, about 37B active.
Llama 4 Maverick: 128 experts, about 400B total and 17B active.