In plain words
A bigger AI model generally knows more, but every extra part makes each answer slower and costlier, because normally the whole model works on every single word.
A mixture-of-experts model is split into many smaller sub-networks, called experts. A small gatekeeper, the router, looks at each word as it arrives and sends it to just a couple of the experts. The rest stay idle for that word.
So the model can hold a vast amount of learned knowledge while spending only a fraction of its effort on each word. Mistral's Mixtral model, for example, has 47 billion parameters but uses only about 13 billion for any one word.
Many of today's large models are built this way, including DeepSeek's and Meta's Llama 4.
Everyday example
The front desk at a big hospital
A large hospital has dozens of specialists, but a patient does not see all of them. The front desk looks at each case and sends it to the two most suitable doctors, who each give an opinion. The hospital has huge combined expertise, yet each visit takes only two doctors' time. The front desk is the router; the doctors are the experts.
Where the example breaks: Hospital specialists have clear subjects like heart or skin. AI experts do not: when Mistral studied its model, routing followed things like word type and position, not topics such as biology or philosophy. And the router decides for every word, not once per question.
How it actually works
In a transformer, mixture of experts usually replaces one part of each layer: the feed-forward network that processes each token after attention.
Many experts per layer
Instead of one feed-forward network, each layer has several, often 8 to over 100. Each is an ordinary small neural network. Attention stays shared.
A router picks the top few
A small learned router scores every expert for the current token and keeps the top one or two (Mixtral uses two of eight). Their outputs are blended using the router's scores as weights.
Total versus active parameters
Total parameters measure what the model stores; active parameters measure the work per token. DeepSeek-V3 has 671 billion in total but 37 billion active per token, so it runs much faster than a dense model of the same size.
The costs
All the experts must still sit in memory, so MoE models need lots of it even if they compute little. Training has to keep work balanced so that a few experts do not get every token while others sit unused.
- Original idea
- 1991, "Adaptive Mixtures of Local Experts" by Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton
- Made huge
- January 2017, Noam Shazeer and colleagues at Google: an MoE layer with up to 137 billion parameters
- Mixtral 8x7B
- Mistral AI, December 2023: 8 experts per layer, 2 used per token; 47B total, 13B active
- Recent examples
- DeepSeek-V3 (671B total, 37B active), Llama 4 Maverick (400B total, 17B active, 128 experts), Google's Gemini 1.5
Questions people ask
What does MoE mean in AI?
MoE stands for mixture of experts: a model built from many sub-networks where a router sends each token to only a few of them. It lets a model be very large without each answer costing as much as a full-size model would.
What is the difference between total and active parameters?
Total parameters are everything the model has learned and stored. Active parameters are the ones actually used for each token. In an MoE model, active is a small fraction of total, which is why it can be fast despite its size.
Are the experts specialists in subjects like maths or history?
Not in a way people would recognise. Routing is learned automatically and tends to follow word-level patterns. Mistral found no obvious topic specialisation in Mixtral, though it saw some patterns, such as indentation in code going to the same experts.
Is GPT a mixture of experts?
OpenAI has not published the design of its flagship GPT models, so we cannot say for sure. Its open-weight gpt-oss models (August 2025) are MoE, as are Mixtral, DeepSeek-V3, Llama 4 and Google's Gemini 1.5.