Model architecture
Mixture of experts
Also called MoE.
What is mixture of experts?
A mixture-of-experts model contains many specialised sub-networks and routes each token through only a few of them. Total parameters are large while active parameters per token are small, so it costs like a large model in memory and a small one in compute.
The trade is explicit and worth stating plainly: every parameter must be resident in memory, but only a fraction participates in any given token. That is excellent when memory is cheap relative to compute and poor when the reverse holds.
Fine-tuning them is harder. Routing can become unbalanced during training, concentrating work on a subset of experts, and the memory requirement during training is set by total rather than active parameters.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.