Paper summary: Mixtral of Experts

Problem

Bigger dense models are better but cost more per token. A sparse mixture of experts (SMoE) decouples total parameters from compute per token.

Method

Mixtral keeps the Mistral 7B architecture but replaces each layer’s feed-forward block with 8 experts. For every token at every layer, a router picks 2 experts and combines their outputs. Different tokens, and the same token at different layers, can use different experts. So each token has access to 47B parameters but uses only about 13B active ones. It’s trained with a 32k-token context.

Key results

From the abstract:

• Mixtral outperforms or matches Llama 2 70B and GPT-3.5 on all evaluated benchmarks, and is much stronger than Llama 2 70B on maths, code and multilingual tasks.
• Mixtral 8x7B Instruct surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro and Llama 2 70B chat on human evaluation benchmarks.
• Both models are released under Apache 2.0.

Limitations

There’s no dedicated limitations section. Although only 13B parameters are active per token, all 47B must be held in memory, so memory needs are those of the full model.