Paper summary
Paper summary: Mixtral of Experts
Mixtral 8x7B is a sparse mixture-of-experts model: each layer has 8 feed-forward experts and a router picks 2 per token, giving 47B total but 13B active parameters.
- Each token goes through only 2 of the 8 experts per layer, so it uses 13B of the 47B parameters.
- The paper reports Mixtral matching or beating Llama 2 70B and GPT-3.5 on the benchmarks it evaluates.