Llama 4 Scout and Maverick: Meta moves to mixture-of-experts
Meta's Llama 4 introduces two natively multimodal MoE models with 17B active parameters each: Scout (~109B total, 16 experts) and Maverick (~400B total, 128 experts), trained on up to 40T tokens.
What happened
Meta released Llama 4 Scout and Llama 4 Maverick, with day-one support in Transformers and TGI.
The technical details
From the Hugging Face post:
- a new auto-regressive Mixture-of-Experts architecture
- Maverick: 17B active out of about 400B total, with 128 experts
- Scout: 17B active out of about 109B total, with 16 experts
- early fusion for native multimodality (text and images)
- trained on up to 40 trillion tokens across 200 languages, with specific fine-tuning support for 12 languages including Hindi
- Scout fits on a single server-grade GPU with on-the-fly 4- or 8-bit quantisation, and Maverick ships in BF16 and FP8
- released under the custom Llama 4 Community License
Why it matters
The Llama family switched from dense models to mixture-of-experts, following the path Mixtral opened in December 2023. “Early fusion” is the same design idea as the Chameleon paper (summarised here in May 2024).
#llm#mixture-of-experts#multimodal#meta
Sources
- Welcome Llama 4 Maverick & Scout on Hugging Face — Hugging Face
This is a summary written from the linked public sources. Figures and numbers are as reported there.