#mixture-of-experts

29 Oct 2026 · AI updates
Mistral Large 4 ("Le Chonk"): ~1T-parameter MoE with sparse attention
Mistral Large 4 is a multimodal MoE of about 1T total parameters with a 1M-token context, combining expert sparsity with sparse attention and a Pixtral vision encoder, released in FP8 and NVFP4 under an MIT-derived open-weight licence.
15 Jul 2026 · AI updates
Inkling: a ~1T-parameter open multimodal model with 1M context
Thinking Machines released Inkling on Hugging Face, a ~1T-parameter decoder-only multimodal MoE (41B active) that takes image, text and audio, has a 1M context, and was trained on 45T tokens; a smaller Inkling-Small followed.
5 Aug 2025 · AI updates
gpt-oss: OpenAI releases open-weight models under Apache 2.0
OpenAI released gpt-oss-120b and gpt-oss-20b, mixture-of-experts reasoning models quantised with MXFP4 and licensed Apache 2.0; the large one fits on one H100 and the small one in 16GB.
5 Apr 2025 · AI updates
Llama 4 Scout and Maverick: Meta moves to mixture-of-experts
Meta's Llama 4 introduces two natively multimodal MoE models with 17B active parameters each: Scout (~109B total, 16 experts) and Maverick (~400B total, 128 experts), trained on up to 40T tokens.
15 Jan 2025 · Paper summary
Paper summary: MiniMax-01: Scaling Foundation Models with Lightning Attention
MiniMax-01 combines lightning attention, a linear-attention variant, with a 32-expert MoE (456B total, 45.9B active) to train on 1M-token contexts and extrapolate to 4M at inference.
1 Apr 2024 · Paper summary
Paper summary: Jamba: A Hybrid Transformer-Mamba Language Model
Jamba interleaves Transformer and Mamba layers and adds mixture-of-experts in some layers; the released configuration fits on one 80GB GPU and handles contexts up to 256K tokens.
9 Jan 2024 · Paper summary
Paper summary: Mixtral of Experts
Mixtral 8x7B is a sparse mixture-of-experts model: each layer has 8 feed-forward experts and a router picks 2 per token, giving 47B total but 13B active parameters.
11 Dec 2023 · AI updates
Mixtral 8x7B arrives on Hugging Face
Mistral released Mixtral 8x7B, a sparse mixture-of-experts LLM under Apache 2.0; Hugging Face integrated it into Transformers, TGI and Inference Endpoints on launch day.