Paper summary: MiniMax-01: Scaling Foundation Models with Lightning Attention
MiniMax-01 combines lightning attention, a linear-attention variant, with a 32-expert MoE (456B total, 45.9B active) to train on 1M-token contexts and extrapolate to 4M at inference.
Problem
Standard softmax attention makes very long contexts expensive. The authors want top-tier quality and million-token contexts at affordable cost.
Method
- Lightning attention, an efficient linear-attention approach, is the core, scaled efficiently.
- It’s combined with Mixture of Experts: 32 experts, 456B total parameters, 45.9B active per token.
- They develop an optimised parallel strategy and computation–communication overlap for MoE and lightning attention, to train and serve at this scale.
The series includes MiniMax-Text-01 and MiniMax-VL-01.
Key results
From the abstract: performance comparable to top-tier models, with a context window of up to 1 million tokens in training that extrapolates to 4 million at inference.
Limitations
There’s no dedicated limitations section in the extracted text. Very long context support doesn’t guarantee equally good use of all that context.
#llm#long-context#mixture-of-experts
Sources
- MiniMax-01: Scaling Foundation Models with Lightning Attention (arXiv:2501.08313) — arXiv
- Hugging Face Daily Papers, 2025-01-15 — Hugging Face
- MiniMax-01: Scaling Foundation Models with Lightning Attention (arXiv HTML full text) — arXiv
This is a summary written from the linked public sources. Figures and numbers are as reported there.