Paper summary
Paper summary: MiniMax-01: Scaling Foundation Models with Lightning Attention
MiniMax-01 combines lightning attention, a linear-attention variant, with a 32-expert MoE (456B total, 45.9B active) to train on 1M-token contexts and extrapolate to 4M at inference.
- Linear-style lightning attention makes million-token contexts affordable.
- 456B parameters in total, but only 45.9B active per token.