#long-context
- 17 Jun 2025 · Paper summary
Paper summary: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
MiniMax-M1 is an open-weight hybrid-attention reasoning model (456B total, 45.9B active, 1M-token context) trained with large-scale RL and a new algorithm, CISPO, at a reported rental cost of $534,700.
- 15 Jan 2025 · Paper summary
Paper summary: MiniMax-01: Scaling Foundation Models with Lightning Attention
MiniMax-01 combines lightning attention, a linear-attention variant, with a 32-expert MoE (456B total, 45.9B active) to train on 1M-token contexts and extrapolate to 4M at inference.
- 28 Aug 2024 · Paper summary
Paper summary: Writing in the Margins: Better Inference Pattern for Long Context Retrieval
Writing in the Margins processes a long context chunk by chunk during prefill, writes short "margin" notes per chunk, keeps the relevant ones, and uses them to answer, improving off-the-shelf models without fine-tuning.
- 11 Apr 2024 · Paper summary
Paper summary: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Infini-attention adds a compressive memory to standard attention, combining local masked attention with long-term linear attention so Transformers can stream unbounded inputs with bounded memory.
- 22 Feb 2024 · Paper summary
Paper summary: LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
LongRoPE extends a pretrained LLM's context window to 2048k tokens by searching for non-uniform RoPE interpolation, extending progressively, and readjusting for short contexts.
- 22 Sept 2023 · Paper summary
Paper summary: LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
LongLoRA extends the context window of pretrained LLMs cheaply by fine-tuning with shifted sparse local attention plus a LoRA variant that also trains embeddings and norms.