Paper summary
Paper summary: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Infini-attention adds a compressive memory to standard attention, combining local masked attention with long-term linear attention so Transformers can stream unbounded inputs with bounded memory.
- Each block mixes normal local attention with retrieval from a fixed-size compressive memory of everything earlier.
- The paper reports 1M-token passkey retrieval and 500K-token book summarisation with 1B and 8B models.