Paper summary: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Problem

Standard attention’s memory grows with the context: the KV cache keeps every past token. To handle unboundedly long inputs, a model needs memory that stays the same size.

Method

Infini-attention builds a compressive memory into the attention mechanism itself. Within a single Transformer block it combines:

• masked local attention over the current segment, as usual
• long-term linear attention that reads from and updates a fixed-size memory summarising all earlier segments

The input is processed segment by segment as a stream, so memory and compute stay bounded.

Key results

The abstract reports effectiveness on long-context language-modelling benchmarks, on 1M-length passkey retrieval and on 500K-length book summarisation, using 1B and 8B LLMs. It adds only minimal memory parameters and allows fast streaming inference.

Limitations

The extracted text has no dedicated limitations section. A fixed-size compressive memory necessarily loses some detail of the distant past.