Paper summary: Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Problem

Transformers’ attention costs grow quadratically with sequence length. Many subquadratic alternatives exist (linear attention, gated convolutions, recurrent models, structured state-space models or SSMs), but on language they had lagged behind attention. The authors locate the weakness in an inability to do content-based reasoning: to decide, from the current token, what to keep and what to ignore.

Method

1. Selection: make the SSM’s parameters functions of the input, so the model can selectively propagate or forget information depending on the current token.
2. This breaks the convolution trick earlier SSMs used for fast training, so the authors design a hardware-aware parallel algorithm that computes the recurrence efficiently on GPUs.
3. They build a simplified architecture, Mamba, with no attention and no separate MLP blocks.

Key results

From the abstract: Mamba has fast inference, about 5× the throughput of Transformers, and linear scaling in sequence length. The paper reports strong results on language, audio and genomics. On language modelling, Mamba-3B outperforms Transformers of the same size and matches Transformers twice its size, both in pre-training and downstream evaluation. Code and checkpoints are open-sourced.

Limitations

The paper’s discussion includes a “no free lunch” point. Structured SSMs came from continuous systems and carry an inductive bias toward continuous signals like audio or video. The selection mechanism that helps on discrete data like text can trade off against that, and the authors discuss this spectrum rather than claiming a universal win.