Paper summary
Paper summary: Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Mamba makes state-space model parameters depend on the input, so the model can choose what to remember or forget, and pairs this with a hardware-aware scan: linear-time sequence modelling without attention.
- Letting the SSM parameters depend on the current token gives the model the content-based selection that earlier SSMs lacked.
- The paper reports about 5× higher inference throughput than Transformers and linear scaling in sequence length.