#attention

• 25 Jun 2025 · Study

  NLP · Attention (Bahdanau)

  Additive attention lets the decoder look at every encoder state instead of one summary vector; the same reversal task with attention, and its alignment weights.

• 8 Oct 2024 · Paper summary

  Paper summary: Differential Transformer

  Differential attention computes attention as the difference between two softmax maps, cancelling common-mode noise; Diff Transformer is reported to beat Transformers on scaling, long context, retrieval and hallucination.

• 11 Apr 2024 · Paper summary

  Paper summary: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

  Infini-attention adds a compressive memory to standard attention, combining local masked attention with long-term linear attention so Transformers can stream unbounded inputs with bounded memory.