Paper summary: Differential Transformer

Problem

According to the authors, Transformers tend to over-allocate attention to irrelevant context. Some attention “noise” spreads over tokens that don’t matter, which can drown the relevant signal.

Method

Differential attention computes two separate softmax attention maps and uses their difference as the attention scores. The paper compares it to noise-cancelling headphones: attention that both maps give to the same irrelevant tokens cancels out, and sparse attention patterns emerge. Diff Transformer replaces standard attention with this mechanism.

Key results

From the abstract:

• outperforms the Transformer across settings that scale model size and training tokens
• advantages in long-context modelling, key information retrieval, hallucination mitigation (question answering and summarisation) and in-context learning
• fewer activation outliers

Limitations

There’s no dedicated limitations section in the extracted text. Each layer computes two attention maps, so the mechanism changes the attention computation and its cost compared with standard attention.