Paper summary: Vision Transformers Need Registers

Problem

Attention maps and feature maps of large Vision Transformers, both supervised and self-supervised (e.g. DINOv2), contain artifacts: a few tokens with unusually high norm, mostly in low-information background regions. They make attention maps noisy and hurt methods that rely on clean dense features, like object discovery.

Method

The authors characterise these tokens and interpret them as the model recycling patches it doesn’t need (background) as scratch space for global computation. Their fix is to give the model that space explicitly: append a few extra learnable tokens, registers, to the input sequence. They’re used during processing and discarded at the output.

Key results

According to the paper, registers:

• remove the artifacts entirely in both supervised and self-supervised models
• set a new state of the art for self-supervised visual models on dense prediction
• make object-discovery methods work with larger models
• give smoother feature and attention maps

Limitations

There’s no dedicated limitations section. The fix requires (re)training with registers, so existing checkpoints don’t benefit automatically. The interpretation of why the artifacts appear comes from the authors’ analysis, not a formal proof.