Paper summary: SAM 2: Segment Anything in Images and Videos

Problem

The original Segment Anything Model (SAM) segments any object in a single image from a click or box. Video adds time: objects move, get occluded and reappear, and you want to prompt once and have the mask follow the object.

Method

• Model: a simple Transformer with a streaming memory. Frames are processed one at a time, and the memory carries information about the target object forward, which enables real-time video processing.
• Data engine: model and annotators improve each other through interaction. The engine collected the largest video segmentation dataset to date.

Key results

From the abstract: in video segmentation, better accuracy with 3× fewer interactions than prior approaches. In image segmentation, more accurate and 6× faster than SAM. The model, dataset, training code and demo are released.

Limitations

From the paper:

• may fail across shot changes
• can lose or confuse objects in crowded scenes, after long occlusions or in long videos (extra clicks on other frames usually recover it)
• struggles with very thin, fast-moving details and with nearby objects that look alike, such as identical juggling balls; the authors suggest more explicit motion modelling