Paper summary: ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

Problem

Both video-language models and text-to-video models depend on captions, and most video captions are short and vague. Human annotation doesn’t scale, and the authors found that asking GPT-4V naively (several frames at once, or frames stitched together) gave less detailed and sometimes temporally confused captions.

Method

The series has three parts:

1. ShareGPT4Video: 40K dense captions of videos of various lengths and sources, made with GPT-4V through a carefully designed filtering and annotation strategy. The strategy targets precise changes between frames, detailed content within frames, and scaling to long videos.
2. ShareCaptioner-Video: a captioning model trained on that data, used to caption 4.8M high-quality, aesthetic videos.
3. ShareGPT4Video-8B: a video LVLM trained with the captions.

Key results

The abstract reports that ShareGPT4Video-8B reaches state-of-the-art performance on three video benchmarks.

Limitations

From the paper:

• the caption pipeline can’t use audio, because GPT-4V doesn’t take audio input
• the large set of generated captions wasn’t manually checked for social bias
• the source videos may contain human faces, so users must follow the original videos’ licences