Paper summary
Paper summary: ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
ShareGPT4Video builds dense, precise video captions (40K with GPT-4V, then 4.8M with its own captioner) to improve both video understanding models and text-to-video generation.
- Naively feeding frames to GPT-4V gave captions that missed detail and confused the order of events, so the authors designed a dedicated captioning strategy.
- One caption pipeline serves two goals: better video-language models and better text-to-video models.