Paper summary: Movie Gen: A Cast of Media Foundation Models

Problem

Producing high-quality, long, high-resolution video with matching sound, and editing or personalising it, needs several capabilities that are usually built separately.

Method

Movie Gen is a “cast” of foundation models that:

• generates 1080p HD video in different aspect ratios with synchronised audio
• performs instruction-based video editing
• generates personalised videos from a user’s image

The largest video model is a 30B-parameter Transformer trained with a context of up to 73K video tokens, enough for 16 seconds at 16 frames per second. The report describes innovations and simplifications across architecture, latent spaces, training objectives, data curation, evaluation protocols, parallelisation and inference.

Key results

From the abstract: new state of the art on text-to-video, video personalisation, video editing, video-to-audio and text-to-audio. The paper also describes the evaluation benchmark it uses.

Limitations

There’s no dedicated limitations section in the extracted text. Clip length is bounded by the context window (16 seconds at the reported settings).