Paper summary: Lumiere: A Space-Time Diffusion Model for Video Generation

Problem

Most video diffusion systems first generate distant keyframes and then fill in the frames between them with a cascade of temporal super-resolution models. The authors argue this makes globally coherent motion inherently hard, because no single model ever sees the whole clip.

Method

Lumiere’s Space-Time U-Net generates the entire temporal duration of a low-resolution, full-frame-rate video in one pass. It downsamples and upsamples in time as well as space, so the network processes the clip at several space-time scales. It builds on a pretrained text-to-image diffusion model, adding the temporal parts.

Key results

The paper reports state-of-the-art text-to-video results. The same design also supports image-to-video, video inpainting and stylised generation.

Limitations

There’s no dedicated limitations section in the extracted text. The base model directly generates a low-resolution video, and the paper uses a spatial super-resolution stage on top of it to reach the final resolution.