Paper summary: Emu3: Next-Token Prediction is All You Need

Problem

Next-token prediction drives LLMs, but multimodal tasks were dominated by other designs: diffusion for generation (Stable Diffusion) and compositional systems for perception (CLIP plus an LLM).

Method

Emu3 tokenises images, text and videos into one discrete token space, and trains a single Transformer from scratch on mixed multimodal sequences with only next-token prediction. No diffusion and no separate vision encoder. A video is generated token by token, like text.

Key results

From the abstract: Emu3 outperforms several well-established task-specific models in both generation and perception, surpassing SDXL and LLaVA-1.6. It also generates high-fidelity video. The authors argue that focusing purely on tokens simplifies multimodal model design.

Limitations

There’s no dedicated limitations section in the extracted text. Generating images and video token by token means long sequences, so inference cost is a practical concern for this approach.