Paper summary: Chameleon: Mixed-Modal Early-Fusion Foundation Models

Problem

Many multimodal models bolt a vision encoder onto a language model and can only read images. Generating arbitrary sequences that mix images and text, in any order, needs a different design.

Method

Chameleon turns images into discrete tokens and processes them in the same sequence as text tokens from the start (early fusion). This makes it one model for both understanding and generating either modality. The paper’s emphasis is on making this setup trainable: a stable training approach from the start, an alignment recipe, and an architectural parameterisation tailored to mixed-modal tokens.

Key results

From the abstract:

• state-of-the-art performance on image captioning
• outperforms Llama-2 on text-only tasks and is competitive with Mixtral 8x7B and Gemini-Pro
• non-trivial image generation, all in one model
• on a new long-form mixed-modal evaluation, it matches or exceeds Gemini Pro and GPT-4V according to human judgments

Limitations

There’s no dedicated limitations section in the extracted text. The headline mixed-modal result uses a new evaluation introduced by the authors, judged by humans.