Paper summary: Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion

Problem

Diffusion models generate images either directly in pixel space or in a compressed latent space. The question here is how to combine the strengths of an “image prior” design (as in DALL·E 2) with efficient latent diffusion, and release it openly.

Method

The Kandinsky model described in the paper has three parts:

1. An image prior, trained separately, that maps a CLIP text embedding to a CLIP image embedding.
2. A latent diffusion model that generates image latents conditioned on that embedding.
3. A modified MoVQ autoencoder that decodes latents to pixels.

The whole system has 3.3B parameters. A demo supports text-to-image, image fusion, variations, and text-guided inpainting and outpainting. Code and checkpoints are released.

Key results

The paper reports an FID of 8.03 on COCO-30K, which the authors describe as the top measured result among open-source models at that time.

Limitations

From the authors’ own section: the images look natural, but semantic coherence between text and image can be improved, as can absolute FID and human-judged image quality.