Paper summary: A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation

Problem

Text-to-image diffusion models often ignore parts of a prompt: wrong counts, wrong positions, missing objects. The authors point at the training data. Datasets like LAION pair web images with their HTML alt-text, which often describes the image poorly or not at all.

Method

RECAP keeps the images and fixes the captions:

1. Fine-tune an automatic captioning model to produce detailed, accurate captions.
2. Relabel the training corpus with these captions.
3. Train the text-to-image model on the recaptioned data.

The paper also analyses different ways to mix captions, and argues the method both reduces the mismatch between training captions and real user prompts and gives the model more information per example.

Key results

As reported in the abstract, compared with training on the original captions:

• FID 14.84 vs 17.87, and a 64.3% improvement in faithful image generation according to human evaluation
• semantic object accuracy 84.34 vs 78.90
• counting alignment errors 1.32 vs 1.44
• positional alignment 62.42 vs 57.60

Limitations

The paper doesn’t have a dedicated limitations section in the extracted text. By construction, the improvement depends on the captioning model used for relabelling, and the reported numbers come from the authors’ own model and dataset setup.