Paper summary: Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

Problem

The best VLMs are proprietary. Many open-weight VLMs get their quality from synthetic data produced by proprietary VLMs, which effectively distils closed models. So the community lacked knowledge of how to build strong VLMs from scratch.

Method

The Allen Institute team built PixMo, a set of datasets collected without using external VLMs:

• highly detailed image captions for pre-training
• a free-form image Q&A dataset for fine-tuning
• a 2D pointing dataset, so the model can point at locations in an image

Molmo models are trained on these with careful modelling choices and a well-tuned pipeline. The authors stress that data quality was the most critical factor.

Key results

From the abstract: the best 72B Molmo outperforms other models that are open in both weights and data. The authors describe the family as state of the art for its class of openness.

Limitations

There’s no dedicated limitations section in the extracted text. “State of the art in its class of openness” compares mainly with other open models, not every proprietary one.