Sahaj.blogs
19 Aug 2024 · Paper summary

Paper summary: xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

BLIP-3 (xGen-MM) is Salesforce's open framework for large multimodal models (datasets, recipe, architecture and models), releasing 4B and 14B base and instruction-tuned models that handle interleaved image–text input.

Figure from the paper "xGen-MM (BLIP-3): A Family of Open Large Multimodal Models"
Figure from Xue et al. (2024), licensed CC BY-SA 4.0. Figure 1 : We introduce BLIP-3, a framework (b) for developing Large Multimodal Models (LMMs). Our framework improves upon BLIP-2 (a) [ 42 ] by (1) increasing the richness, scale, and diversity of training data, (2) …

Problem

Open multimodal models often come without the data and recipes needed to rebuild or improve them.

Method

BLIP-3, also called xGen-MM, is presented as an open framework: curated datasets, a training recipe, model architectures and a suite of models. Figure 1 compares it with BLIP-2: richer, larger and more diverse training data, plus other design changes. The authors release 4B and 14B models, both pre-trained base and instruction-tuned.

Key results

From the abstract: competitive performance among open-source LMMs of similar size, evaluated on single- and multi-image benchmarks, with the ability to understand interleaved image–text input. Training code, models and all datasets, including three new large-scale ones, are to be open-sourced.

Limitations

There’s no dedicated limitations section in the extracted text. The claim is competitiveness at a given size, not state of the art overall.

#multimodal#open-source

Sources

  1. xGen-MM (BLIP-3): A Family of Open Large Multimodal Models (arXiv:2408.08872) — arXiv
  2. Hugging Face Daily Papers, 2024-08-19 — Hugging Face
  3. xGen-MM (BLIP-3): A Family of Open Large Multimodal Models (arXiv HTML full text) — arXiv

This is a summary written from the linked public sources. Figures and numbers are as reported there.