Paper summary: PaliGemma 2: A Family of Versatile VLMs for Transfer

Problem

PaliGemma was designed as a VLM meant to be fine-tuned for specific tasks. The questions here: how do model size and image resolution affect transfer, and how far can transfer reach?

Method

PaliGemma 2 combines the SigLIP-So400m vision encoder (as in PaliGemma) with the full Gemma 2 range, from 2B to 27B. Models are trained in multiple stages at three resolutions: 224, 448 and 896 px. This grid of base models lets the authors study transfer factors such as learning rate, and how task type, model size and resolution interact.

Key results

The paper widens the set of transfer tasks beyond PaliGemma, including table structure recognition, molecular structure recognition, music score recognition and long, fine-grained captioning. The abstract describes these as state-of-the-art results on a wide range of tasks.

Limitations

There’s no dedicated limitations section in the extracted text. The models are base models intended to be fine-tuned, not general chat assistants out of the box.