Paper summary: Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

Problem

Open multimodal LLMs are usually trained with pre-training plus supervised fine-tuning. The authors argue this leaves a distribution shift between training and inference that particularly limits chain-of-thought reasoning.

Method

Two parts:

• Data: an automated pipeline builds MMPR, a large, high-quality multimodal reasoning preference dataset (preferred vs rejected reasoning).
• Model: Mixed Preference Optimization (MPO), a simple method that integrates preference optimisation into MLLM training.

Key results

The approach improves the multimodal reasoning of both InternVL2-8B and InternVL2-76B. From the abstract: InternVL2-8B-MPO scores 67.0 on MathVista, 8.7 points above InternVL2-8B and comparable to the 10× larger InternVL2-76B. Code, data and model are released.

Limitations

There’s no dedicated limitations section in the extracted text. The preference data is built automatically, so its quality depends on that pipeline.