Paper summary: Merlin:Empowering Multimodal LLMs with Foresight Minds

Problem

People can anticipate what’s likely to happen next from what they see. The authors call this “foresight minds”. Multimodal LLMs mostly describe the present, which limits how well they understand how things work and what the subjects in a scene intend.

Method

The key choice is the subject trajectory, a structured representation of where objects move across consecutive frames, used as a learning target. Two training stages are modelled on how LLMs are trained:

1. Foresight Pre-Training (FPT): jointly train on trajectory-centred tasks, so the model learns to attend to and predict whole trajectories from an initial observation.
2. Foresight Instruction-Tuning (FIT): the model first predicts the trajectories of relevant objects, then reasons about possible future events based on them.

The result is a unified MLLM called Merlin that can output spatial coordinates or tracklets from single images or multiple frames.

Key results

According to the paper, Merlin does well on traditional vision-language tasks and shows strong future-reasoning ability.

Limitations

The extracted limitations section mainly restates the contribution. Its point is that trajectory is a useful learning objective for dynamic information. The approach depends on trajectory annotations.