Paper summary: VGI-Bench: Probing Visual Intelligence in Video Generation Models

Problem

Some work suggests video generators can reason visually through the frames they generate (“Thinking with Video” is summarised here in Nov 2025). Evaluating this reliably is hard: inputs must suit the models, solutions must show a valid process, and difficulty must be challenging but partly feasible.

Method

VGI-bench: 27 tasks and 810 instances, organised by a two-level taxonomy of task domains and skill tags (Figure 1). Beyond scores, the authors analyse failure modes, sensitivity to input conditions, how far synthetic fine-tuning transfers, and the denoising process itself.

Key results

From the abstract:

• current systems solve some visually grounded reasoning tasks but are far from reliable
• the strongest model, Seedance 2.0 (summarised here in April 2026), achieves only 51.0%
• self-correction is limited: later denoising steps mainly refine early hypotheses rather than fix reasoning errors

Limitations

From the paper:

• tasks fit current clip lengths (about 5–10 s), so long multi-step procedures are out of scope
• image-to-video only, at a fixed 16:9 ratio
• English-only prompts and rubrics
• the suite is representative, not exhaustive