Paper summary: Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

Problem

Video benchmarks are saturating: leaderboard scores look high while real abilities lag behind.

Method

Video-MME-v2 uses:

• a progressive three-level hierarchy: multi-point visual information aggregation, then temporal dynamics, then complex multimodal reasoning
• group-based non-linear evaluation: credit requires consistency across related questions and coherent multi-step reasoning, so fragmented or guessed correctness is penalised
• a controlled human annotation pipeline: 12 annotators, 50 reviewers, 3,300 human-hours, up to 5 rounds of quality assurance

Key results

From the abstract: a substantial gap between the best current model, Gemini-3-Pro, and human experts, with a clear hierarchy in difficulty across the levels.

Limitations

There’s no dedicated limitations section in the extracted text. Human-built benchmarks are expensive to extend.