Paper summary
Paper summary: Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
Video-MME-v2 is a harder, human-built video understanding benchmark with a three-level hierarchy and group-based scoring that only credits consistent, well-reasoned answers; the best model, Gemini-3-Pro, still trails human experts substantially.
- Group-based scoring penalises lucky guesses: related questions must all be answered consistently to earn credit.
- 3,300 human-hours and up to five rounds of quality checks went into the data.