#benchmark
- 16 Oct 2026 · Paper summary
Paper summary: OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
OmniCompass is a manually annotated benchmark for video grounding and understanding, with a vision track and an audio-visual track (when, where and what); across 50 MLLMs, the best scores are only around 40%.
- 5 Oct 2026 · Paper summary
Paper summary: RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations
RealCompanion releases ten real long-term human–AI companion relationships (27,218 messages over up to 120 days) and finds that people rarely refer back, that AI can't tell when the past matters, and that it reads more into people than they revealed.
- 27 Aug 2026 · Paper summary
Paper summary: VGI-Bench: Probing Visual Intelligence in Video Generation Models
VGI-Bench tests visual reasoning in video generation models with 27 tasks and 810 instances that require a valid process, not just a plausible final frame; the best model, Seedance 2.0, scores 51.0%.
- 8 Apr 2026 · Paper summary
Paper summary: Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
Video-MME-v2 is a harder, human-built video understanding benchmark with a three-level hierarchy and group-based scoring that only credits consistent, well-reasoned answers; the best model, Gemini-3-Pro, still trails human experts substantially.
- 24 Feb 2026 · Paper summary
Paper summary: A Very Big Video Reasoning Suite
VBVR is a video-reasoning dataset of 200 curated tasks and over a million clips, about 1,000× larger than earlier ones, with a rule-based benchmark used for an early study of how video reasoning scales.
- 30 Oct 2024 · Paper summary
Paper summary: CLEAR: Character Unlearning in Textual and Visual Modalities
CLEAR is an open benchmark for removing information about specific people from multimodal models, with 200 fictitious individuals, 3,700 images and linked Q&A, used to compare 11 unlearning methods.
- 23 Nov 2023 · Paper summary
Paper summary: GAIA: a benchmark for General AI Assistants
GAIA is a benchmark of 466 real-world questions that are easy for people but need reasoning, browsing, multimodality and tool use; humans scored 92% against 15% for GPT-4 with plugins.