Paper summary
Paper summary: GAIA: a benchmark for General AI Assistants
GAIA is a benchmark of 466 real-world questions that are easy for people but need reasoning, browsing, multimodality and tool use; humans scored 92% against 15% for GPT-4 with plugins.
- GAIA deliberately targets tasks that are simple for humans but hard for AI, the opposite of "harder exams".
- Answers are short and unique, so grading is automatic, but the reasoning trace isn't evaluated.