Paper summary: Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

Problem

GPT-4 is often used to grade other models’ answers, but closed judges raise transparency, controllability and cost problems. According to the abstract, existing open judges (including the first Prometheus, summarised here in October 2023) gave scores that diverged from humans, supported only one of the two main formats, and mostly judged generic qualities like helpfulness rather than custom criteria.

Method

Prometheus 2 is trained to do both direct assessment (score one response) and pairwise ranking (pick the better of two), in each case against a user-defined evaluation criterion.

Key results

The paper evaluates on four direct-assessment and four pairwise-ranking benchmarks. It reports that Prometheus 2 has the highest correlation and agreement with human judges and proprietary LM judges among the open evaluators tested.

Limitations

From the paper’s own section:

• Agreement with humans or GPT-4 and Claude-3-Opus is an indirect measure of evaluation quality, and may not be the best one.
• Model-based evaluation should be used together with human evaluation, not as a replacement.
• Claims about generalisation rest on an overlap analysis between training and evaluation data.