Paper summary: Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

Problem

Using GPT-4 as a judge of long-form answers had become standard. For practitioners, though, a closed model is a problem: versions change silently, costs add up at scale, and custom criteria (“is this readable for a child?”) are hard to control.

Method

The authors built the Feedback Collection with GPT-4: about 1K fine-grained score rubrics, 20K instructions and 100K responses with written feedback. They trained Prometheus, a 13B open model, to take an instruction, a response, a reference answer and a rubric, and output written feedback plus a score.

Key results

As reported in the abstract:

• On 45 custom rubrics, Prometheus’s scores correlate with human evaluators at Pearson 0.897, on par with GPT-4 (0.882) and far above ChatGPT (0.392).
• Across 1,222 rubrics on four benchmarks, its correlation with GPT-4 follows similar trends.

Limitations

The evaluator is trained on GPT-4-generated feedback, so it inherits that teacher’s biases. It also relies on having a good reference answer and rubric, which the abstract makes a condition (“when the appropriate reference materials … are accompanied”).