Paper summary
Paper summary: Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Prometheus 2 is an open evaluator LM that does both direct scoring and pairwise ranking against user-defined criteria, built to track human and GPT-4 judgements more closely than earlier open judges.
- One open judge model handles both common formats: scoring a single answer and choosing between two.
- The authors stress that model-based evaluation should be used with human evaluation, not instead of it.