Paper summary
Paper summary: Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
Prometheus is an open 13B evaluator LLM trained on GPT-4-generated feedback that scores long-form answers against a user-provided rubric, approaching GPT-4's agreement with human raters.
- Given a reference answer and a scoring rubric, a 13B open model can grade answers about as well as GPT-4, according to the paper.
- Fine-grained, user-defined rubrics are the key input, not a generic "rate this 1–10".