Study
LLMs · Evaluation
How LLMs are evaluated: perplexity, multiple-choice benchmarks scored by log-likelihood, exact match, LLM-as-a-judge, and the contamination problem, with a tiny log-likelihood multiple-choice eval on GPT-2.
- Many multiple-choice benchmarks are scored by which answer option the model assigns the highest likelihood, not by free-form generation.
- A benchmark leaked into training data stops measuring generalisation.