#evaluation
- 27 Nov 2024 · Study
Machine Learning · Cross-validation
k-fold, stratified and grouped cross-validation, why one train/test split is noisy, and avoiding leakage by preprocessing inside the folds.
- 19 Nov 2024 · Study
Machine Learning · Model evaluation metrics
Confusion matrix, accuracy and why it fails on imbalanced data, precision, recall, F1, ROC-AUC, and regression metrics.
- 10 Jul 2024 · Paper summary
Paper summary: Vision language models are blind: Failing to translate detailed visual features into words
BlindTest's seven simple visual tasks, such as whether two circles overlap or how many times two lines cross, trip up strong vision-language models: four state-of-the-art VLMs averaged 58.07%.
- 3 May 2024 · Paper summary
Paper summary: Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Prometheus 2 is an open evaluator LM that does both direct scoring and pairwise ranking against user-defined criteria, built to track human and GPT-4 judgements more closely than earlier open judges.
- 23 Nov 2023 · Paper summary
Paper summary: GAIA: a benchmark for General AI Assistants
GAIA is a benchmark of 466 real-world questions that are easy for people but need reasoning, browsing, multimodality and tool use; humans scored 92% against 15% for GPT-4 with plugins.
- 12 Oct 2023 · Paper summary
Paper summary: Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
Prometheus is an open 13B evaluator LLM trained on GPT-4-generated feedback that scores long-form answers against a user-provided rubric, approaching GPT-4's agreement with human raters.