Paper summary: Qwen2.5 Technical Report

Problem

Qwen2 (summarised here in July 2024) was strong among open models. Qwen2.5 aims to improve both pre-training and post-training.

Method

• Pre-training: high-quality pre-training data scaled from 7 trillion to 18 trillion tokens, for common sense, expert knowledge and reasoning.
• Post-training: supervised fine-tuning with over 1 million samples, plus multistage reinforcement learning. This improves human preference alignment, long text generation, structured-data analysis and instruction following.
• A range of sizes, with open-weight base and instruction-tuned models, plus quantised versions.

Key results

From the abstract: top-tier performance across benchmarks for language understanding, reasoning, maths, coding and preference alignment. The open-weight flagship Qwen2.5-72B-Instruct is competitive with Llama-3-405B-Instruct, which is about 5× larger. Two hosted MoE variants (Qwen2.5-Turbo and Qwen2.5-Plus) are proprietary.

Limitations

There’s no dedicated limitations section in the extracted text. Results are reported by the developers.