Paper summary: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Problem

Earlier reasoning gains relied heavily on human-annotated demonstrations of reasoning, and models still fell short on complex problems.

Method

The paper uses pure reinforcement learning to bring out reasoning, without human-labelled reasoning trajectories:

• DeepSeek-R1-Zero is trained with large-scale RL directly from a pretrained checkpoint.
• DeepSeek-R1 is trained through a multi-stage pipeline built on that, to also improve broader areas such as writing and open-domain question answering.

Rewards come from verifiable outcomes (correct maths answers, passing code), so no human judges each step.

Key results

The authors report that advanced patterns, self-reflection, verification and dynamic strategy adaptation, emerged during RL. The resulting model does better on verifiable tasks (maths, coding competitions, STEM) than counterparts trained conventionally on human demonstrations. They conclude that pretrained checkpoints already hold substantial reasoning potential, unlocked by hard questions, a reliable verifier and enough RL compute.

Limitations

The paper lists capability limitations, starting with structured output and tool use, where DeepSeek-R1 is weaker than existing models. The extracted section continues with further limitations that are outlined in the paper.