Paper summary: Agentic Reinforced Policy Optimization

Problem

RL with verifiable rewards works well for single-turn reasoning. Real agents interleave reasoning with multiple tool calls, and current RL algorithms balance long-horizon reasoning and multi-turn tool interaction poorly.

Method

The authors first observe that the entropy of the model’s generated tokens rises sharply right after a tool returns: the model becomes uncertain. ARPO uses this:

• an entropy-based adaptive rollout mixes whole-trajectory sampling with extra step-level branching at those high-uncertainty points
• advantage attribution estimation helps the model learn which tool-use steps made the difference

Key results

From the abstract: across 13 benchmarks in computational reasoning, knowledge reasoning and deep search, ARPO beats trajectory-level RL algorithms and reaches better performance using half the tool-use budget of existing methods. Code and datasets are released.

Limitations

There’s no dedicated limitations section in the extracted text. Results are on the authors’ chosen 13 benchmarks.