Paper summary
Paper summary: Agentic Reinforced Policy Optimization
ARPO is an RL algorithm for multi-turn tool-using agents that samples extra rollouts right after tool calls, where the model's token entropy spikes, and is reported to beat trajectory-level RL with half the tool calls.
- Right after a tool returns, the model's next tokens become much more uncertain, so that's where ARPO explores more.
- The paper reports better results with only half the tool-use budget of existing methods.