Paper summary: OpenClaw-RL: Train Any Agent Simply by Talking

Problem

According to the abstract, every agent action is followed by a next-state signal (a user reply, a tool output, a terminal or GUI change), yet no existing agentic RL system used it as a live, online learning source.

Method

Infrastructure: the RL server hosts the policy behind an inference API, user terminals stream interaction data back over HTTP, and a separate asynchronous server extracts training signals, so neither extraction nor optimisation blocks inference.

Method: from each next state it extracts:

• evaluative signals: broadly available, like a reward
• directive signals: richer, token-level guidance, but sparser

A hybrid RL objective combines both in one update. For stability, overlap-guided hint selection picks the hint whose teacher distribution best overlaps the student’s top-k tokens, and a log-probability-difference clip bounds per-token advantages.

Key results

The abstract presents a personal agent that improves simply by being used, recovering signals from user re-queries and corrections.

Limitations

There’s no dedicated limitations section in the extracted text. Learning online from real users raises obvious questions about noisy or adversarial feedback, which the extracted text doesn’t address.