OpenClaw-RL: learn from agent conversations

OpenClaw-RL (arXiv:2603.10165) trains a personal agent to satisfy the preferences of its user. It learns from their conversations: the user's next message shows whether a reply was accepted, and the model is updated while the agent stays in service.

What it does #

The learning signal comes from the conversation itself: when the user accepts a reply and moves on, the turn counts as positive, and when the user complains, it counts as negative. Reef reads this from the traffic it already records, so the agent does not send training reports. For reliable correlation, the harness should send a stable x-reef-tag-session value, unique to the conversation within its scenario, on every inference in that conversation. Reef stores this opaque tag with each record and the judging stage uses it to bind turns in arrival order, so one tag must not be shared by concurrent or distinct conversations. Without the tag, the judging stage falls back to matching the transcripts that clients resend on later requests.

How Reef implements it #

The judging stage rebuilds sessions from the recorded traffic. It prefers the x-reef-tag-session value supplied by the harness, which also supports agents that keep or rewrite their history locally. If the tag is absent, a chat agent must resend an extending transcript so each request can be matched to its session by trace. Identical histories can be ambiguous, while history changes that no longer preserve a prefix, such as compaction or sliding windows, break the chain and leave the pre-rewrite turn unmatched. The included Hermes example uses a small header shim because Hermes cannot set the tag itself.

A turn that never correlates is never judged and never trains, so a run on the fallback can produce an empty training set with nothing else to show for it. The recipe therefore says so: it warns once when it sees untagged traffic, warns again when sessions keep expiring without ever binding a turn, and counts both under correlation in the prm_record_file line. max_open_sessions caps the table those unmatched requests fill, evicting the least recently active sessions above it.

When a session's next state arrives, the finished turn is judged through an independently deployed PRM. The PRM votes on whether the message shows acceptance, and on acceptance it also proposes a hindsight hint, a short instruction that would have produced this reply if the user had given it up front. Judged turns are batched for training directly.

OpenClaw-RL trains with two signals, reinforcement learning and on-policy distillation (OPD). The RL term is a PPO clipped surrogate on the sampled tokens, with the turn's raw reward as its advantage. The OPD term pulls the policy toward a teacher on the top-K token candidates the engine recorded at generation time. The teacher is the frozen base model conditioned on one of the accepted hindsight hints, and the objective selects which hint by how well the teacher's top-K overlaps the policy's. The weights of the two terms are --openclawrl-w-rl and --openclawrl-w-opd in training.options, both 1.0 by default.

Configuration #

The deployment's pipeline section configures the recipe's three stages. The recipe's own classes are the defaults, so the section needs no module. The record-to-record stage correlates sessions and runs the PRM judge. The task-to-batch stage sets the batch size and writes the PRM record file. The record-to-task stage takes no config.

recipe:
  implementation: recipes.openclawrl.recipe:OpenClawRLRecipe
pipeline:
  record2record:
    config:
      prm_url: http://127.0.0.1:23001
      prm_tokenizer_path: /models/prm
  task2batch:
    config:
      batch_size: 16

Reef checks the stage config when it loads the recipe. An unknown key or an invalid value stops the deployment at startup. A stage key in recipe.config, such as prm_url, also stops it, and the error names the key's pipeline path.

The values above are the defaults. The example's serve.yaml overrides several of them. It points prm_url and prm_tokenizer_path at the example's independently deployed PRM engine. It raises prm_timeout_s to 3600 because a thinking model can spend minutes on a single vote. It sets prm_record_file, so each batch writes one line with its batch_id, the reward split, and the judge counters.

Reef coordinates inference and training only. OpenClawRL's example Compose file owns PRM and student-model startup, health checks and GPU allocation. The judging stage consumes pipeline.record2record.config.prm_url and requires the matching prm_tokenizer_path; it owns judge request handling. Reef does not register, schedule or stop the PRM process. The student-model endpoint is used by the simulation harness, outside Reef.

Run the example #

The example reproduces the paper's personal-agent experiment. A simulated student brings 72 GSM8K homework problems to a Hermes agent with one session per problem. Each session is scored on whether the agent's first solution reply satisfies the student's preferences.

docker build -f docker/Dockerfile.reef -t reef-openclawrl .
hf download Qwen/Qwen3-4B-Thinking-2507 --local-dir ~/models/Qwen3-4B-Thinking-2507   # policy and PRM
hf download Qwen/Qwen3-32B --local-dir ~/models/Qwen3-32B                             # the student

bash recipes/openclawrl/examples/openclawrl/run.sh

run.sh builds the simulated student's image, starts the stack that serve.yaml describes, and runs the 72 sessions in order. The stack uses seven GPUs: four for the tensor-parallel actor, one for the rollout engine, one for the PRM, and one for the Qwen3-32B student model.

Results #

The example's README records one run over the first 36 sessions of the stream.

A session passes when the agent's first solution reply already matches the student's preferences, with the work shown and the correct answer. The student wants homework that does not look AI-written, so a reply that uses bold text or a bullet or numbered list draws a complaint. The bold rate and the list rate measure this habit over the run, as the fraction of the last ten sessions whose first reply still contains bold text or a list. From the curves we see both fall as training goes on, and the run reaches the paper's adaptation criterion (three passed sessions in a row) at session 14.

The demo above replays two sessions from the same run. In session 1 the student rejects a formatted reply, reef keeps the training going, and by session 16 the first reply passes directly.

Related guides #

• Inference and feedback quickstart: learn the request, receipt, and report workflow.

• Train model weights from agent feedback: set up the GPU stack and inspect published updates.

• HTTP API reference: connect your agent and query feedback, scenarios, and releases.