OPD: learn from a separate teacher on your own responses

On-policy distillation lets a frozen teacher score the tokens the student actually generated. Reef trains the student's weights on that signal and publishes them before the next batch is sampled. The method follows Thinking Machines Lab's write-up.

The recipe lives in recipes/opd/ and uses Reef's Slime distillation backend. Inference runs on SGLang; training runs on Megatron.

The learning loop #

Samplethe student answers prompts through Reef

Reportsubmit one empty teacher-context report per inference receipt

Teacherscore the exact recorded prompt and response ids with frozen weights

Trainminimize sampled-token reverse KL with one optimizer update

Publishserve the updated student weights

wait for publication before generating the next batch

Teacher and student see identical token sequences. The processor uses the captured inference tokens, preserving thinking switches and assistant prefixes, and loads no tokenizer. It rejects a non-empty teacher context. A reported score is kept as metadata and does not enter the loss.

Report contract #

Each report references exactly one inference receipt:

JSON

{
  "references": ["<inference receipt>"],
  "metadata": {"teacher_context": ""}
}

Configuration #

Select recipes.opd.recipe:OPDRecipe. batch_size responses form one optimizer step. max_staleness defaults to zero: wait for each published update before sampling the next batch.

batch_size1responses in one optimizer step.

max_teacher_tokens0maximum recorded sequence length; a longer report is released and counted in teacher_overflow_reports. 0 disables the check.

The Slime options belong in training.options:

opd-teacherseparatethe teacher's weights come from opd-teacher-checkpoint.

opd-teacher-checkpointrequiredthe teacher checkpoint, loaded into the actor's layout for the teacher pass.

opd-divergencereversesampled-token KL(student | | teacher).

opd-top-k1selects the sampled-token reverse-KL estimator; the sampled token's probability is normalized over the full vocabulary.

opd-teacher-update-rate0the teacher stays frozen.

opd-importance-sampling-cap0no truncated importance-sampling correction: one on-policy update per batch.

Also set loss-type: custom_loss, use-rollout-logprobs: true and disable-compute-advantages-and-returns: true. The family rejects multiple optimizer updates per batch. The shared backend's other divergences and representations are reachable through the inherited opd-* flags; changing them changes the method.

The separate checkpoint is swapped into the actor's architecture for the teacher pass, so teacher and student must share the architecture, parallel layout and token ids; a larger model with only the same tokenizer runs on a teacher engine instead. The teacher copy takes additional host memory and reuses the actor's GPUs without an optimizer update. In LoRA mode the student's adapter is cleared while the teacher's weights are loaded.

Teacher engine #

A teacher that shares only the student's token ids, such as Qwen3-32B for a Qwen3-8B student, is served as a frozen SGLang engine on its own GPUs. The deployment's teacher section declares it and replaces opd-teacher-checkpoint:

YAML

teacher:
  model-path: Qwen/Qwen3-32B
  num-gpus: 2
  port: 30001
  options:
    context-length: 32768
training:
  options:
    opd-teacher: separate
    opd-top-k: 1
    rollout-temperature: 1.0

num-gpus reserves the teacher's GPUs beyond the student's and sets its tensor parallel size, port is its internal endpoint, timeout bounds one scoring request, and options are native SGLang options. Before each optimizer step the bridge sends every recorded prompt and response to the engine and stores the teacher's log-probability of each sampled token, so the loss stays the sampled-token reverse KL. The engine path requires rollout temperature 1, because SGLang's input log-probs are not tempered, and the teacher-selected top-K representation; the teacher's tokenizer must map tokens to the student's ids, and its context limit must cover the longest recorded sequence. A scoring failure fails the step before it starts, so no partially scored batch trains. perf/distill_teacher_time reports the scoring time of each step in seconds.