SDFT: learn from demonstrations without forgetting

Self-Distillation Fine-Tuning (arXiv:2601.19897) learns from demonstrations on policy. The model that reads a demonstration in its prompt is the teacher. The same model without the demonstration is the student. The student samples the response and the loss pulls its next-token distributions toward the teacher's. The paper reports higher new-task accuracy and less forgetting than supervised fine-tuning on the same demonstrations.

Evolves	model weights
Signal	one report with the demonstration as teacher_context per rollout
Loss family	sdft
Package	recipes/sdft/
Processor	reported feedback, singleton
Needs	GPUs, and a backend that captures tokens and log-probs
Example	SDFT on a skill stream

What it does #

The harness sends a request through Reef. It gets a demonstration of the response from somewhere else such as a reference solution or a stronger model. It reports the demonstration against the request's receipt. With the default batch_size of 1 each report is one training step.

Rolloutthe student answers a request

Demonstrationa reference response for the same request

Reportthe demonstration as teacher_context against the rollout's receipt

Stepdistil the demonstration-conditioned teacher on the student's own tokens

Versionpublish the updated weights to the engine

the next request is served by the updated weights

How Reef implements it #

The processor is the shared DistillProcessor (Processors). It turns every TeacherContextReport into one TrajectoryItem. The item carries the student's recorded tokens and teacher_tokens. teacher_tokens is the teacher's request rendered with the served model's chat template and followed by the student's response ids.

SDFT's processor adds the demonstration block to the request's final user message. It adds the block as a new user message when the request ends in a tool result. context_template sets the block's text and its default is the reference implementation's wording. A harness that needs another layout subclasses the processor and overrides teacher_request.

The sdft loss family is a thin family on the Slime backend's distillation base in reef/train/slime_backend/distill/ (Loss families). It sets the reference's defaults and runs as a custom_loss. Before each step the base runs one forward pass over every sample's teacher sequence and keeps the teacher's next-token distribution at each response position. The loss compares the student's distribution at the same positions with it.

The teacher is the model itself (--sdft-teacher self) and --sdft-teacher-update-rate sets how its weights move. The default is 0.01. The teacher starts as a copy of the initial weights and moves 1% toward the policy after every step. A rate of 1 makes the current policy the teacher and a rate of 0 freezes the initial weights. In the skill stream example a teacher that follows the policy drifted with it and training collapsed. The example freezes the teacher with a rate of 0.

The loss is the per-token divergence over the full vocabulary. The forward KL is the default and the reverse KL and the generalized JSD are switches. A top-K representation of the teacher is another switch. Each sample contributes the mean over its trained response tokens. That mean is weighted by the truncated importance-sampling ratio between the policy and the rollout engine's log-probs.

The report contract #

A report references one inference record and carries the demonstration as metadata.teacher_context. A score is optional metadata and the recipe never trains on it.

JSON

{
  "references": ["<receipt of the student's request>"],
  "metadata": {"teacher_context": "<the demonstration>"}
}

Configuration #

batch_size1rollouts per optimizer step. Must equal the driver's --global-batch-size because each sample is its own data-parallel unit.

tokenizer_pathrequiredthe served model's tokenizer directory. It renders the teacher prompt with the engine's chat template.

max_teacher_tokens0a report with a longer teacher sequence is skipped and counted in teacher_overflow_reports. 0 disables the check. Set it to the trainer's window.

context_templatethe reference's blockthe text added to the final user message. {context} marks where the demonstration goes.

max_staleness0accepted lag between the producing and serving version.

The Slime driver takes --loss-type custom_loss and --use-rollout-logprobs and --disable-compute-advantages-and-returns. The family adds its own flags:

--sdft-teacherselfwho scores the samples. self is the model itself reading the demonstration. separate is another checkpoint of the same architecture set by --sdft-teacher-checkpoint.

--sdft-divergenceforwardforward is KL(teacher | | student). reverse is KL(student | | teacher). jsd is the generalized Jensen-Shannon divergence and --sdft-jsd-beta sets the teacher's mixture weight with a default of 0.5.

--sdft-top-k0keep the teacher's top-K log-probs and its log-prob at the sampled token. 0 keeps the whole distribution.

--sdft-teacher-update-rate0.01fraction of the current policy mixed into the teacher's weights after every step. 1 makes the current policy the teacher and 0 freezes the initial weights.

--sdft-importance-sampling-cap2.0cap of the truncated importance-sampling weight. 0 disables the correction.

--sdft-skip-response-tokens0response tokens at the start of every sample left out of the loss. The paper's runs used 3.

The teacher pass keeps one float16 block per sample on the host between the pass and the step. The block has one row per response token and one column per vocabulary entry on the rank. That is about 2 GB for a 16k-token response of a 248k-vocabulary model at tensor parallel 4. Size max_teacher_tokens with that in mind for long contexts. A moving teacher copy also costs the actor's weights once more on the host per rank.

Run the example #

The example trains one model on Tool Use and then on Science Q&A on a four-GPU stack. A judge scores both skills every ten steps during both stages. The example's README describes the protocol and the settings.

Shell

cd recipes/sdft/examples/skill_stream
hf download Qwen/Qwen2.5-7B-Instruct --local-dir ~/models/Qwen2.5-7B-Instruct
./run.sh

Results #

The figure shows one run of SDFT and one run of an SFT control on Qwen2.5-7B-Instruct.

Both skills' test accuracy against gradient steps for SDFT and the SFT control

Tool Use training comes first and both methods learn it to about the same level. SFT pushes Science Q&A below the base model and SDFT does not.

Science Q&A training comes second and both methods learn the new skill. SDFT forgets only a little Tool Use and SFT forgets much more.

Inference and feedback quickstart: learn the request, receipt, and report workflow.
Train model weights from agent feedback: set up the GPU stack and inspect published updates.
Loss families: how a family such as sdft plugs into the Slime backend, and the distillation base it is built on.