Train with Tinker
The optional tinker backend runs LoRA training and sampling through Tinker's SDK. Reef runs on a CPU host and retains ownership of requests, feedback, candidate selection, and the versioned snapshot chain. By default it launches no Ray cluster, no Slime and no inference engine; Serve on inference engines describes the option to serve the trained adapters on SGLang engines. The tested SDK interface is pinned to tinker==0.28.1.
Install and run #
Use Python 3.12 or newer on Linux or macOS, the version Reef itself requires.
uv pip install -e '.[tinker]' -e ./third_party/reef-clientSet TINKER_API_KEY in the service environment. Credentials are read only when the selected runtime starts; neither resolved configuration nor checkpoint manifests contain the key. training.options.api-key-env selects a different environment variable. An optional project-id selects the Tinker project.
training.options.base-url selects another Tinker-compatible service, such as a Spindle deployment on Modal. Unset, Reef uses Thinking Machines' hosted Tinker. The trainer, its sampler and inference.serving: trainer all use that service. A checkpoint path such as tinker://<run>/weights/<name> resolves only on the service that made it, so each checkpoint manifest records the service, and the component refuses a checkpoint that another service holds. A service may lack some calls; Spindle, for example, serves no checkpoint archives, so Reef's own engines can serve only hosted Tinker's checkpoints (Serve on inference engines). Reef makes a base-model sampling client only at its first use (a KL term, a task rollout or a request); on Spindle such a client starts a sampler pool.
The four-rollout smoke includes a complete config and runner. It creates one adapter, generates four short responses, and performs one optimizer step with synthetic rewards. Running it uses Tinker credits; it is a mechanism check, not a benchmark or a model-quality claim.
Choose training.backend: tinker, an available Tinker base model in inference.model-path, and a persistent training.options.state-dir. The model remains a remote identifier: deployment does not download its weight snapshot. The SDK obtains tokenizer resources for prompt rendering. The runtime requires exclusive ownership of its state directory. The Tinker training component lets several scenarios share it, because each job carries its own start. When Tinker samples (inference.serving: trainer, the default), a deployment still trains one scenario. When Reef's engines serve the exported adapters (Serve on inference engines), they serve each scenario its own adapter, so several scenarios can train on one deployment.
Supported inference and training #
The backend supports /v1/chat/completions with textual system, user, and assistant messages, including a final assistant prefill. It accepts one output per request, max_tokens or max_completion_tokens, temperature, top_p, top_k, seed, and stop. The supported template option is chat_template_kwargs.enable_thinking. Unknown options, tool calls, multimodal messages, and Anthropic routes fail explicitly.
A request without max_tokens or max_completion_tokens may reply up to the rest of the model's context window, or 1024 tokens when the service states no window for the model. A prompt that fills the window gets a 400 whose message starts with context_length_exceeded. A thinking model's reasoning comes in the message's reasoning_content, and its answer in content.
Streaming is buffered: the complete sampled response is emitted as valid OpenAI SSE events. It does not provide token-by-token latency. A streamed answer carries only content, so a streaming client gets no reasoning. Prompt IDs come from the tokenizer's chat template.
Serving returns and records no training tensors. The training component renders each recorded call with the base model's tokenizer and training.options.chat_template_kwargs, which must equal the chat template arguments of serving, for example {enable_thinking: false}. It computes the rollout log probabilities with a forward pass at the job's start, before the first optimizer step. A sample that cannot render, or whose tokens after the first are more than the model's context window, does not train, and the job metric unrenderable_samples counts it: Tinker refuses a whole optimizer step for one such sample. A task job's own rollouts keep the sampled IDs and log probabilities that Tinker returns.
Reef's normal WeightTrainingRecipe drives the integration. The backend is one training component (reef.train.tinker_backend) and one inference runtime (reef.inference.tinker). Reef submits each job's ready data to the component as one training job. They share nothing in process: each saved checkpoint carries a tinker-checkpoint.json manifest that names its remote training state and sampler. The inference side serves the sampler that a snapshot names. Each job starts from the scenario's last saved checkpoint, which state-dir keeps across restarts.
A new scenario starts from a base root: a base.json file that names the model, the LoRA rank and the service. Control writes it from the configuration and calls no Tinker service for it. Serving it samples the base model. The first job from it makes a new LoRA from seed. With seed: null, the training service saves one untrained adapter when the first such job runs, keeps its manifest in <state-dir>/base/, and starts every such job from it.
training.options.batch-size (default 1) is the batch size: the samples in one optimizer step. pipeline.sample2job.config.job_samples is the job size: the training pipeline submits a job when it holds this many samples. Set it to the batch size, so that one job is one optimizer step. Job data takes whole record sets, so it can hold more; the component then cuts it into optimizer steps of the batch size (see Job size). A step keeps each rollout of the objective's rollout_unit whole: one sample, or one comparison set. A job's optional scheduling parameter sets the epochs, the shuffle, and the partial, drop or error policy for the last step.
Tinker computes the loss on its side from a built-in loss function and the per-sample inputs that function reads. The built-in importance_sampling family places the trajectory advantage on every response token the loss mask selects. The built-in sft family trains Tinker's cross_entropy on the same tokens, so the SFT recipe runs on Tinker; its samples need no rollout log probabilities. A method gives its own TinkerLoss a loss_family name and decorates the class with @register_loss_family (reef.train.tinker_backend.loss_families), in the module of its TinkerObjective; the class is imported when a job first names that objective. Its loss_fn names the Tinker built-in it trains with and inputs shapes that function's loss_fn_inputs. A TinkerCustomLoss instead computes the loss on the Reef host from the token log probabilities Tinker returns, through forward_backward_custom; that path needs torch locally and no shipped recipe uses it. The TTTD recipe's recipes/tttd/tinker.py keeps its unmasked policy term and masked, centered frozen-base KL on importance_sampling, matching its Slime objective. training.options.kl-coef controls that penalty. The base sampler scores the same token IDs, and those probabilities never replace the rollout log probabilities. SAO and OpenClaw-RL losses are not implemented by this integration; selecting an unregistered loss family fails before training.
A job's data can also be Harbor tasks (a task job), as the generated-task recipe makes. Before it trains, the component runs rollout.rollouts_per_task rollouts of each task on Tinker's sampler at the job's start weights, so the rollouts sample the weights that the job trains. The agent's model calls go to a sampler endpoint on this host, which answers Tinker's text chat with the training capture; the task's verifier scores each rollout. The component trains on each task's group of samples, keeps the rollouts in its job directory so a resumed job trains the same samples, and reports each task's rewards in the job's metrics. The rollout job parameter sets the rollouts, the agent and the sandbox. Reef's serving takes no part, so another runtime, such as an inference component, can serve the snapshots.
Each job trains the Tinker loss of the objective that its recipe selected (Train each kind of sample with its own method). Tinker preparation needs one advantage per trajectory, so a supervised or distillation objective fails before training. kl-coef applies to every job. When it is not zero, a job whose loss has no KL penalty, such as importance_sampling, also fails before training.
lora-rank, seed, and learning-rate configure adapter initialization and Adam updates. Other Adam settings currently use the pinned SDK's defaults (beta1 0.9, beta2 0.95, epsilon 1e-12, zero weight decay and gradient clipping). learning-rate applies until a recipe selects a learning-rate schedule. training.timeout-s bounds SDK futures and configures transport timeouts; inference.timeout-s bounds sampling. These timeouts do not cancel a remote update that the service has already accepted.
Jobs, checkpoints, and recovery #
A job trains the optimizer steps of its data from the job's start checkpoint. It restores the saved weights and optimizer state in a new Tinker training session. It then saves the state (save_state) and the sampler weights (save_weights_for_sampler). Each save has one explicit unique name and no expiration. Then the session closes. The job parameter save_interval, which recipe.config.job_parameters can set, gives the optimizer steps between two saved checkpoints. Without it, a job saves one checkpoint when it ends. Each saved checkpoint is a candidate. Candidate evaluation decides whether Reef publishes it; a rejected candidate changes no served sampler. The next job starts from the last saved checkpoint, whether or not Reef selected it.
An uncertain API result may leave an unreferenced remote checkpoint and consume credits. A retry starts from the same checkpoint in a fresh session; it cannot apply the uncertain gradient a second time to that model. A job that a restart interrupts continues after its last saved checkpoint. This favors recovery correctness over session reuse and training throughput.
Reef publishes a small local tinker-checkpoint.json artifact containing the schema version, base model, LoRA rank, training-state URI, and sampler URI. After a recipe selects a learning-rate schedule, it also records the schedule's progress. It contains references, not portable weight tensors. Restore, activation and train-from require continued access to both remote checkpoints in the originating account/project. Local disk retention does not remove remote files. Use Tinker's checkpoint management for remote retention, preserving all checkpoints referenced by Reef.
Requests that are already admitted keep the sampler of the snapshot they started with. Restart creates a fresh runtime-load incarnation and serves the active selection. Train-from works: Reef imports the older artifact's manifest into the component, cancels the running job that continues the start before, and the next job that names no start begins from that artifact's weights and optimizer state.
Serve on inference engines #
Reef's own SGLang engines can serve the adapters that Tinker trains, in place of Tinker's sampler. The engines run on an inference component (python -m reef.inference.remote), a service separate from the trainer. The Reef control plane starts the engines on the component and loads each saved adapter into them. Tinker still trains in the HTTP service's process, and the trainer never talks to the engines.
Select inference.backend: sglang with the base model that Tinker trains in inference.model-path, inference.num-gpus and, for tensor-parallel engines, inference.tensor-parallel-size. Any engine setting selects inference.serving: engines:
inference:
model-path: Qwen/Qwen3-8B
backend: sglang
num-gpus: 2
max-loaded-adapters: 2
training:
backend: tinker
options:
state-dir: /var/lib/reef/tinker
lora-rank: 32reef serve then starts an inference component on this host before the HTTP service. The component listens on a free loopback port and starts its own Ray session; with training.ray-address it joins that cluster instead. reef serve generates the two component tokens (REEF_REMOTE_CONTROL_TOKEN and REEF_REMOTE_INFERENCE_TOKEN) unless they are set, and both children inherit them. The HTTP service starts the engines before it serves, so its readiness deadline is at least training.ready-timeout.
The engines start with LoRA serving enabled: max-lora-rank is lora-rank and lora-target-modules is all. max-loaded-loras and max-loras-per-batch come from inference.max-loaded-adapters. The engines keep at most that many adapters resident. Each scenario keeps its served adapter resident, so this value must be at least the number of training scenarios; a scenario's new adapter replaces its own served adapter when no slot is free.
Each scenario trains its own adapter. A job runs the optimizer steps of its data on Tinker from the scenario's last saved checkpoint. A snapshot still holds only its manifest. When the control plane loads a snapshot, it downloads the adapter that the manifest's sampler_path names, as tinker-cookbook's weights.download does, and converts it into a PEFT adapter with weights.build_lora_adapter. It keeps the result under <state-dir>/sampler-weights/, one directory per sampler path, so a snapshot downloads once. A service that serves no checkpoint archives, such as Spindle, fails the load. The control plane loads the PEFT adapter into every engine under a new adapter name for the scenario. Requests reach that adapter through the scenario's served_models route. A rejected candidate loads nothing. A restart starts the engines again and loads each scenario's committed adapter before the scenario serves. The converter is pinned in the tinker extra; it brings torch and transformers.
The trajectories then come from Reef's engines while Tinker computes the loss, so the base model and tokenizer on both sides must match. Set training.options.chat_template_kwargs to the engines' chat template arguments, so that Tinker renders each sample as the engines served it. reef serve refuses training.colocate: there are no training GPUs to share. Tinker's adapters also carry a LoRA on lm_head, which the Slime configuration never targets; whether the installed SGLang accepts that module is checked only when the engines load the adapter.
Serve from a remote GPU host #
The component can also run on another host, such as a rented GPU machine or a managed GPU platform, while Reef and the Tinker trainer stay on a CPU host. No tensors cross the network: the control plane uploads each selected adapter directory to the component, and the component loads it into its engines.
On the GPU host, install Reef with SGLang. Set two different tokens and start the component:
export REEF_REMOTE_CONTROL_TOKEN=... # starts engines and replaces weights
export REEF_REMOTE_INFERENCE_TOKEN=... # sends inference requests
python -m reef.inference.remote --port 8000The component starts its own Ray session and no engines. Its control routes other than health answer 409 until a control plane starts the engines. It starts only the engine backend that --engine names (default sglang) and refuses any other backend that a control plane asks for. python -m reef.inference.remote --help lists the bind address, the upload directory and the Ray options.
On the Reef host, set the same two variables. Keep inference.backend: sglang and add an inference.remote section with the component's URL:
inference:
model-path: Qwen/Qwen3-8B
backend: sglang
num-gpus: 2
remote:
url: https://gpu-host.example.com
training:
backend: tinker
options:
state-dir: /var/lib/reef/tinker
lora-rank: 32reef serve then starts no local component. The control plane asks the remote component to start the engines with the settings described above. The GPU host downloads model-path; the Reef host keeps it as a remote identifier.
The HTTP service sends inference requests to the component, and the component forwards each request and its answer unchanged. Version stamps therefore come from the engines. When the control plane stops, the component stops its engines and keeps running. When the component restarts, for example because its host replaced the container, it runs no engines. The control plane notices within 30 seconds, starts the engines again and loads each scenario's served adapter into them.
The control token gives its holder control of the GPU host. Its holder chooses the model, the engine options and the served weights, and the engines run a model's own code (trust_remote_code). Protect it like a login to that host: serve the component over HTTPS, keep the token out of logs and deployment files, and keep it separate from the inference token.
Each selected candidate uploads one adapter directory, so the upload time grows with the adapter size and the network path. The component serves one control plane at a time: a second start replaces the first one's engines. The inference section lists the inference.remote settings.
tests/tinker_backend/test_tinker_local_engine_ray.py runs this topology on a CPU. The component runs a stub engine in its own process and Ray session, and the test process is the control plane with the Tinker SDK faked. It trains one TTTD step, checks that the component serves the adapter, restarts the control plane, and checks that the component serves the committed adapter again. With REEF_TEST_TINKER_REAL=1 and TINKER_API_KEY set, it trains, downloads and converts a real checkpoint.
Train all the parameters on Spindle #
The spindle_full backend trains all the parameters of a model on a Spindle deployment's full-parameter engine, through the same job loop as tinker. Use it to keep training a checkpoint whose weights are on Hugging Face, such as one that was trained outside Reef.
Prerequisites:
-
The checkpoint is a Hugging Face revision, hf://<repo>@<commit>, with a full 40-digit commit SHA.
-
The Spindle deployment has a full-parameter engine whose model is <repo> and whose weights are that commit. Reef does not start engines; deploy the engine with Spindle first.
-
The service environment has the Spindle API key, in TINKER_API_KEY or the variable that training.options.api-key-env names, and a Hugging Face token that can read the repo, such as HF_TOKEN.
-
The service has Modal credentials (uv pip install 'reef-infra[modal]', then MODAL_TOKEN_ID and MODAL_TOKEN_SECRET or a profile). In the Spindle deployment's Modal environment, the Secret hf-secret names (default huggingface-secret) holds an HF_TOKEN that can write release-repo and read the base.
inference:
model-path: org/model # the base's repo
training:
backend: spindle_full
options:
state-dir: /var/lib/reef/spindle-full
base-url: https://<workspace>--spindle-server.modal.run
base: hf://org/model@0123456789abcdef0123456789abcdef01234567
release-repo: org/model-releases # one commit per saved checkpoint
modal-environment: spindle # where the deployment runs
learning-rate: 1.0e-6The default start is a reference to the base revision (Import an artifact), which a new scenario's root snapshot holds. Control writes it from the configuration. A job from it starts a new model with the base's own weights.
Every snapshot refers to its weights on Hugging Face. When Spindle saves a checkpoint, it writes the checkpoint's Hugging Face weights to its checkpoint Volume (checkpoint-volume, default spindle-checkpoints). A Modal function then commits those weights, with the base's tokenizer, chat template and processor files, to release-repo. Each saved checkpoint is one commit, which holds exactly that checkpoint's files. The snapshot holds two files:
-
hf-reference.json, the commit with the weights. Any serving runtime can load it: Reef's engines (inference.serving: engines) download it, and a runtime that loads from the Hub reads the reference.
-
tinker-checkpoint.json, a manifest with lora_rank: null. A later job restores its weights and optimizer state, and Spindle's sampler, the default serving (inference.serving: trainer), serves its sampler path.
An import into a spindle_full scenario must be a full-parameter manifest of the base model on the same service beside a reference to full weights, or the base's own files, by reference or as a copy. The engine trains only its base, so other full weights are refused with HTTP 400. To train another checkpoint, deploy a Spindle engine for it and a scenario with it as base. A component trains one base.
Score centering #
A policy gradient whose token weight is not the full importance ratio p / q drifts the trained policy p toward the sampler q. Score centering (arXiv:2609.20807) removes that drift. With training.options.score-centering: true, a spindle_full component trains each importance_sampling job, such as a generated-task or task job, as Spindle's cispo loss with the centering term:
-
cispo weights each sampled token's score by clamp(p / q, score-centering-weight-low, score-centering-weight-high). The default bounds, 1 and 1, are REINFORCE. Wider bounds, such as 0 and 2, are truncated importance sampling. The term is zero under the full ratio, so the weight cannot be the full ratio.
-
A task job's rollouts record the sampler's top score-centering-top-k log probabilities (default 128) at each sampled token. Those must come from the distribution that the trainer reads, so the rollouts sample at temperature 1 without truncation. A model call that sets another temperature, or a top_p or top_k, gets HTTP 400. Harbor's terminus-2 agent sets none of them by default.
-
Only a task job's rollouts record the sampler's top-k. Serving records none, so a job of samples that serving recorded, or of uploaded trajectories, fails with score centering on.
-
sft jobs train as they are. A job of another policy-gradient loss family is refused, since the term must match the gradient it corrects. So is a nonzero kl-coef.
training:
backend: spindle_full
options:
score-centering: true
score-centering-top-k: 128 # sampler log probabilities per sampled token
score-centering-weight-low: 1.0 # 1 and 1: REINFORCE
score-centering-weight-high: 1.0
score-centering-min-tail-mass: 1.0e-6The Spindle deployment must implement the sampler's topk_sample_logprobs and score centering on cispo (loss_fn_config.score_centering with the sampler_topk_token_ids and sampler_topk_logprobs inputs). A deployment without them refuses the rollouts' sample requests or the training call, so the job fails. Spindle reports the term in each job's metrics, as score_centering:sum, score_centering_sampler_head_mass:mean and score_centering_trainer_head_mass:mean.
A full-parameter run is single tenant: each job holds the engine's training GPUs while it runs, for example 8 H200 GPUs to train and 4 to sample for a 27B model. Each job also starts a new model on the engine, so it pays the engine's start time, and unloads the model when it ends, so the next job gets the engine. For a 4B model on a scoped Spindle run, the first job took about 6 minutes and a job from a saved checkpoint about 4. The export of the 4B weights (9 GB) took about a minute.
Verification scope #
CPU tests exercise data alignment and loss math, SDK call ordering, rejected and uncertain candidates, commit admission, request capture and SSE, deployment, scenario commit/restart/activation/train-from, and serving on engines (per-scenario adapters, rejection, restart reload) against engine and SDK doubles. The component topology is tested with stub engines only; real SGLang engines have not been run with it. A live Tinker run is additionally needed to verify account model availability, remote persistence, and training quality. No live training or performance result is claimed by this implementation. spindle_full ran live once, on a scoped Spindle run of Qwen3.5-4B's full-parameter engine:
-
two jobs: one from the base, one from the first saved checkpoint;
-
the import check of the second checkpoint;
-
activation and sampling of it;
-
the export of its weights to a private Hugging Face repo, run separately on that checkpoint.
The training-quality effect was not measured.