Deploy and train

A weight-training deployment runs three processes: a Ray head, the slime driver that owns the GPUs and the optimizer, and the reef HTTP service that owns records, versions, and the request plane. One config file starts all three, and reef serve -c <config> is still the only command.

Harness and skill recipes need none of this. They publish text artifacts, so the single-process config from get started is already a real deployment for them, and only the storage and day-two sections below apply.

The stack#

reef serve -c reads the config's services list and starts each process in the order shown. It waits for one service's ready check before starting the next. The config supports:

• ${VAR} to read a launch environment variable.
• ${dotted.path} to reuse another value from the same config.
• cuda: to choose GPUs for a service.
• env: to add service-specific environment variables.

Four complete configs ship with reef:

They are smoke stacks that run real processes with deliberately small models. Copy one and change the model, GPU counts, and storage paths rather than starting from an empty file.

Per-service logs are written under /tmp/reef-stack/ unless run_dir changes the location. If a service fails or misses its readiness timeout, reef stops the stack. It does not automatically restart failed GPU processes.

One reef process trains exactly one scenario for its lifetime. A second training scenario is refused the first time it resolves, and needs its own stack on its own ports. Non-training scenarios are unaffected.

Configure training#

There is no args file to maintain. Slime auto-fills the Megatron architecture flags from config.json in the HF directory at reef.model_path: layers, hidden and FFN sizes, attention heads, GQA, RoPE, norm epsilon, embedding tie, and MoE sizes. It validates them against that same config, so a mismatched checkpoint still fails loudly. What stays yours is the deployment: GPU layout, parallelism, optimizer, sequence length, schedule, and the objective's own knobs. Those live in one literal string that is interpolated verbatim into the slime-driver command, which makes adding a slime flag a YAML-only change:

training:
  slime_flags: >-
    --actor-num-nodes=1 --actor-num-gpus-per-node=1 --rollout-num-gpus=1
    --rollout-num-gpus-per-engine=1 --num-gpus-per-node=2
    --tensor-model-parallel-size=1 --pipeline-model-parallel-size=1
    --seq-length=4096 --optimizer=adam --lr=1e-6 --lr-decay-style=constant
    --loss-type=policy_loss --use-rollout-logprobs

The driver must also be told which loss family it is booting, by REEF_TRAINING_LOSS (the family, named directly) or REEF_TRAINING_RECIPE (a recipe kind, whose declared loss_family answers). Both bundled training configs set the second, from ${reef.recipe}, so the recipe is named once. When both are set, the first wins and the second is informational. Six families ship:

The torch hooks each family needs (--custom-loss-function-path, --custom-pg-loss-function-path, --custom-advantage-function-path) are resolved on the training worker from the family name, and there are no path flags to set. The driver validates the whole combination before any GPU work starts, so a disagreement shows up as a boot failure rather than a wrong gradient. The recipe class is the third party to that agreement, and the rule is three places name the objective. The recipe catalog says which family each bundled recipe declares.

A family reef does not bundle calls register_loss_family on its SlimeAlgorithm from a module imported at boot. It can also be written as a dotted package.module:SPEC reference anywhere a family is expected. Resolving the reference imports and registers it, so the plain name works from then on. REEF_TRAINING_LOSS=my_pkg.losses:ALGORITHM is the whole deployment change.

Two model classes still need hand-written flags. MoE models may want extra router flags in slime_flags, and hybrid or linear-attention models need a layer spec, --spec "slime_plugins.models.<module>" "<function>", from the plugin tree the installed slime distribution ships. slime is a pinned upstream dependency rather than a vendored tree: the pin lives in pyproject.toml's [dependency-groups] runtime, and a GPU image installs it with --no-deps so its CUDA-matched stack is left alone. Adding a model that has no spec there is a contributor task, described in development.

Batch sizes and staleness#

reef.batch_size is the recipe's knob for how much accepted data one reef update consumes, and --global-batch-size is how much data one slime step consumes. For sao and openclawrl each sample is its own rollout, so the two must be equal. tttd has no batch_size knob at all and sizes its step as groups_per_step × rollouts_per_group. Slime splits the payload it is handed into len(rollout_ids) // --global-batch-size steps and rejects a remainder, so a reef batch that is an exact multiple would give one reef version several slime steps. Every bundled recipe pins that ratio to one.

By default training is exact-version: every sample in a batch must come from the weights currently serving. reef.max_staleness (a shared knob, fallback REEF_MAX_STALENESS, default 0) opens a bounded window instead. With a positive value reef fences the job against the serving version it saw at preparation, then admits the batch only when every sample's producing <incarnation>:<sequence> version is from the same incarnation with lag 0..max_staleness. A missing, malformed, future, cross-incarnation, or over-window version drops the whole reserved batch before packing. Admission compares versions. It does not read the recipe, reweight anything, or change the payload. Commits record staleness/samples_fresh and staleness/samples_admitted_stale; drops record staleness/samples_dropped with the reason, the source record ids, and the observed versions.

Capturing exact policy tensors#

A training update needs the engine's own token ids, loss mask, and rollout log-probabilities, which arrive as the response.training block described in connect your harness. reef never rebuilds those from decoded text, so a harness that cannot attach them can still exercise inference and data capture, but its samples never train. Against a colocated SGLang engine the deployment can produce that block itself, with no change to the client API:

reef:
  inference_backend_factory: >-
    reef.train.slime_runtime.reef_adapters.sglang_chat.SGLangChatTrainingInferenceBackend
  inference_backend_config:
    tool_call_parser: qwen25   # must match the engine's parser: --sglang-tool-call-parser on the slime driver (--tool-call-parser on a standalone SGLang server)

Clients keep calling /v1/chat/completions. The backend renders the chat template once, sends input_ids to SGLang's internal /generate with return_logprob: true, returns an OpenAI-shaped response, and stores the sampled ids, mask, and log-probs privately. Streaming still streams, buffered internally so the recorded sample stays atomic. The limits are one completion per request (n must be 1) and text-only message content.

The driver process#

Every training config runs the bundled driver as its slime-driver service, and it is the only supported entry point:

python -m reef.train.slime_runtime.reef_adapters.driver \
  --ready-file=/tmp/reef-slime-bridge.ready \
  --hf-checkpoint=<hf model dir> --ref-load=<megatron torch_dist ckpt> \
  --save=<megatron ckpt dir> --save-hf=<hf export dir>/{rollout_id} ...

It reads RAY_ADDRESS (required), joins that cluster in REEF_RAY_NAMESPACE (default reef), boots the slime training and rollout stack, and publishes it as the named actor REEF_RAY_ACTOR_NAME (default reef-train-bridge). The reef service uses those same three values to find it. Flags may also come from a SLIME_ARGS_FILE, parsed without a shell, with command-line flags appended after. Only after the bridge answers its own health check does the driver write the ready file, which is why ready: test -f ... is a truthful probe. driver healthcheck is the same check on demand: it verifies the marker and RPCs the actor, and exits non-zero otherwise.

Preflight rejects a configuration the bridge cannot honour before any placement group exists:

• a non-positive --num-rollout, the count of externally supplied reef steps, which the bundled configs set effectively unbounded
• --save-hf without a {rollout_id} template, or a missing --save
• --debug-train-only or --debug-rollout-only
• no rollout GPUs
• a missing --disable-compute-advantages-and-returns, unless the loss family keeps slime's advantage pass, since reef supplies the training signal from the recipe's step preparer

Colocating actor and rollout GPUs requires both --offload-train and --offload-rollout. Using --offload-rollout without --colocate is refused outright, because the serving engine has to stay resident while training runs.

The Megatron checkpoint at --ref-load must be in torch_dist format; plain HF weights need a one-time conversion with slime's tools/convert_hf_to_torch_dist.py.

What a commit does#

One reef training job is one transaction: prepare and pack the batch, run the step, save the Megatron checkpoint and export HF weights, publish the new weights into the serving engines, then record the version. The next inference for that scenario uses the result. A retry reuses the same job id, so a duplicate cannot double-train. The on-disk marker can say a job was RUNNING, which happens when a client gives up while the bridge keeps training. In that case the next boot refuses to guess and stops with ambiguous training job, and an operator resolves it. That failure mode is why training-sao.yaml raises reef.train_timeout_s to the bridge's own 14400s ceiling instead of leaving the 300s inference default in place. After the weight update the bridge asks every serving engine for its version and fails the step if they disagree.

checkpoint_every_n_versions decides which versions get durable bytes. A version that is not checkpointed is a live version. Its weights exist only in the engine's memory, and the artifact record points at an engine-side token. A restart cannot reproduce a live version (weight tokens are scoped to a serving incarnation), so serving falls back to the last checkpoint and the next training step republishes a live head. POST /reef/scenarios/{scenario}/rollback only accepts a version with durable bytes; a live one is refused as not restorable. Rollback never rewinds the chain. It activates the target and appends a new version recording what it restored. For weights, rollback additionally needs a runtime that can restore a checkpoint into the engine, which the bundled Ray/slime runtime does not implement, so today rollback is a harness-artifact operation.

Exported checkpoints live under a byte budget: training.checkpoint_retention sets max_storage_fraction (default 0.80 of the filesystem), min_free_space_fraction (0.10), and policy. training-local-slime.yaml declares the block and passes all three to the driver as --reef-checkpoint-* flags; training-sao.yaml omits it and takes the driver's built-in defaults, which are the same numbers. Byte-valued variants (--reef-checkpoint-max-storage=200GB) suit a shared filesystem. latest keeps the newest checkpoint pairs and best_reward keeps the highest-scoring ones. The current head, the tracker's rollouts, and any checkpoint the run booted from are protected under either policy. If the checkpoint root contains assets the bridge did not write, retention refuses to delete anything and blocks startup rather than touching a file it does not own. Move them out and the run proceeds.

Storage#

Production must persist all of these. The defaults are relative to the working directory. The two training configs move all of them under /var/lib/reef, and the inference configs move only some, so check every row before taking one to production:

The checkpoint directory must be visible to both the driver and the reef service until publication completes. The driver writes the export, and reef commits those bytes into the artifact repository.

Day two#

GET /healthz is the only unauthenticated route and answers as soon as the HTTP service is up. GET /reef/status says whether learning is happening: the training thread's last error, per-scenario scenario_step and current_weight_version, the checkpoint storage plan, and any scenario that failed to reload at boot. GET /reef/scenarios/{scenario}/versions lists the chain behind that number.

Before taking traffic:

• Set reef.token. An empty token disables authentication entirely, and the bundled configs read it from $REEF_TOKEN, so an unset variable is an open service.
• Terminate TLS and do tenant authorization in front of reef, and keep provider credentials in the environment rather than the config file.
• Route each scenario to one logical reef writer. Distributed ownership and failover do not exist.
• Monitor the driver and the serving engine separately from reef. Their failures surface as reef errors only when a step is attempted.

Design explains why the transaction is shaped this way and what the version chain guarantees.