Operate a deployment

What to check while a deployment runs, how to read and steer its release chain, how to keep an eye on training, and what survives a restart.

For

whoever runs a Reef deployment

Before you start

a running deployment from Evolve your model or Evolve your harness, its URL and token

You will have

the routine checks, the version operations, and the recovery rules

The examples use REEF_URL and REEF_TOKEN as in HTTP API.

Check health and status #

curl -f "$REEF_URL/healthz"
curl -sS -H "Authorization: Bearer $REEF_TOKEN" "$REEF_URL/reef/status"

/healthz answers as soon as the HTTP service is up; it says nothing about training. /reef/status is the training side: the last asynchronous error, model preload failures, and for every scenario its step counter, latest committed step outcome, runtime load ID being served, checkpoint storage state, whether a batch is waiting, the processor's state, and whether inference is admitted or paused for a weight update. The committed outcome includes the recipe-owned metrics, so a skipped or rejected update is distinguishable from one that is still running. It is the first place to look when requests keep being served by an old version.

The service and every process reef serve started write logs under run_dir (/tmp/reef-stack/ by default), one <service>.log and one <service>.pid each.

Read the release chain #

curl -sS -H "Authorization: Bearer $REEF_TOKEN" \
  "$REEF_URL/reef/scenarios/code-repair/releases"

Newest first. Each row names the release, its parent and content, whether it is a durable checkpoint (checkpoint, restorable), what produced it (operation: creation, training, rollback, recovery), whether it is the one currently served, and for training rows the step's metrics. Content can be live (content_kind: live_weights: the engine has the weights, the repository has only the record) or saved (content_kind: saved_artifact, a Git LFS commit).

For harness scenarios, GET /reef/harness/releases lists the same chain oldest first with each step's gate metrics, and GET /reef/harness?release_id=<id> returns any listed tree.

Pin a version #

A client that must keep answering from one version sends x-reef-release-id: <id> with its requests. Pinning is per request and changes nothing on the server; a pin that conflicts with the scenario's binding is refused with 409.

Roll back #

curl -sS -X POST -H "Authorization: Bearer $REEF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"release_id": "<id>"}' \
  "$REEF_URL/reef/scenarios/code-repair/rollback"

Rollback republishes the target as a new commit and makes it current; history is not rewritten, and the step counter keeps increasing. Only versions marked restorable qualify: durable checkpoints. Live runtime load IDs that were never checkpointed cannot be restored, and the bundled Ray/Slime runtime does not implement checkpoint restoration, so rollback currently applies to harness artifacts; for weights, redeploy from the checkpoint you want.

Set the checkpoint cadence #

checkpoint_every_n_versions (default 1) decides how many accepted updates go by between durable checkpoints. Between checkpoints, new weights live only in the engine's memory: their versions are recorded, but their bytes are not. A restart restores the last checkpoint, and the step counter, algorithm state, and record progress continue from the log. Raise the cadence only when checkpoint writes are the bottleneck and losing live versions on a restart is acceptable.

On a training deployment, checkpoint retention runs under --reef-checkpoint-policy (latest or best_reward) with storage-fraction limits; when storage is blocked the step is deferred rather than failed, and /reef/status shows checkpoint_storage.

Track experiments with W&B #

Tracking is optional and off by default; observability.wandb in Configuration lists every key. Export WANDB_API_KEY before starting the stack; there is no key field, and the Slime driver refuses --wandb-key so a credential never enters a command line or a run config.

What you see in W&B: one group per scenario and one run per scenario, plus a new run after every rollback, so each post-rollback branch is its own curve. Every training result lands on the run-local train/step axis carrying the monotonic reef/step that joins it to the commit log, and the commit metrics record experiment/run_id, so a Reef version leads to its run and the run's reef/training_job_id leads back. Tracking failures are logged and never fail a training step or its commit.

Restart and recovery #

What survives a restart, provided the storage paths are persistent:

The record store, commit logs, and repository live under .reef/ by default (agent_record_dir, artifact_repository, artifact_work_dir, artifact_cache_dir). On ephemeral storage none of the guarantees above hold past its loss. Each scenario needs one Reef writer; run a second deployment on other ports and storage paths rather than two services on one store. Architecture describes the commit ordering behind the table.