State Model: Records, Commits, and Releases

Records #

Reef stores every exchange of inference and every report as AgentRecord. It includes a record id, a scenario, an inference payload (for inference exchange) or feedback (for report), and necessary metadata (e.g. request type or artifact identifier used for serving).

RecordStore defines record append, replay, compaction, and audit operations. It does not know about trainers, artifact publication, or committed scenario steps. SQLiteRecordStore and PostgresRecordStore share SQL record operations while supplying their own database connections, schemas, and transactions.

Compaction retires records from training while retaining their bodies for audit. It retires only rows the processor marks releasable, and Reef recomputes that set from current state on every read. Separate retention maintenance physically purges old compacted bodies while keeping retry hashes and commit metadata.

With a database path configured, the SQLite store uses WAL journalling and synchronous = FULL. The default in-memory database is for tests and does not survive a restart.

The release chain #

Every accepted update creates a release with a parent. Three identities stay separate: release_id names Reef's publication decision, content_id names the selected model or harness content, and runtime_load_id names a concrete serving-engine weight load. A release may refer to durable bytes or to live weights held only by the current process; its identity is stored durably either way.

The training backend decides when live weights become durable: each version whose checkpoint the backend saves is published as an artifact. A Slime deployment saves each save-interval-th version, so any number of live steps may occur between checkpoints. A live release's runtime_load_id is an opaque <incarnation>:<sequence> token, where the incarnation keeps tokens unique across training-group restarts. The release record is durable; the bytes are not, so a restart restores the last checkpoint. The step counter, algorithm state, and record progress do survive.

A release binds one or more named components. Every shipped recipe declares one, so its releases are flat: the artifact is the component and its content_id is the component's. A recipe that declares several components (weights and harness, say) serves releases that keep one directory per component and carry a component manifest in their metadata. Each step publishes one component and the committer carries the others forward from the previous checkpoint, so the release still binds the whole combination: its content_id derives from the component content ids, a rollback restores every component together, and only a component whose content the engine does not already serve is loaded or activated. Such a scenario must checkpoint every step, because a live weight release names an engine load and nothing else.

Durable releases are Git-backed, one ref per scenario, with LFS patterns for weight files and a reef-artifact.json manifest in every release. Heads move only by compare-and-swap: advance_current requires the expected head, publish requires the expected parent, and the push carries a lease, so a stale publication conflicts instead of overwriting. Rollback does not rewrite history. It activates an earlier release's content_id and publishes it under a new release_id, keeping step numbers monotonic.

Commit ordering #

ScenarioStore owns a scenario's committed state, its RecordStore and its task stores. It validates step progression and settles record consumption with each commit. Deployment assembly supplies ScenarioStorage; ScenarioFactory handles registration and release selection, then recover_scenario opens a session and returns the complete Scenario. The scenario owns that session and coordinates trainer and artifact operations. The default SQLiteScenarioStorage assembles SQLiteRecordStore with CommitHistoryScenarioStore and keeps commits in append-only JSONL; its fsynced append is the commit point. PostgresScenarioStorage keeps commits in a table; the committed row is the commit point. A committed step records its step number, artifact ref, checkpoint flag, algorithm state, record high-water mark, consumed record IDs, compaction retirements, metrics, and training job identity. Existing databases and logs require no conversion.

commit_step(expected_step=..., commit=...) validates the expected scenario step before accepting its successor. Identical retries return the original commit; different content for that step or a stale expected step conflicts. The default adapter appends the commit before applying SQLite compaction. These are separate writes: once the append is durable, a compaction failure does not undo the commit. Retrying the same commit or running startup recovery repairs the remaining compaction. A database adapter may settle both in one database transaction while preserving this public contract.

ScenarioCommitter keeps artifact operations outside the store. For a durable store, the order is:

Every trainer is bound to a release component (records when the recipe serves none). With several trainers, every commit record names the component that made it. A scenario of one trainer names none, so its records, checkpoint metadata and drop receipts keep the format they had before components.

A scenario runs one training step at a time, whatever the number of its trainers:

• A local cycle holds the scenario's cycle lock from preparation to commit.

• The training thread holds the same lock for a dispatched turn, from the reservation of the batch to the acknowledgement of the job. While a local cycle holds the lock, the training thread skips the scenario. The end of the cycle wakes the thread again.

• A local cycle prepares nothing while a dispatched batch is reserved, for example while checkpoint storage blocks the job.

• A rollback or a promote refuses to run while any batch is reserved.

Thus the head does not move under a step, and every step commits on the release that it was prepared against.

When a commit fails, Reef rebuilds the scenario from durable state at once. A failed local cycle rebuilds it itself, and the training thread rebuilds it after a failed turn. A local cycle that still runs on the old instance keeps that instance open until it ends. Then it commits nothing and looks again on the rebuilt instance, which has not read its rows. A local cycle whose scenario is deleted under it also ends without a commit.

A commit retires no row, so the trainers share every stored row and each keeps its own consumption: a row one trainer consumed is still there for the others. A batch that a trainer drops as stale goes on a consumption receipt that names its component. On restart each trainer recovers its state and read cursor from its own commits and skips the rows its own commits and receipts name, which needs durable commit storage.

Without a durable store, live and local saved releases advance the serving head before settling the in-memory commit. A conflicting head therefore rejects the step before its records are compacted. In-memory commit history supports retries only for the lifetime of that session.

Artifact metadata stores scenario registration and checkpoint commit data under scenario_commit_record. Recovery decodes it into a CommitRecord and reconciles it with the store before restoring the trainer. Initial registration has no commit and passes None. A committed checkpoint ahead of the backend pointer repairs that pointer. An older checkpoint ahead of the local log can be adopted into the log. Recovery repairs compaction, replays retained records through the committed watermark while excluding every committed step's consumed IDs, and restores the read cursor. The watermark is a read position: retained records behind it can still belong to an incomplete batch.

Runtime work happens outside the transaction, so the training step must succeed before its batch is acknowledged. After an ambiguous commit error, history is authoritative: callers must reconcile or retry the same commit before publishing another version. Each scenario still needs exactly one logical Reef writer; expected-step validation does not provide distributed leases or make external training operations atomic. Those operations must be idempotent or reconcilable after a crash. See the Python storage contract for custom adapters.

These guarantees require persistent storage. Configuration lists the paths, and Operate a deployment tabulates what each one survives.