Get started

Start on a CPU by running reef in front of an HTTP model provider. You will send one request, attach feedback to its receipt, and confirm that reef recorded both. The optional second half uses two GPUs to train a new version.

Install#

Install from a checkout. The clone is also where the bundled deployment configs and the runnable examples live, and the reef name on PyPI belongs to an unrelated project.

git clone https://github.com/Human-Agent-Society/reef.git
cd reef
pip install -e .

Python 3.10 or newer is required. The install above is enough for the CPU walkthrough. The GPU walkthrough later on uses the supported Docker image and the optional training dependencies.

Write a service config#

A service config tells reef where to listen, which recipe to use, and which processes to start. This minimal config runs reef as a recording proxy in front of an HTTP provider.

# ~/reef-demo/reef.yaml
reef:
  host: 127.0.0.1
  port: 8900
  recipe: recipe
  token: reef-local

services:
  - name: reef
    command: python3 -m reef.service
    ready: curl -sf http://127.0.0.1:8900/healthz

Use an absolute path with -c for configs outside the repository.

Start it#

mkdir -p ~/reef-demo && cd ~/reef-demo
export REEF_TOKEN=reef-local
export REEF_UPSTREAM_URL=https://api.openai.com
export REEF_API_KEY=sk-...               # omit for an unauthenticated upstream
reef serve -c ~/reef-demo/reef.yaml

Set REEF_UPSTREAM_URL to the provider host without a trailing /v1. Wait for reef: ready, then call the health route:

curl -f http://127.0.0.1:8900/healthz
# {"ok": true}

/healthz is the one route that skips authentication, so a liveness probe never needs the token. Every other route wants Authorization: Bearer $REEF_TOKEN and answers 401 invalid service token without it.

Send a request through it#

Two headers turn an ordinary OpenAI request into a recorded reef request: x-reef-scenario names the workload, and x-reef-recipe chooses how that workload learns. Send both when creating a scenario. Later requests only need the scenario header.

curl -sS -D headers.txt -o response.json \
  http://127.0.0.1:8900/v1/chat/completions \
  -H "Authorization: Bearer $REEF_TOKEN" \
  -H "Content-Type: application/json" \
  -H "x-reef-scenario: hello-reef" \
  -H "x-reef-recipe: recipe" \
  -d '{"model": "gpt-4o",
       "messages": [{"role": "user", "content": "Return exactly: reef is ready"}]}'

response.json is the provider's own body, forwarded unchanged in both directions. reef leaves model, the messages, and the completion exactly as they came. The one thing it adds is a response header:

grep -i x-reef-agent-record-id headers.txt
# x-reef-agent-record-id: ee5aa401634b4567bf9dae21816abde4

That is the receipt. It names the stored record of this exact exchange, including which artifact version was serving when the answer was produced.

Report on it#

Feedback is a separate POST that quotes receipts. It can arrive seconds or hours later, and from a different process than the one that made the request. The receipt is the only thing tying the two together.

export RECEIPT=$(awk 'tolower($1)=="x-reef-agent-record-id:"{gsub("\r","",$2);print $2}' headers.txt)

curl -sS http://127.0.0.1:8900/reef/report \
  -H "Authorization: Bearer $REEF_TOKEN" \
  -H "Content-Type: application/json" \
  -H "x-reef-scenario: hello-reef" \
  -d "{\"score\": 1.0, \"feedback\": \"matched\", \"references\": [\"$RECEIPT\"]}"
# {"agent_record_id": "cce17dd7...", "scenario": "hello-reef", "request_type": "report"}

The report gets its own record id. There is no need to repeat x-reef-recipe, because the first request already made the scenario's binding permanent.

Here is the same loop in Python, using the client the examples use:

import os

from reef_client import ReefClient

client = ReefClient("http://127.0.0.1:8900", token=os.environ["REEF_TOKEN"])
response, receipt = client.inference_with_record(
    "hello-reef",
    "/v1/chat/completions",
    {"model": "gpt-4o", "messages": [{"role": "user", "content": "..."}]},
    recipe="recipe",
)
client.report("hello-reef", {"score": 1.0, "feedback": "matched"}, references=[receipt])

Read the version chain#

curl -sS -H "Authorization: Bearer $REEF_TOKEN" \
  http://127.0.0.1:8900/reef/scenarios/hello-reef/versions

The response contains one initial version. It will not advance yet because the recipe kind records traffic but does not train. At this point you have confirmed the complete request-and-feedback integration.

Make a version advance#

To see the version advance, switch to sao, the smallest bundled training recipe. Each scored response triggers one training step. The runnable example uses three problems with automatically verified answers.

This example needs two GPUs, a Hugging Face model directory, and a compatible Megatron checkpoint. Set their paths in examples/sao/serve.yaml and run the commands inside the supported image.

cd examples/sao
pip install -e . "tide-eval[harbor]"     # this example's harness, plus the runner
export REEF_INFERENCE_HOST=$(hostname -I | awk '{print $1}')
reef serve -c examples/sao/serve.yaml

Startup can take several minutes. Wait until all three services report ready before running the example.

With the stack up, run python3 run.py in a second shell to drive the loop. It works through the three IMO problems in order, makes SAO_ROLLOUTS scored attempts at each one, and reports every score against its receipt. The order matters, because task N+1 is served by what task N taught. Watch the chain grow:

curl -sS -H "authorization: Bearer reef-local" \
  http://127.0.0.1:8900/reef/scenarios/sao-smoke/versions

Each successful training step adds a version, and the newest version becomes current. See the version chain for checkpoints, rollback, and the fields returned by this endpoint.

When it does not start#

Next#

Connect your harness is the wire contract in full: every header, streaming and the Anthropic-shaped route, what a report may carry, and how a harness reads an updated artifact back.