Updating the fleet with `fleet-update`¶

scripts/fleet-update is the human-run way to move the hub and the workers to a new commit. You run it on the hub, as the operator account. It updates one host at a time, checks each host before it moves on, and stops at the first failure.

It replaces hub self-upgrade, release epochs and source convergence, which have been deleted: in the 90 days before it was written, release epochs aborted 62% of the time and hub self-upgrade never succeeded. The older deploy script (deploy/deploy-mac-fleet.sh) and its node installer have been deleted too.

The release manifest names every component of one exact release, so a host's state can be compared with more than its Git HEAD.

Usage¶

scripts/fleet-update [--dry-run] [--hermes] [--yes] [--no-drain] \
    [--allow-unpromoted] [--runtime-image REF --runtime-input-sha256 SHA] \
    <hub|HOST|all> [sha]

Argument	Meaning
`hub`	Update the hub's control plane, then restart the hub's own worker.
`HOST`	Update one managed host, named as in the fleet inventory (`~/.mac/fleets.yaml`). The hub's own name updates the hub.
`all`	The hub first, then each worker in turn. A worker still busy after the drain timeout is skipped, and the run ends with exit code 3 and a rerun command for each skipped worker. Any other failure stops the run.
`[sha]`	A promoted commit on `main`. A short sha is fine. When omitted, the newest promoted commit on `main` is selected, rather than blindly using `origin/main`.
`--dry-run`	Print the plan, including any migrations, and change nothing. The only write is `git fetch`, which updates remote-tracking refs.
`--hermes`	Also restart the Hermes gateway (`ai.hermes.gateway` on the hub, `hermes-gateway` user unit on Linux).
`--yes`	Don't ask for confirmation before each host.
`--no-drain`	Restart the hub without first draining the workers (see step 0 under Hub). Tasks in flight lose the hub and its model router for the length of the restart, and usually fail. Use it only when the hub must go down now.
`--allow-unpromoted`	Emergency-only override for an explicit SHA. It bypasses promotion checks and prints a prominent warning in both the log and final summary. It cannot be combined with automatic target selection.
`--runtime-image REF --runtime-input-sha256 SHA`	Repin each worker's sandbox image to a published `ghcr.io/jordanhubbard/mac-openshell-runtime@sha256:...` digest and its CI frozen-input identity, both from the image's CI publication. Give both or neither. This changes only the image, never the OpenShell version.

Every line of output also goes to ~/.mac/logs/fleet-update-<timestamp>.log.

Hosts¶

Hosts come from the fleet inventory, ~/.mac/fleets.yaml (override the path with FLEET_UPDATE_INVENTORY), read through mac.fleet_inventory. There is no other host list and no built-in default. mac admin fleet inventory prints every entry with its disposition:

Disposition	Meaning
`managed`	This fleet's updater owns it. `all` updates every managed worker, in file order.
`external`	Another named mechanism owns it (`managed_by`; `instance_kind: fungible` hosts belong to the fungible provisioner). Never updated here.
`retired`	`enabled: false` or `lifecycle: retired`. Never updated.
`other_fleet`	Enrolled in a different fleet in the same file.

Each entry's identity is name, target (ssh), agent_id, role (hub or worker), os, arch and lifecycle. Any of agent_id (default agent_<name>), role (the fleet's hub_agent is the hub) and lifecycle may be omitted, and the plan marks what it derived. Write them explicitly when the default is wrong:

agents:
- {name: worker-1, target: <user>@<mesh-ip-1>, os: linux, arch: aarch64}
- {name: lab-box, target: <user>@<ip>, os: linux, lifecycle: external, managed_by: lab-team}

The script stops before touching any host when the inventory is unreadable or inconsistent (two entries claiming one name, ssh host or agent id), when the named host is not managed, or when the machine answering at a worker's ssh target reports a different MAC_AGENT_ID than the inventory's agent_id. A replaced host therefore needs an explicit inventory update. The inventory is read with the script's own source tree (FLEET_UPDATE_PYTHON, default the venv's python, supplies pyyaml).

The hub's fleet view (/fleet, the Fleet section of every agent's runtime context) classifies each agent row from the same inventory: worker, hub, virtual_service (the operator persona and reviewer), operator_session (interactive CLI sessions) or unenrolled. Only worker rows count as execution capacity.

Other knobs, mostly useful for testing: FLEET_UPDATE_SRC (the checkout, by default ~/.mac/src/mac), FLEET_UPDATE_HUB_URL (default http://127.0.0.1:8789), FLEET_UPDATE_MAC_CLI (default mac), FLEET_UPDATE_POLL_SECONDS, FLEET_UPDATE_DRAIN_TIMEOUT (default 600) and FLEET_UPDATE_HEALTH_TIMEOUT (default 180).

Preflight (every target)¶

git fetch origin in ~/.mac/src/mac on the hub.
Resolve [sha] to a full commit. A promoted commit must be on origin/main, have a successful completed ci.yml push run, and have that run's complete mac-release-manifest and passed OpenShell runtime publication receipt. The runtime digest in those artifacts must carry the receipt's tested-<frozen-inputs-sha256> GHCR tag. The script refuses before any hold or drain and names the missing condition. This check also runs during dry runs. With no SHA, completed main runs are searched newest-first and the first release satisfying all these checks is selected.
Read the commit the host currently runs. On the hub that is the checkout's HEAD; on a worker it is read over ssh.
Compare src/mac/data/postgres/migrations/*.sql between the two commits:
migrations that the target adds are listed;
if the deployed commit has a migration the target lacks, the script refuses. It never goes backwards across a migration.
The checkout must be clean (git status --porcelain empty).

Hub (macOS)¶

Drain. The control plane also serves the model router, so while the hub is down every in-flight task loses its lease renewals, its heartbeats and its model calls. The backup in step 2 alone takes minutes. For each worker in the inventory, the script records any existing dispatch hold and leaves it alone. Otherwise it runs mac agent hold <agent_id> --reason "fleet-update hub <sha12>". It then waits up to the drain timeout (FLEET_UPDATE_DRAIN_TIMEOUT, default 600s) for every worker to be idle. If one is still busy, the script releases its own holds, names the busy workers and their tasks, and stops. Nothing has changed at that point. Rerun later, or pass --no-drain. The holds are released once the new hub is healthy (after step 8), or once a rollback has the old hub serving again. A hub left stopped or unhealthy keeps its workers held, and the script prints the mac agent resume command for each one.
sudo -n launchctl bootout system/com.mac.control-plane, then wait for the mac.hub_serve process to exit.
Run ~/.mac/venv/bin/mac-pg-backup --json --out ~/.mac/backups/fleet-update-<ts> and require restore_verified == true. MAC_DATABASE_URL is read from ~/.mac/mac.env and passed through the environment, never on a command line.
Run git checkout --detach <sha>.
Do the locked install in place (python -m mac.native_runtime, the same command as step 4 of "Provision a new host") if pyproject.toml or uv.lock changed. Also do it if python -m mac.native_runtime --check reports that the venv's locked baseline no longer holds. Worker pip installs check that baseline and fail closed without it, and a plain uv pip install -e leaves it stale. A failed repair of a stale baseline is only a warning.
Run ~/.mac/venv/bin/python -c 'import mac.services'.
Run mac-schema-migrate --status, then mac-schema-migrate --applied-by fleet-update:<user>@<host>:<sha12>.
Run sudo -n launchctl bootstrap system /Library/LaunchDaemons/com.mac.control-plane.plist.
Poll /health, then require /startup-attestation's source_commit to equal the target. An empty source_commit is a failure, not a pass. mac-service sets it from the HEAD of ~/.mac/src/mac.
Run python -m mac.node_files --source ~/.mac/src/mac --role hub (see Managed files). The hub role owns only mac-service; the transition removes previously managed task-worker wrappers. A failure is logged, never fatal.
Run python -m mac.fleet_context_service --source ~/.mac/src/mac. It installs or repairs the LaunchAgent com.mac.fleet-context, which refreshes the live fleet block in this agent's runtime context every 3 minutes, and runs one refresh. Its JSON result is logged; a failure never fails the update. Then run python -m mac.slack_relay install, which installs the kept-alive LaunchAgent com.mac.slack-relay and restarts it on the new code. The hub runs no task agent, so this is what posts agent reports (mac agent report) and questions to the Slack home channels and relays thread replies back as answers. Its log is ~/.mac/logs/slack-relay.log.
Run launchctl bootout gui/<uid>/com.mac.agent; an already absent service is a successful no-op. Rename ~/Library/LaunchAgents/com.mac.agent.plist to .plist.disabled so login does not start it again, and remove MAC_WORKER_TOKEN from ~/.mac/mac.env (backup mac.env.bak-hub-agent-<ts>): it belongs to the deleted agent, and mac admin fleet refresh-context prefers it over MAC_API_TOKEN. Decommission the inventory's hub agent record with mac agent delete <hub-agent-id> --actor fleet-update, which is idempotent and prevents dispatch even without a hold. With --hermes, kickstart gui/<uid>/ai.hermes.gateway.

Hermes, the fleet-context LaunchAgent, heartbeat capability reporting, and the hub review verifier are control-plane or dedicated LaunchAgent services; none uses com.mac.agent. The historical hub agent row is not needed by those services and should be decommissioned with mac agent delete <hub-agent-id>; operational docs must not use mac agent show <hub-agent-id> as a health check.

~/.mac/bin/mac-service is a managed file of the hub role, installed from deploy/bin/mac-service: it runs ~/.mac/venv/bin/python -m mac.hub_serve from ~/.mac/src/mac and nothing else. The script no longer writes the legacy ~/.mac/current/source-commit and generation-id markers; nothing reads them.

Hub rollback¶

Failure before the schema can have changed. This covers a backup that fails or isn't verified, a failed checkout, install or import, and any failure when --status reported no pending migrations. The script checks out the old commit, reinstalls it if the dependency files differ, bootstraps the old hub and exits non-zero.
Failure once migrations have run. This covers a migration that fails part-way (the hub is left stopped) and a hub that won't come up healthy after migrating. The script stops and prints what to do. It never downgrades the schema. Fix forward with a new sha if you can. A full restore discards every write made after the backup, so do it by hand:
Stop the hub.
pg_restore the dump into the database. The dump is listed in the backup's manifest.
Run git -C ~/.mac/src/mac checkout --detach <old>.
Bootstrap the LaunchDaemon.

If the script exits while the hub is stopped, its last line says so and gives the launchctl bootstrap command.

Linux worker¶

Prestage. Before any hold, over ssh: the checkout must be clean, git fetch must succeed, and the target commit must then exist locally. If not, the script stops and nothing has changed on the host, so a network failure after the drain can't strand an unprepared install.
Hold. Read the agent with mac agent show <agent_id>. If it already has a dispatch hold, note it and leave it alone; this run will neither add nor remove a hold, such as an operator's fleet-wide dispatch pause. If there is no hold, mac agent hold <agent_id> --reason "fleet-update <sha12>".
Drain. Wait up to 10 minutes for current_task_id to clear. If it doesn't, release the script's own hold (if it set one) and stop. Nothing has changed on the host at that point.
Update over ssh. A single ssh <target> bash -s runs these steps, under an exclusive host lock (~/.mac/.fleet-update.lock; a second concurrent run exits 7 and changes nothing):
git fetch and git checkout --detach <sha> in ~/.mac/src/mac;
the locked install when the dependency files changed or the baseline check fails, as on the hub;
python -c 'import mac'. If the import fails, the old checkout and its dependencies are restored before anything is restarted.
python -m mac.node_files --source ~/.mac/src/mac --role worker (see Managed files). A failure restores the old checkout.
python -m mac.fleet_context_service --source ~/.mac/src/mac, as on the hub but with the mac-fleet-context systemd timer: the units are rendered from deploy/systemd into /etc/systemd/system, a failed timer is reset, and one refresh runs. A failure is logged, never fatal.
python -m mac.openshell_image_pin. A worker pins its sandbox image in one place, ~/.mac/openshell/runtime-image-ref. This step rewrites MAC_OPENSHELL_CREATE_ARGS --from in ~/.mac/mac.env to match that pin and removes MAC_HUB_VERIFY_IMAGE, because the test gate now reads the pin too. mac.env is backed up before it changes. A host with no managed pin is left alone. With --runtime-image, the step first pulls the image, checks its build-revision and frozen-input labels, and writes the pin files. If that fails, the old checkout is restored and nothing is restarted. A plain sync failure is logged, never fatal.
sudo -n systemctl restart mac-agent. With --hermes, also systemctl --user restart hermes-gateway.
after 10 seconds, systemctl is-active mac-agent.
Health. Within 3 minutes, the hub must report that the restarted process runs the new commit: resources.source_state.loaded_commit and commit_sha both equal <sha>, restart_pending == false and dirty == false. A checkout that moved without the process restarting fails here.
Resume. The script resumes dispatch only if it placed the hold itself, and only if the hold reason is still fleet-update <sha12>. If someone replaced the hold meanwhile, the script leaves it alone.
Failure. The rollout stops, the host stays held, and the script prints an inspect command, a rollback command and the resume command.

Managed files¶

mac.node_files declares the files each role owns and is the only code that installs them. Workers get deploy/bin/{mac-agent-service, mac-agent-startup-self-test, mac-task-executor} (0700), deploy/bin/mac-task-executor.py (0600) and deploy/mac-crash-observer.py as bin/mac-crash-observer (0755). The hub gets the same plus deploy/bin/mac-service (0755).

Each file is renamed into place, so a running wrapper keeps its old inode.
What was installed is recorded, with digests, in ~/.mac/managed-files.json. A recorded file that a later release no longer declares is removed, after a copy into ~/.mac/backups/node-files-<ts>/. One edited by hand since is kept and reported as kept_modified.
Files it never recorded are never touched.
One installer per host (~/.mac/.node-files.lock). A second run is a no-op.
--plan reports what would change without changing anything.

Release identity¶

A worker reports, in resources.source_state, the commit its process loaded at boot (loaded_commit, loaded_tree, boot_id, booted_at) as well as the checkout on disk (commit_sha, dirty), and restart_pending when the two differ. It also reports when its runtime-context Fleet block was last refreshed (resources.fleet_context.refreshed_at).

The operator selects the release the fleet must run:

mac admin fleet select-release <40-hex sha> --reason "..." [--converge-minutes 60]
mac admin fleet release          # the selection and each worker's drift
mac admin fleet clear-release

While a release is selected, every agent that reports a MAC source state must be running exactly that commit, from a clean checkout, with no restart pending. Otherwise it is excluded from dispatch with release_mismatch, release_restart_pending, release_dirty or release_unknown (missing evidence is never a match). Interactive sessions and the hub's in-process reviewer and operator report no source state and aren't gated. An offline worker stays excluded until it comes back on the selected commit.

mac admin fleet release (GET /fleet/release) lists, per worker, liveness, the loaded and on-disk commits, the verdict, the dispatch hold and the fleet-context age as separate fields, with a repair command for each drifted worker. The dispatcher tick records a fleet.release.drift event whenever the drifted set or the deadline state changes, OVERDUE once --converge-minutes has passed. It only observes: updates still go through fleet-update.

Nothing is selected by default, and then there is no release gate. Select the release after rolling it out, or before to hold back work from hosts that haven't converged.

Worker rollback¶

ssh <target> 'git -C ~/.mac/src/mac checkout --detach <old-sha> && sudo -n systemctl restart mac-agent'
mac agent resume <agent_id>      # only if fleet-update placed the hold

GKE pods (supervisord mac-agent) are not handled by this script.

Worker tokens¶

mac admin worker-token issues long-lived worker bearer tokens using the existing worker_credentials table. It runs on the hub, against MAC_DATABASE_URL (or mac --db <dsn>).

mac admin worker-token issue  <agent_id> [--days 365] [--out FILE] [--install HOST]
mac admin worker-token rotate <agent_id> [--days 365] [--out FILE] [--install HOST]
mac admin worker-token list   [<agent_id>]

issue and rotate are the same operation. Each mints a new credential that expires after --days days (default 365) and makes it the agent's only active credential; earlier ones become superseded.
The token is shown exactly once:
with no options, the raw token goes alone on stdout and a summary goes to stderr;
--out FILE writes the token to a mode-0600 file instead;
with --install, the token is not printed at all.
Without --install, the old token stops working immediately. Install the new one straight away.
--install HOST keeps the old token working until the new one is in place. While the new credential is still pending (the old one keeps authenticating), the command runs ssh HOST bash -s, with the token carried on stdin, never in argv. On the host, that session:
rewrites every MAC_WORKER_TOKEN / MAC_WORKER_TOKEN__<FLEET> key that ~/.mac/mac.env already has (or adds MAC_WORKER_TOKEN if it has none), and moves any hub-facing alias that held the old token;
runs ~/.mac/venv/bin/python -m mac.hermes_chat_config --hermes-home ~/.hermes --mac-env ~/.mac/mac.env, because Hermes embeds the token;
restarts mac-agent and hermes-gateway.

Only after that succeeds is the new credential activated. If the install fails before mac.env is rewritten, the new credential is revoked and the old one stays active. If mac.env was rewritten but the Hermes resync or a restart failed, the new credential is activated anyway (the host already holds it) and the command tells you to restart the services by hand. - list prints credential metadata. It never prints the token or its hash.

Provision a new host¶

fleet-update only moves an already-provisioned host to a new commit. The deleted deploy scripts did the rest, so do these by hand. $HERMES_HOME is the host's Hermes home (default ~/.hermes).

Install Tailscale and join the tailnet. Confirm that ssh <host> works from the hub without a password.
Install git 2.38 or newer (merge-tree --write-tree; Ubuntu 22.04 ships 2.34), gh with HTTPS credentials (GH_TOKEN=... gh auth setup-git --hostname github.com), and the review key at ~/.ssh/mac_github_review_id (mode 0600, plus a Host github.com IdentityFile/IdentitiesOnly yes entry in ~/.ssh/config). The hub needs a working GitHub SSH identity; workers only use it for reviews.
Grant passwordless sudo for what fleet-update runs: on a Linux worker, <user> ALL=(root) NOPASSWD: /usr/bin/systemctl restart mac-agent; on the macOS hub, <user> ALL=(root) NOPASSWD: /bin/launchctl. Check with sudo -n true.
git clone <origin> ~/.mac/src/mac, then do the locked install and put mac on PATH:

PYTHONPATH=~/.mac/src/mac/src python3 -m mac.native_runtime \
  --source ~/.mac/src/mac --venv ~/.mac/venv \
  --snapshot ~/.mac/logs/native-runtime-packages.json \
  --footprint ~/.mac/agent-footprint.json --uv "$(command -v uv)" \
  --record ~/.mac/logs/native-runtime-locked.json
ln -sf ~/.mac/venv/bin/mac ~/.local/bin/mac

The operator side of fleet-update calls plain mac (FLEET_UPDATE_MAC_CLI). 5. Create ~/.mac/mac.env (mode 0600). deploy/systemd/mac.env.example is written for a hub and lacks most worker keys, so set these by hand on a worker: MAC_CONTROL_PLANE_ROLE=client, MAC_HUB_URL and MAC_URL, MAC_FLEET_NAME, MAC_AGENT_ID, MAC_WORKER_TOKEN (written by step 7), MAC_WORKER_MODE=loop, MAC_WORKER_AGENT_NAME, MAC_WORKER_HOSTNAME, MAC_WORKER_CAPABILITIES, HERMES_HOME, HERMES_REDACT_SECRETS=true, MAC_MEMORY_TOPOLOGY_FILE, GH_TOKEN, plus the OpenShell keys your hosts use. The mac-agent wrapper refuses to start without MAC_HUB_URL and MAC_WORKER_TOKEN. 6. Provider keys live only in the hub vault, never in worker env. On the hub, store each as <provider>-upstream: printf %s "$KEY" | mac admin secret set openai-upstream --from-stdin --scopes '{"capabilities":["router-upstream"]}' --created-by <you>. On a worker, remove {NVIDIA,OPENAI,ANTHROPIC,PERPLEXITY}_API_KEY and _BASE_URL, NVIDIA_API_BASE, NVIDIA_IMAGE_BASE_URL, PERPLEXITY_API_BASE, FAL_KEY, VLLM_API_KEY, HAIMAKER_API_KEY, LLM_KEY, LLM_URL, QDRANT_API_KEY and FIRECRAWL_API_KEY from mac.env and $HERMES_HOME/.env. 7. On the hub, enroll the host in ~/.mac/fleets.yaml and check it with mac admin fleet inventory. A token can only be issued to a registered agent, so register it first: mac admin machine register <hostname> --machine-id <machine_id> and mac agent register <machine_id> <name> --agent-id <agent_id>. Then run mac admin worker-token issue <agent_id> --install <host>. 8. Hub only: run deploy/install-postgres-service.sh, deploy/install-qdrant-service.sh, deploy/install-firecrawl-gateway.sh and deploy/install-webdav-server.sh (each with MAC_HOME, WORKSPACE and FLEET_NAME set). The Postgres installer writes MAC_DATABASE_URL into ~/.mac/mac.env; confirm it is there. The others write QDRANT_URL, the FIRECRAWL_* and MAC_WEB_SEARCH_* keys and MAC_PUBLISH_*. 9. Install the systemd units from deploy/systemd. mac-agent.service is rendered from deploy/systemd/mac-agent.service.in; the sed command is at the top of that file. Then run sudo systemctl daemon-reload && sudo systemctl enable mac-agent. 10. Prepare Hermes: - write $HERMES_HOME/mac-memory-topology.json (schema mac.hermes.memory_topology.v1, naming the hub's Qdrant and Firecrawl URLs) and, in $HERMES_HOME/.env, MAC_MEMORY_TOPOLOGY_FILE, QDRANT_URL, FIRECRAWL_API_URL, MAC_WEB_SEARCH_PROVIDER=firecrawl, HERMES_WEB_SEARCH_BACKEND=firecrawl and HERMES_WEB_EXTRACT_BACKEND=firecrawl; - force secret redaction: redact_secrets: true in $HERMES_HOME/config.yaml and HERMES_REDACT_SECRETS=true in $HERMES_HOME/.env; - write the runtime context, without which prepare refuses to run:

  ```console
  ~/.mac/venv/bin/python -m mac.hermes_runtime \
    "$HERMES_HOME/mac-runtime-context.json" "$HERMES_HOME/mac-runtime-context.md" "$HERMES_HOME/.env" \
    --agent-name <name> --fleet-name <fleet> --mac-url "$MAC_HUB_URL" \
    --hermes-home "$HERMES_HOME" --mac-home ~/.mac --workspace ~/.mac/src/mac \
    --tenant-id "$MAC_FLEET_TENANT_ID" --persona-id "$MAC_HERMES_PERSONA_ID" \
    --hermes-instance-id "$MAC_HERMES_INSTANCE_ID" --agent-id <agent_id>
  ```

- run `deploy/openshell/bootstrap-openshell.sh`, then
  `deploy/hermes/install-hermes-gateway.sh prepare`.

Copy each deploy/skills/fleet/<skill>/ that has a SKILL.md into $HERMES_HOME/workspace/skills/. On a GPU host (nvidia-smi -L works), also tar xzf deploy/skills/omniverse-skills.tar.gz -C $HERMES_HOME/workspace/skills and add the local-gen media units <fleet>-gen-server (port 8189), <fleet>-gen-audio-server (8190) and <fleet>-gen-video-server (8191). Each runs deploy/local-gen/{openai_image_server,audio_server,video_server}.py with ~/.mac/mac.env sourced.
From the hub: scripts/fleet-update <host> <sha>. This installs deploy/bin into ~/.mac/bin and starts the worker.

Updating the fleet with fleet-update¶