Synchronized Fleet Cut-over¶
Cut-over gate: the typed protocol in
fleet-cutover-transaction-protocol.mdis implemented, but production publication remains frozen until its complete repository and fault-matrix gates pass. This document is the operator path; it does not override those release blockers.
The fleet deployer uses a three-phase epoch: hold and drain the complete selected cohort, deploy and prove every new generation, then atomically hand the deployment holds to one successor hold in the hub transaction. A cohort is a failure domain, not a synonym for the whole fleet. Production rollouts select one supervisor family per cohort: systemd, launchd, or supervisord. Every lane converges to the same reviewed source and image identities, but a launchd fault cannot prevent a proved systemd or supervisord lane from committing. No selected worker becomes dispatchable between deployment and synchronized- pipeline activation. A worker that cannot join its lane remains held; the deployer never degrades an exact lane cut-over into a partial transition.
Synchronization here is a logical barrier, not a claim that heterogeneous hosts share a wall-clock start time. Each node may reach a barrier at its own speed, but no member may enter the next mutation or ownership phase until the controller has exact, generation-bound evidence from the entire selected lane. Monotonic deadlines bound local waiting without making clock agreement part of the safety proof. A deliberately fleet-atomic change may still select a mixed cohort, but normal software rollout must use supervisor-homogeneous lanes so an implementation-specific fault stays inside its own failure domain.
Architecture and acceptance invariants¶
The cut-over is complete only when all of these invariants hold together. A change that does not advance or protect one of them belongs outside the cut-over.
| Invariant | Enforcement boundary | Required evidence |
|---|---|---|
| Immutable release | Source commit, certifier image, and worker runtime are exact digest-pinned identities | CI publication receipts and local anonymous-pull verification name the same commit |
| Exact cohort | Stable agent IDs and current targets come from the frozen fleet registry | Selected set and release receipt contain the same IDs exactly once |
| Quiescent boundary | Every selected agent is held, drained, task-free, service-claim-free, and free of old supervisor- or daemon-owned resources before artifact mutation | Hub drain records, bounded supervisor-unload proof, exact OpenShell inventory, cross-runtime container inventory, and a generation-bound local quiescence receipt |
| Recoverable node generation | Source, virtualenv, service definitions, and supervisor topology change as one fenced generation; a failed replacement cannot be accepted as deployable | Prior-state receipt, owner-private artifact snapshots, mutation journal, bounded rollback receipts, successful remote transaction exit, and controller-side post-manifest revalidation |
| Crash-recoverable cohort | The controller persists intent before every mutation and retains the newest held node state for successor repair after a proved-absent hub commit; prior-generation restore is explicit break-glass only | Owner-fenced cohort journal, forward-retention receipts, retained diagnostic bundles, optional rollback receipts, and exact epoch-status reconciliation |
| Atomic ownership handoff | Deployment holds become one successor hold in a single database transaction | No committed row is unheld; stale ownership rolls back the entire cohort |
| Idempotent epoch | Epoch identity binds holds, outcome, successor reason, generation, credential principal, and report-executor expectations | Same request replays one receipt; any changed input is rejected |
| Attested generation | Every worker proves the deployed generation, bound principal, startup self-test, and read-only report executor | Exact post-deploy rows and controller approval match the immutable runtime |
| Controlled activation | Successor holds remain in place through activation | Runtime health passes before any explicitly selected canary is released |
1. Bootstrap an older hub by itself¶
An older hub may not expose reason-CAS or batch-release routes. Bootstrap only the configured hub while it is already operator-held:
MAC_DEPLOY_ALLOW_LEGACY_CAS_BOOTSTRAP=1 \
MAC_DEPLOY_OPENSHELL=1 \
MAC_DEPLOY_OPENSHELL_ARGS='--enable --fail-closed' \
MAC_DEPLOY_OPENSHELL_RUNTIME_IMAGE='ghcr.io/jordanhubbard/mac-openshell-runtime@sha256:<tested-main-digest>' \
deploy/deploy-mac-fleet.sh --hub <hub-agent> <hub-agent>
Legacy bootstrap preserves the existing operator hold. It rejects hold adoption, exact-full-cohort mode, and any non-hub target. Afterward, confirm the hub OpenAPI document contains all eight routes required by the cohort deployer:
/agents/{agent_id}/dispatch-hold/acquire/agents/{agent_id}/dispatch-hold/release/agents/dispatch-hold/release-batch/agents/dispatch-hold/transition-batch/agents/dispatch-hold/epochs/{epoch_id}/agents/{agent_id}/attestation-key/recover/agents/{agent_id}/report-repository-executor/approve/agents/{agent_id}/report-repository-executor/revoke
2. Authorize exact adoption of existing holds¶
The deployer does not infer permission to clear an operator hold. If selected
agents are already held, create a one-invocation authority file from freshly
read live rows. Use stable agent_... IDs and copy each reason exactly; never
put credentials or authenticated URLs in this file.
{
"schema": "mac.dispatch_hold_adoptions.v1",
"fleet": "<fleet-name>",
"hub_agent": "<hub-agent-name>",
"source_commit": "<exact-tested-main-commit>",
"adoptions": [
{
"agent": "agent_<stable-id>",
"reason": "<exact-current-dispatch-hold-reason>"
}
]
}
The file must be a regular, non-symlink file owned by the invoking user, be owner-readable, have no group/other permission bits, and be at most 1 MiB:
The deployer opens it without following a final symlink, validates duplicate keys and exact schema, and snapshots it once into the deployment's private temporary directory. It rejects a wrong commit, hub, fleet, unselected agent, duplicate agent, unheld agent, or reason drift.
3. Deploy the complete cohort¶
Resolve every target again from ~/.mac/fleets.yaml, then name every selected
agent explicitly. Supplying --hold-adoptions automatically enables
--require-release-all-selected:
MAC_DEPLOY_OPENSHELL=1 \
MAC_DEPLOY_OPENSHELL_ARGS='--enable --fail-closed' \
MAC_DEPLOY_OPENSHELL_RUNTIME_IMAGE='ghcr.io/jordanhubbard/mac-openshell-runtime@sha256:<tested-main-digest>' \
deploy/deploy-mac-fleet.sh \
--hub <hub-agent> \
--hold-adoptions /path/to/hold-adoptions.json \
--successor-hold-reason 'synchronized pipeline activation refreeze <exact-tested-main-commit>' \
<hub-agent> <agent-2> <agent-3> <agent-4> <agent-5>
Before hub epoch open or service mutation, the deployer resolves each route
from the frozen ~/.mac/fleets.yaml, binds its live endpoint identity, arms the
generation-specific phase-1 restore contract, and obtains a read-only
mac.fleet_prerequisite_bundle.v1. The bundle proves the selected supervisor,
required executables, owner-private input files, source and immutable image
identity, and required service readiness over either loopback or the configured
private Tailscale/Headscale mesh. Credential-bearing URLs, public endpoints, and
unmanaged network targets are rejected. Any missing prerequisite
fails the complete cohort before the hub is changed or any service is stopped.
Prerequisite repair and package or image installation are separate operations;
the synchronized transaction never performs them opportunistically.
The hub then opens one typed epoch that atomically validates the selected rows, exact prior holds, absence of active work, staged worker principals, candidate attestation keys, report-executor intent, and policy revision. The open receipt contains only identities and digests. Pending worker credentials and candidate keys remain staged alongside the current active identities until commit.
The controller also creates an owner-private durable cohort journal before the first remote mutation. The journal binds the ordered stable-agent set, exact source and runtime identities, deterministic per-agent generations, controller nonce, and every state transition. A phase-1 transition is accepted only with the digest of that node's retained restore contract; a phase-2 transition is accepted only after its exact generation-bound rollback contract is durable. Journal updates use compare-and-swap transitions and atomic, fsync-backed owner-only files. The journal contains no targets or credential material.
Controller exit is therefore a held repair boundary, not permission to abandon
or automatically undo a partly changed cohort. A live controller owns the
journal exclusively. A later invocation may adopt it only after proving the
recorded controller is gone. It closes the incomplete hub epoch, preserves each
node's newest observed state and immutable diagnostic bundle in reverse
mutation order, and releases only the old controller lock so a successor
generation can roll forward immediately. --recovery-policy rollback is a
separate break-glass path; when selected, it invokes only the exact retained
restore_executable.argv and generation rollback recorded before mutation.
The first node recovery action durably binds the policy so interruptions cannot
mix retention and restore.
The hub handoff has a distinct uncertainty rule. The controller durably records
commit-start before calling the atomic epoch endpoint. If the response is
lost, recovery queries the epoch by its exact identity digest. A matching
committed result forces roll-forward; a matching proved result replays the
same commit. Neither permits node rollback. An identity mismatch, transport
failure, or unknown response is a hard stop.
Open and prove requests are stored as owner-private, one-use replay envelopes on the hub before their journal intent is advanced. The durable cohort journal contains only their secret-free plans and digests. Recovery can therefore resolve a lost response without reconstructing credentials from ambient files, and removes each envelope after its receipt is journaled or the epoch aborts.
Bounded per-node fan-out inside each typed barrier¶
Every all-node barrier is still a hard barrier: phase 2 cannot begin until the controller holds one generation-bound receipt for every member of the frozen cohort, and the atomic hub commit still validates the complete cohort as one row set. What changed is how the controller reaches each barrier. Instead of six serial per-node loops plus serial SSH-master setup, the controller first durably records the whole-phase intent in the owner-private cohort journal, then runs the independent remote node operations for that phase concurrently with a conservative bounded fan-out, and only then serializes evidence validation and the journal compare-and-swap before crossing the barrier. Concurrency is confined to the independent remote work; endpoint identity binding, per-node locks, rollback-journal contracts, holds, the atomic hub commit, and fail-closed recovery are unchanged and remain serialized where the safety proof requires it.
The fan-out width is MAC_DEPLOY_NODE_PARALLELISM (default 4, accepted range
1-32). Set it to 1 to force the original strictly serial behavior; raise
it to shorten wall time on a wider cohort when the controller host and mesh have
headroom. The scheduler launches each node operation in its own child with a
distinct log, never exceeds the configured width, waits for every attempted
node, and reaps a child that dies before publishing its atomic status (for
example a SIGKILL) by synthesizing a controller failure rather than polling a
receipt that can never appear. Results are rendered in the frozen selection
order so a fast later failure can neither hide nor reorder an earlier node's
evidence, and each per-node failure persists a durable, secret-safe artifact
before the transient working directory is wiped. A first failure stops
scheduling new work but never abandons siblings that are already in flight; the
barrier is only crossed when every reaped node passed.
Each immutable archive and helper bundle is pre-staged once per node by digest and then reused for both arm and apply. The controller uploads the bundle a single time, records its size and SHA-256 in an owner-private manifest, and the later generation-scoped arm/apply steps verify against that manifest instead of re-uploading. This removes the previous duplicate transfer: the deployer no longer uploads the same ~15.8 MB archive twice per node, so a five-node cohort no longer performs ten serial transfers (~158 MB) of identical bytes.
These two changes attack the measured wall-time cost directly: the serial
per-node loops and the redundant per-node upload. The behavior is covered by
deterministic tests for failure, crash (child killed before status), selection
ordering, retry/fail-fast scheduling, and single-digest stage reuse for arm and
apply in tests/test_deploy_fleet_parallel_staging.py; those tests assert the
barriers, journal CAS, and endpoint/lock invariants above are preserved rather
than merely that the phases run in parallel.
After every selected hub row is held and drained, the controller invokes the
same phase-1 helper on every selected node. The helper stops and proves absent
the worker plus Hermes, OpenClaw, and legacy Nemo gateway identities through
the node's exact supervisor; supervisord nodes inspect every usable distinct
manager instance and deduplicate aliases by manager PID. The stop set
deliberately excludes the control plane. The helper then runs the reviewed
daemon-resource block before any source or image is replaced. Each node
atomically publishes an owner-only
mac.phase1_cohort_quiescence.v1 receipt that binds the supervisor proof, the
pre_source daemon receipt, the exact reviewed function-block digest, source
revision, and deployment generation. Phase 2 cannot begin until the controller
has one unique receipt for every member of the frozen cohort.
The supervisor proof records both prior_state and the final quiescent state
for every exact identity. The prior state is not reconstructed from a service
file or from a later supervisor sample: phase 1 has already stopped the old
generation by then. The receipt validator requires the complete identity set,
rejects duplicates and unknown states, and permits phase 2 only when every
final state is inactive or absent. On launchd, the node installer consumes that
generation-bound prestate as the expected prior state for per-service
compensation; it rejects a gateway or worker found in the system domain and
rejects multiple simultaneously active gateway owners because neither topology
can be restored unambiguously.
On launchd nodes, quiescence is a bounded state transition rather than a
one-shot sample: every managed worker or gateway job must become explicitly
absent in both relevant domains, unexpected inspection errors fail closed, and
all subprocess and total deadlines use a monotonic clock. The managed worker
plist uses a 30-second ExitTimeOut, leaving a strict cleanup margin after
launchd's terminal kill boundary.
The cut-over's worker, control-plane, chat-gateway, and hub-tunnel supervisor lifecycle operations are exact, non-interactive, and bounded on all three supported managers. Manager commands run in their own process groups under per-command bounds; multi-probe transitions and rollback share a total monotonic deadline as well. Service definitions are staged, atomically replaced with their parent directory fsynced, and coupled to an exact manager scope; a failing system-scope command is never retried against a different user-scope manager. systemd and supervisord definitions participate in the node-generation rollback described below rather than being treated as independent best-effort writes.
All repo-resident launchd installers additionally use
deploy/lib/launchd-lifecycle.sh for a per-service transaction: snapshot every
artifact, stage and validate the new plist, prove the old label absent,
atomically replace the plist, bootstrap only from proved absence, prove the new
label loaded, and only then run service health checks. EXIT or signal failure
stops the partial replacement, restores tracked artifacts in reverse order,
and restarts only the jobs recorded active in the prior state. Domain
migrations and auxiliary supervisor jobs use a deferred after-restore hook so
they start only after the primary old generation and every artifact have been
restored. Transported pre-source and rollback scripts embed the same bounded
state machine because the installed shared library is not yet trustworthy at
their execution boundary. For Qdrant, label absence is followed by explicit
removal and absence proof of the daemon-owned named container before the new
wrapper or plist is installed.
Supervisor absence alone is not a daemon-resource proof. Before source is
replaced, the node installer resolves the prior OpenClaw sandbox only from its
owner-private managed/sandbox-name identity (or the strictly parsed legacy
runtime.env field), inventories OpenShell through its node-local JSON API,
runs the old checkpoint wrapper first, and independently proves exact sandbox
absence after bounded deletion. Raw environment or command output is never
replayed into deployment evidence.
The same boundary inventories every provably node-local Docker context and
Podman connection, recording the exact daemon endpoint identities used for the
proof. Unix sockets qualify directly; unproven loopback TCP or SSH endpoints
fail closed because they may be tunnels to a remote daemon rather than proof
of local ownership. A legacy Nemo compose container is classified from its
exact service label plus an independent project, compose-path, or legacy
image-command-name corroborator; ambiguous ownership fails closed. A running
compose-only Nemo gateway is not an automatically recoverable supervisor
topology: deleting it loses the only runnable object, while restoring a unit,
plist, or supervisord program cannot recreate it. The required phase-1 rule is
to reject a running legacy container before mutation. Until the daemon gate
enforces that distinction, any node on which a legacy Nemo container exists is
ineligible for synchronized cut-over. It must first be migrated to a
manager-backed, version-pinned Nemo service with a durable sandbox identity,
checkpoint, and restore manifest, or retired by a separate operation with an
explicit container restore journal. The current reference Nemo installer does
not publish that sandbox/state contract, so a prior active Nemo gateway remains
ineligible for phase 2 even when its supervisor process is manager-backed; a
stable restarted PID alone is not continuity proof. Stopped legacy containers
are not part of the pre-source deletion set and may be retired only after the
new fleet epoch commits. Two successive empty running-container inventories
are required. An owner-only
mac.daemon_resource_quiescence.v1 receipt is first atomically published for
the pre_source boundary, bound to the deployment generation, revision, and
runtime endpoint set. That same receipt must match the current endpoint set.
Every implementation proves pre_install and post_install; OpenClaw also
proves pre_verify and pre_finalize around its ownership handoff. This
prevents a restart policy from making an old container impersonate new gateway
health between lifecycle steps. A failed late proof makes the deployment fail
closed; OpenClaw rollback must itself prove exclusive shutdown or report a
second explicit failure, and its advertisement and pending-verification
artifacts are withdrawn.
The same closed-world rule applies to the optional GPU image, audio, and video servers. Their three systemd units are not yet members of the generation rollback manifest, so phase 1 requires all three units to be absent and a synchronized install does not create them. A GPU node with any of those units present is ineligible for this cut-over; media generation can be re-enabled only after those units, wrappers, enablement intent, and health evidence join the same prepare/rollback contract. Optional does not mean exempt from the transaction boundary.
After supervisor installation, a separate owner-only
mac.gateway_readiness.v1 receipt proves two stable observations of the exact
selected gateway process, with all competing gateway identities non-running.
For gateway_impl=none, it proves all gateway identities non-running. The post
manifest binds the phase-1, daemon-resource, and gateway-readiness receipt
digests; the latest manifest must be byte-for-byte equivalent in those
summaries.
An existing node arms break-glass rollback only after complete prior source and
virtualenv backups exist. Before phase-2 writes, it also snapshots the complete
executable tree, environment and generation markers, successor gateway
definitions, and the exact prior supervisor topology. After the old OpenClaw
sandbox has checkpointed and proved absent, the node snapshots the whole
host-side OpenClaw runtime tree, including managed config, policy, workspace,
state, credentials, image identity, and publication records. That ordering is
intentional: an earlier snapshot loses the final sandbox checkpoint, while a
later snapshot captures successor configuration. A prior active OpenClaw
topology without this owner-private runtime snapshot is ineligible for phase 2.
OpenClaw's script-backed host jobs have one additional manager-owned surface:
systemd-user ${fleet}-openclaw-script-*.service/.timer definitions or
launchd com.${fleet}.openclaw-script-*.plist definitions and their loaded
state. The runtime-tree snapshot covers their runner, job specification, and
output, but not those definitions. Phase 1 now inventories every exact-prefix
definition into an owner-private, digest-bound host-automation journal, records
systemd-user enablement or launchd loaded/disabled-override state, and quiesces
the recorded jobs before phase 2. Synchronized OpenClaw preparation accepts
host scheduling only when the exact phase-1 contract and quiescence receipt
authorize it. Rollback unloads and removes successor-only definitions, restores
the prior bytes and modes, reconstructs enablement/override intent, restarts
only jobs recorded loaded, and proves the resulting topology before publishing
its completion receipt. Loaded jobs without a safe exact definition and active
systemd oneshot services remain ineligible because they cannot be replayed
without inventing prior state.
A new node with no complete prior generation remains held and cannot have a
post manifest accepted as deployable. Any nonzero exit retains the newest
state, writes an owner-only mac.fleet_node_forward_retention.v1 receipt,
keeps the node-local startup barrier and immutable staged bundle, and releases
only the controller lock. Loss of the fence forbids further mutation. A later
top-of-tree deployment adopts the hold and repairs this state in place instead
of first reconstructing an increasingly expensive old world.
Rollback preflights every required backup before changing supervisor state, then uses one bounded helper to stop every exact MAC service identity and prove the control-plane port closed. If the failed generation created an OpenClaw sandbox, that sandbox must also checkpoint or be deleted and prove absent before its host runtime tree is replaced. This is a cut-over eligibility gate, not a best-effort cleanup: a generated rollback that proves only supervisor quiescence does not yet satisfy this contract. Rollback restores source, virtualenv, executable and environment surfaces, the prior OpenClaw tree (or restores its prior absence), and every tracked service definition before any service is restarted. The reviewed safe topology is the topology phase 1 actually recorded: the prior control-plane mode, exactly one of Hermes, OpenClaw, NemoClaw, or no gateway, and the prior active/inactive/absent worker state. The OpenClaw component's in-transaction compensation is withdraw-only; it cannot guess Hermes because only the outer transaction owns this topology record. Two stable process observations plus control-plane HTTP health when applicable are required before a restore receipt is published. If restore fails after a partial start, the helper re-quiesces every identity; if artifact restoration fails mid-journal, reverse compensation returns artifacts to their pre-rollback state. Either failure leaves the node held and produces no success receipt; the failed remote transaction prevents the controller from accepting any post manifest left by the interrupted generation.
After every node reaches prepared, the controller sends the complete set of
generation-bound install receipts, pending-principal heartbeats, node-authored
candidate-key challenges, and report-executor startup evidence to prove.
Partial proof never advances the hub. The controller journals commit-start
and calls commit with the exact epoch identity. In one database transaction,
the hub revalidates the cohort and promotes the staged principals, attestation
keys, report-executor policy, and successor holds while withdrawing service
claims. The committed receipt names the exact cohort, identity digest,
generation bundle, and successor-hold reason.
Hub commit is irreversible but not terminal. Each node is then finalized by the exact helper digest recorded when phase 2 was armed. Finalization removes only generation-owned barriers, locks, and retained transient artifacts and publishes a typed receipt. A controller crash after commit is recovered by resuming these finalizers; it never rolls nodes back. The cohort journal becomes terminal only after all finalization receipts are durable.
Before commit, recovery first rebinds every current route to its journaled endpoint identity. It resolves any lost hub open or prove response from the one-use envelope, aborts the exact hub epoch, discards only unreserved pending credentials created by that epoch, and then records forward retention for nodes in reverse mutation order. Current active credentials, unrelated pending credentials, installed files, and diagnostic bundles are never removed. The result remains held and is immediately eligible for a successor roll-forward deployment. Explicit rollback uses the same route and evidence fences but is never selected by default.
Attestation-key recovery and report-executor approval¶
Every selected loop worker receives an epoch-owned pending principal and candidate attestation key while its current principal and key remain valid. Secret material is relayed only through owner-private files, installed under the node deployment fence, and proved after restart by a node-authored challenge response. The worker never receives attestation-rotation authority, and a bound worker credential cannot call an administrative recovery endpoint.
For an OpenShell-enabled loop worker, the controller then fetches the exact
current report_repository_executor_attestation plus the matching startup
self-test timestamp and installs approval with an admin-only compare-and-set.
The hub derives report_repository_executor; workers cannot submit that marker
or its approval themselves. A mismatch or failed approval explicitly revokes
eligibility. Both the per-node preparation gate and the atomic hub epoch commit
revalidate the derived marker and its matching startup proof. This path is the
same for launchd, systemd, supervisord, and the SSH-managed selected GKE worker
pods; a selected target that cannot complete it remains held.
4. Release work only after the epoch is proven¶
Keep the successor holds in place during the fleet cut-over. Only once all agents report the new source commit, generation, bound worker principal, idle/healthy state, and the exact successor-hold receipt may work be released, and then only the intended canary or work cohort.