Production Deployment¶

Two supported topologies:

Single host, systemd — one machine, one PostgreSQL database, one FastAPI process. Suitable for dev fleets, personal Hermes runtimes, and pilot deployments. See deploy/systemd/.
Containerized, single-instance — image at Dockerfile. Same PostgreSQL topology, but lifecycle is managed by Docker Engine/Moby or k8s as a single-replica deployment. See the container section below.

PostgreSQL is the only supported control-plane authority in every topology. Set MAC_DATABASE_URL to an explicit PostgreSQL DSN. The legacy MAC_DB variable accepts a PostgreSQL DSN too; a SQLite path is rejected. There is no implicit home-directory database or client-side replica.

Ordinary service startup verifies the ordered schema-migration ledger and performs no DDL. Bootstrap and upgrades run mac-schema-migrate explicitly before starting the candidate service; see the versioned schema decision.

Only a hub or stateless API replica is a control-plane server. Fleet spokes are clients: their generated environment has MAC_CONTROL_PLANE_ROLE=client, no MAC_DB or MAC_DATABASE_URL, and their worker/gateway processes use MAC_HUB_URL. A redeployed spoke archives an inactive legacy mac.db; active legacy tasks block deployment until explicitly migrated to the hub. Store construction also rejects MAC_CONTROL_PLANE_ROLE=client even if a stale database variable is reintroduced.

Required configuration¶

Variable	Required	Purpose
`MAC_SECRET_KEY`	yes	32+ char secret; HKDF input for the Fernet key that encrypts secret values. Refuses to start without it.
`MAC_CONTROL_PLANE_ROLE`	yes for fleet hosts	`hub` for the single database-owning authority; `client` for database-free spokes.
`MAC_DATABASE_URL`	conditional	PostgreSQL DSN (`postgresql://...` or `postgres://...`). Preferred authority setting; overrides `MAC_DB`. Startup verifies the migration ledger; deploy applies migrations explicitly.
`MAC_PG_POOL_SIZE`	no	`psycopg_pool` max connections per `mac-api` replica. Default `10`.
`MAC_DB`	conditional	Legacy name for an explicit PostgreSQL DSN. SQLite paths are rejected; no private client ledger is created.
`MAC_API_TOKEN`	no	Single admin bearer token. Set empty string is rejected.
`MAC_API_TOKENS`	no	JSON `{token: [scopes,...]}` for scoped auth. Mutually exclusive with `MAC_API_TOKEN`.
`HERMES_HOME`	no	Hermes state directory checked at startup. Default `~/.hermes`.
`MAC_HERMES_AGENT_DIR`	no	Hermes checkout inspected for the `slack_accounts.json` activation shim. Falls back to `HERMES_AGENT_DIR`, then `~/Src/hermes-agent` if present.
`MAC_HERMES_APPLY_SLACK_ACCOUNT_SHIM`	no	Set `0` to disable startup patching of an explicit `MAC_HERMES_AGENT_DIR`. Default enabled only when the checkout path is explicit.
`MAC_HERMES_APPLY_GATEWAY_RUNTIME_SHIM`	no	Set `0` to disable startup patching of Hermes gateway model/runtime overrides. Default enabled for explicit checkout paths.
`MAC_HERMES_GATEWAY_MODEL`	no	Per-agent model used by Hermes gateway conversations and mirrored to `HERMES_INFERENCE_MODEL` for oneshot worker execution.
`MAC_HERMES_GATEWAY_PROVIDER`	no	Runtime provider for the per-agent model. Fleet hosts normally use `custom` so Hermes sends OpenAI-compatible requests through MAC's router.
`MAC_HERMES_GATEWAY_BASE_URL`	no	OpenAI-compatible base URL for Hermes. Fleet hosts use the hub's local in-mac `/v1` endpoint on the hub and the hub or wing-router `/v1` endpoint on spokes.
`MAC_HERMES_STARTUP_CHECK`	no	Set `0` to disable Hermes state and Slack startup checks. Enabled by default.
`MAC_REQUIRE_HERMES_STARTUP_READY`	no	Set `1` to fail `mac` startup when Hermes soul/memory/state references or Slack activation are not ready.
`MAC_HERMES_SLACK_HOME_CHANNEL_NAME`	no	Slack home-channel name, without `#`, used to write `~/.hermes/slack_home_channels.json` from `slack_accounts.json`. Empty skips discovery.
`MAC_HERMES_SYNC_SLACK_HOME_CHANNELS`	no	Set `0` to preserve existing Slack home-channel files without discovery. Default enabled.
`MAC_URL` / `MAC_HUB_URL`	no	MAC API endpoint used by Hermes-side `mac-hermes` operations. Fleet hosts point this at the hub.
`MAC_CLIENT_PRINCIPALS_FILE`	no	Hub-local hashed registry for scoped client enrollment. Fleet hosts set `$MAC_HOME/client-principals.json`; permissions must be `0600`. The API hot-reloads issuance and revocation.
`MAC_HERMES_INSTANCE_ID`	no	Hermes instance id for this runtime. Fleet hosts use a deterministic `hermes_<agent>` id and registers it in MAC.
`MAC_WORKER_HERMES_INSTANCE_ID`	no	Worker agent binding to the Hermes instance id. This keeps MAC agent rows linked to their Hermes soul/runtime.
`MAC_AGENT_ID`	no	Deterministic MAC agent id for this runtime. Fleet hosts use `agent_<agent>`.
`MAC_HERMES_RUNTIME_CONTEXT_FILE`	no	Hermes-visible task/project runtime contract JSON. Default `~/.hermes/mac-runtime-context.json`.
`MAC_HERMES_RUNTIME_CONTEXT_MARKDOWN`	no	Human/agent-readable runtime contract summary. Default `~/.hermes/mac-runtime-context.md`.
`MAC_HERMES_RUNTIME_CONTEXT_REQUIRED`	no	Set `1` to make startup readiness fail if the MAC task/project runtime contract is missing, invalid, or not injected into the Hermes prompt builder. Fleet hosts enable this.
`MAC_HERMES_WORKSPACE`	no	Source workspace Hermes should treat as equivalent to an operator/Codex shell in the MAC repo. Fleet hosts set this to `$MAC_HOME/src/mac`.
`MAC_PROJECT_CONTRACT_FILE`	no	Repository contract file for the Hermes direct-session capability bridge. Fleet hosts set this to `$MAC_HERMES_WORKSPACE/.mac/project.yaml`.
`MAC_WORKER_EXECUTOR`	no	Executor command used by loop-mode workers. The default `~/.mac/bin/mac-hermes-task-executor` is part of the Hermes direct-session capability proof.
`GH_TOKEN` / `GITHUB_TOKEN`	no	GitHub HTTPS credential used by task, review, publication, and pushed-ref verification commands. The credential may appear only in the individual Git command and must not persist in `origin`, evidence, logs, or memory.
`GITEA_TOKEN`	no	Gitea HTTPS credential for the same Git operations. `MAC_TASK_GIT_TOKEN` is the host-mode fallback when no host-specific token is set.
`MAC_DEPLOY_GH_TOKEN`	no	Fleet-deploy input copied into the managed runtime as `GH_TOKEN`. Keep it in the host-local `~/.mac/.env`, never in `fleets.yaml` or a committed spec.
`MAC_REPOSITORY_REF_RECONCILER_MODE`	no	Managed task-branch reconciler mode: `off`, `audit`, or `prune`. Runtime default `off`; fleet hubs normally use `prune` and spokes `off`.
`MAC_REPOSITORY_REF_RECONCILER_INTERVAL_SECONDS`	no	Delay between automatic passes, bounded from `60` through `604800`. Hub default `86400` (daily).
`MAC_REPOSITORY_REF_RECONCILER_INITIAL_DELAY_SECONDS`	no	Delay before the first automatic pass, bounded from `0` through `86400`. Hub default `300`.
`MAC_REPOSITORY_REF_RECONCILER_GRACE_DAYS`	no	Fallback cleanup grace for legacy lifecycle records, bounded from `0` through `365`. Default `7`.
`MAC_REPOSITORY_REF_RECONCILER_REMOTE` / `MAC_REPOSITORY_REF_RECONCILER_BASE_REF`	no	Git remote name (default `origin`) and optional explicit `<remote>/<branch>` ancestry target. Without a base override, the remote HEAD is auto-detected.
`MAC_REPOSITORY_ACCESS_FAILURE_COOLDOWN_SECONDS`	no	How long a newest authentication/authorization failure excludes a reviewer for the matching project, repository host, and operation. Default `1800`.
`MAC_REPOSITORY_ACCESS_SUCCESS_TTL_SECONDS`	no	How long a successful repository-access learning stays current and supersedes an older failure. Default `86400`.
`MAC_SUPERVISOR_KIND`	no	Runtime supervisor on this host: `systemd`, `launchd`, or `supervisord`.
`MAC_MEMORY_TOPOLOGY_FILE`	no	Hermes-visible memory topology JSON. Default `~/.hermes/mac-memory-topology.json`.
`MAC_SHARED_SERVICES_MANAGER_AGENT`	no	Agent that owns hub-managed shared services. Defaults to the configured fleet hub.
`QDRANT_URL` / `QDRANT_ADDRESS` / `QDRANT_FLEET_URL`	no	Shared Qdrant level-2 memory endpoint. When set, Hermes startup readiness validates `/collections`.
`MAC_REQUIRE_QDRANT_MEMORY`	no	Set `1` to require shared Qdrant memory readiness. Fleet hosts enable this.
`MAC_QDRANT_MEMORY_ALLOW_DEGRADED`	no	Temporary operator override that allows startup when required Qdrant is missing or unreachable.
`QDRANT_PIDS_LIMIT`	no	Container PID/thread cap for the hub-managed Qdrant (supervisord wrapper + systemd unit). Default `4096`. Raise on very-high-core nodes; see Troubleshooting.

Generate a secret key once:

openssl rand -base64 48

Store it in a secrets manager. Rotating the key requires re-encrypting all secret values; today this is a manual procedure (re-emit every secret with the new key).

Environment variable precedence¶

MAC resolves each MAC_* variable according to a three-level contract:

Process environment wins. A variable already present in the invoking process environment (e.g. set by the supervisor unit, injected by a secret manager, or exported by the operator shell) is used as-is and is never overridden by the env file.
Env file supplies defaults. A variable absent from the process environment receives its value from the operator env file (typically ~/.mac/.env or the path given by MAC_ENV_FILE).
Env file is the operator default store. Operators should record stable deployment values — tokens, URLs, feature flags — in the env file. Runtime overrides belong in the process environment and are not written back to the file.

See env-config-reference.md for the full variable catalog.

Systemd¶

# 1. Create the service user and data directory.
sudo groupadd --system mac
sudo useradd --system --gid mac --home-dir /var/lib/mac \
    --shell /usr/sbin/nologin mac
sudo install -d -o mac -g mac -m 0750 /var/lib/mac

# 2. Install mac globally (or into a venv at /usr/local/lib/mac).
sudo pip install /path/to/mac-0.1.0-py3-none-any.whl

# 3. Write the env file (mode 0600, owner root:mac).
sudo install -d -o root -g mac -m 0750 /etc/mac
sudo install -o root -g mac -m 0640 deploy/systemd/mac.env.example /etc/mac/mac.env
sudo $EDITOR /etc/mac/mac.env       # set MAC_SECRET_KEY, optionally MAC_API_TOKEN

# 4. Install and start the unit.
sudo install -o root -g root -m 0644 deploy/systemd/mac.service \
    /etc/systemd/system/mac.service
sudo systemctl daemon-reload
sudo systemctl enable --now mac.service

# 5. Verify.
sudo systemctl status mac.service
curl -fsS http://127.0.0.1:8789/health

The unit is started with python -m mac.hub_serve, which reads ${MAC_BIND_HOST} and ${MAC_PORT} from /etc/mac/mac.env. Worker agents bind 127.0.0.1. Mesh hubs (Tailscale/Headscale) bind loopback plus the CGNAT address from tailscale ip -4 and refuse 0.0.0.0, so the API cannot accidentally listen on a public NIC. The overlay is the transport; it is not a host firewall unless the process also refuses other interfaces. In-cluster images with no mesh provider may still bind 0.0.0.0 inside the container network namespace. Put a TLS-terminating reverse proxy in front only when the hub must be reached off-mesh.

Provisioning and updating fleet hosts¶

Fleet hosts are provisioned by hand and updated with scripts/fleet-update. Both are described in Updating the fleet with fleet-update:

New host: follow its "Provision a new host" checklist.
Update: run scripts/fleet-update <hub|HOST|all> <sha> on the hub.

In all mode, a worker that remains busy after the drain timeout is left unchanged and its update hold is released; the rollout continues with the remaining workers. The command then prints the skipped workers, their task IDs, and a fleet-update --yes <worker> <sha> command for each, and exits with code 3 rather than reporting success. Other update failures still stop the rollout immediately. Updating a single busy worker retains the existing immediate failure behavior.

The old setup wizard (setup.sh / make setup), the declarative mac.fleet_setup.v1 specs and make deploy / deploy/deploy-mac-fleet.sh have been deleted.

Reaching the Hub Node¶

The hub control plane binds to hub_url as declared in ~/.mac/fleets.yaml. How you reach it from a client machine depends on the network topology.

Client bootstrap: mac admin login now combines strict host-key verification, a managed local tunnel, hub-local scoped enrollment, authenticated validation, and atomic profile installation. Do not copy the hub's ~/.mac, admin token, mac.db, MAC_SECRET_KEY, provider keys, or SSH private keys to a new client.

For a directly reachable hub, the interim client setup is:

export MAC_API_URL=https://mac.example.internal
export MAC_API_TOKEN=<scoped-client-token>
mac admin diagnostics
mac task stats
mac agent list

Use a mode-0600 env file or a process environment supplied by a secret manager. --token exists for automation/recovery but is not the preferred interactive path because argv can be visible in shell history and process inspection.

Direct access (same network or VPN)¶

Hub is directly routable — no tunnel needed:

# Confirm health
curl http://<hub-host>:8789/health

SSH port forward and scoped enrollment¶

Hub lives behind a bastion or inside a K8s cluster. Record the route in the home-scoped fleet registry so deploy, credential recovery, snapshots, migration, and Electron use the same resolver:

# ~/.mac/fleets.yaml
fleets:
  my-fleet:
    hub_agent: hub
    control_port: 8789
    defaults:
      ssh_jump: horde@bastion.example.com:2222
      identity_file: ~/.ssh/mac-my-fleet
      ssh_known_hosts_file: ~/.ssh/mac-my-fleet-known-hosts
      ssh_host_key_policy: strict
    agents:
      - name: hub
        target: horde@my-hub.cluster.local
        os: linux

Provision and verify the hub and bastion host keys in ~/.ssh/known_hosts before opening the tunnel. Disabling host-key verification is not a supported bootstrap substitute.

The normal client path is one command:

mac admin login --fleet my-fleet --profile my-fleet --client-id my-laptop
mac admin login status --profile my-fleet
mac task stats
mac agent list

For a reproducible production profile the fleet route should resolve an explicit identity file and a verified known-hosts/host-CA source; login validates and pins whatever is supplied. An equivalent route can be given without a fleet registry using --ssh, --identity-file, --known-hosts-file, optional --ssh-port, and optional --proxy-jump. A directly reachable hub may use --host-key-fingerprint SHA256:...; proxy-jump routes require a prepared known-hosts file. Anything omitted falls back to OpenSSH's own resolution (default identities/agent, ~/.ssh/known_hosts with accept-new), so mac admin login --ssh <host> works like ssh <host> — but such a profile then depends on the enrolling machine's ambient ssh state rather than being fully portable.

The commands below remain the low-level recovery procedure when diagnosing SSH or enrollment independently.

# Confirm the route is portable and does not depend on ~/.ssh/config.
mac admin fleet ssh-spec --fleet my-fleet --agent hub --portable --json

# Forward hub control port to localhost and open the UI
ssh -N -F /dev/null \
  -o BatchMode=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile="$HOME/.ssh/mac-my-fleet-known-hosts" \
  -o ProxyJump=horde@bastion.example.com:2222 \
  -i "$HOME/.ssh/mac-my-fleet" \
  -L 8789:127.0.0.1:8789 \
  horde@my-hub.cluster.local
# then: curl http://localhost:8789/health
# or:   open http://localhost:8789/ui

With that tunnel open, use a second verified SSH session to invoke enrollment locally on the hub and stream the one-time token directly into the client profile store:

ssh -T -F /dev/null \
  -o BatchMode=yes \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile="$HOME/.ssh/mac-my-fleet-known-hosts" \
  -o ProxyJump=horde@bastion.example.com:2222 \
  -i "$HOME/.ssh/mac-my-fleet" \
  horde@my-hub.cluster.local \
  'mac --json admin client enroll my-laptop \
    --fleet my-fleet --profile my-fleet \
    --api-url http://127.0.0.1:8789 \
    --scopes read,write,dispatch' \
  | mac admin client profile install -

mac --profile my-fleet diagnostics
mac --profile my-fleet task stats

The API hot-reloads $MAC_HOME/client-principals.json, so the new credential works immediately. For normal operation, renewal rotates and validates the bearer before replacing the local credential; revoking logout invalidates it before deleting local state:

mac admin login renew --profile my-fleet
mac admin logout --profile my-fleet --revoke

The equivalent low-level recovery commands are:

ssh -T horde@my-hub.cluster.local 'mac --json admin client renew my-laptop' \
  | mac admin client profile install - --profile my-fleet

ssh -T horde@my-hub.cluster.local 'mac admin client revoke my-laptop'
mac admin client profile remove my-fleet

Use the explicit SSH options above for renewal and revocation as well. The short form keeps the lifecycle example readable. See SSH Client Bootstrap Contracts for schemas, file modes, failure rules, and legacy migration.

Tailscale mesh (`provider: tailscale`)¶

Hub and spokes join the same Tailscale network. The hub is reachable at its Tailscale IP or MagicDNS name without any SSH tunnel:

# ~/.mac/fleets.yaml
defaults:
  network:
    provider: tailscale
    tailscale:
      auth_key_env: MAC_DEPLOY_TAILSCALE_AUTH_KEY

# Hub is reachable at its Tailscale IP, e.g. 100.x.x.x:8789
curl http://100.x.x.x:8789/health

Join each host to the tailnet by hand (deploy/install-tailscale.sh) before provisioning it.

Headscale (self-hosted control plane, `provider: headscale`)¶

Headscale manages the WireGuard mesh. The fleet registry must declare the login server, DNS assumption, health URL, and pre-auth key source:

# ~/.mac/fleets.yaml
defaults:
  network:
    provider: headscale
    headscale:
      manage: false          # true if mac should install/manage the headscale binary on the hub
      login_server: https://headscale.example.com
      health_url: https://headscale.example.com/health
      preauth_key_source: env
      preauth_key_env: MAC_DEPLOY_HEADSCALE_PREAUTHKEY
      dns: magicdns
      ip_prefix: "100.64.0.0/10"

# Hub reachable at its headscale-assigned IP or MagicDNS name
curl http://hub.headscale.example.com:8789/health

deploy/install-headscale.sh installs and configures a headscale server on the hub node when you manage it yourself.

Host layout and rollback¶

Every host runs the hub or worker from a git checkout at ~/.mac/src/mac with a venv at ~/.mac/venv; scripts/fleet-update moves that checkout to a new commit. Provisioning, the per-host update steps and rollback are in Updating the fleet with fleet-update.

Upstream provider credentials stay centralized on the hub, resolved by the in-mac router from MAC's encrypted vault or the host-local environment; workers receive only their hub-facing MAC token. Put GH_TOKEN in a worker's ~/.mac/mac.env (mode 0600) so confined tasks can clone and publish; never put it in ~/.mac/fleets.yaml, task metadata, or source control.

On a macOS hub, Docker containers for system services are expected. Qdrant, Firecrawl, test PostgreSQL, and telemetry collectors are long-lived network services whose container isolation is not part of an executor attestation. This does not permit OpenShell execution sandboxes on darwin: those tasks run as host applications and attest macos_host, because Docker Desktop's LinuxKit kernel cannot enforce Landlock against the macOS host. Use docker ps to confirm that observed containers are known system services; no container may be treated as a sandbox or as evidence of execution confinement.

Worker Agents¶

The control-plane service does not execute tasks by itself. Each execution host must run a worker process that registers or refreshes its machine/agent row, heartbeats, then claims eligible open work with a real executor. On fleet hosts it runs as the mac-agent service (deploy/systemd/mac-agent.service.in). In heartbeat mode a host is visible in the hub registry without claiming work:

mac-agent --url http://hub.example.internal:8789 --register \
  --agent-name worker-1 --hostname worker-1.local \
  --capabilities python,ops,review --resources '{"capacity":2}' \
  --heartbeat-only

mac-agent --url http://hub.example.internal:8789 --register \
  --agent-name worker-1 --capabilities python,ops,review \
  --workspace ~/.mac-agent/workspaces --loop \
  --executor ~/.mac/bin/mac-hermes-task-executor

Use --heartbeat-only during deploy validation when you want fleet visibility without claiming migrated ACC work. Start the --loop form only after the executor command is the intended production worker. Successful executions write log evidence, move tasks to needs_review, and ask the control plane to run the default review workflow. The worker's pre-push verifier run, on a fresh clone of the exact pushed commit, is the review verdict: once the executor evidence validates (for repository changes that includes a real verifier pass), the virtual hub-reviewer signs a separate review_verdict over it and the workflow publishes/completes the task. No reviewer agent is selected and no tests are re-run on the hub. A reviewer assigned explicitly through the review API must be independent of the executor; its signed verdict is honoured. Failed executions fail the task with evidence attached.

For high-risk work, set metadata.review.risk_level to high or critical. Approval then fails closed unless the signed executor and verdict manifests identify different model families and different upstream providers; merely using two versions of Claude, GPT, or another single lineage is insufficient. Unknown lineage/provider metadata also blocks approval. The same constraints can be enabled individually with review.require_different_model_family: true and review.require_different_model_provider: true.

{
  "review": {
    "risk_level": "high",
    "require_independent_reviewer": true
  }
}

How approved work lands: pull requests, not pushes to main¶

Publication is how an approved task reaches the canonical branch. The default is a pull request, matching the worktree-and-PR discipline the repository already asks of humans:

The agent works in its own worktree on its own branch and pushes only that branch (the executor finalizer's guarded_push, which pushes an explicit HEAD:refs/heads/<task-branch> refspec and refuses if the canonical tip is not already contained in the branch).
The agent opens its own pull request, onto the task's canonical branch (from the repository contract — never assumed to be main), and records it in the worker evidence manifest as repo.pull_request.
On approval, the hub clones the exact canonical tip, runs the merge gate, makes sure the reviewed commit is on the task branch on the remote, and reuses the agent's pull request. It opens one itself only when the evidence carries none — an older worker, or a forge the agent could not reach — and that fallback is recorded (opened_by: hub_fallback) rather than being indistinguishable from the normal path.
The forge lands that pull request, pinned to the reviewed head SHA (the pull-request equivalent of --force-with-lease: if the branch moved underneath us, the merge is refused rather than landing something nobody reviewed) — through a squash merge, one landing at a time per repository, see The serial land loop below.
The hub verifies the resulting canonical tip and records it as the canonical-integration proof.

The hub never pushes the canonical branch on this path, and it does not author pull requests: opening one is the agent's decision about how its own work lands, and the hub's job is to record that it happened and to gate completion.

The agent's forge credential. The agent uses the credential already in its own process environment (GH_TOKEN/GITHUB_TOKEN/GITEA_TOKEN — the same variable guarded_push authenticates with, written into ~/.mac/mac.env at deploy time). When it has none, it resolves github.token from the hub's secret store by name, at the moment of use — audited, never cached, never written to evidence, never carried in a bus message or task metadata. A resolved secret that does not claim the forge's capability is refused rather than sent to the API.

Verify, do not assume. The party requesting the merge confirms the required contexts actually passed for the reviewed head SHA before asking, rather than trusting the forge to refuse if they have not. This is not belt-and-braces pedantry: the fleet authenticates as the repository owner, and the ruleset carries a RepositoryRole bypass with mode always (deliberate operator break-glass), so the merge request inherits that bypass and can land straight past required checks — the first publication under this flow merged two seconds after the pull request was opened, with every required context reported SKIPPED. Nothing gated it: not the forge (bypassed) and not the hub (which skips its own contract re-projection precisely because the forge reports required checks).

A context that has not reported, is still running, or reported skipped is not a pass; publication defers with publication_failure_kind=pull_request_checks_pending. A context that reported a real failure raises pull_request_checks_failed. An empty required-contexts list is a different case from a required list that is still pending, and the two are recorded separately in the publication evidence (required_check_verification.case: none_configured, pending, failed, unverifiable, or verified) — an unprotected repository is gated by the worker's own verifier run instead, and a repository whose checks simply have not started is not mistaken for one.

The serial land loop¶

One test gate per landing. A repository with required status checks is gated by those checks; a repository without them is gated by the worker's own verifier run, which ran the contract on a fresh clone of the reviewed head after the finalizer synced it onto the canonical tip (repo.canonical_sync.canonical_tip in the evidence). The hub runs no tests.

One land step per publication attempt, serialized per repository. Each publication attempt takes a PostgreSQL advisory lock keyed on (repository, canonical branch) for the whole land step. A second land step for the same repository does not queue behind it; it waits a tick (publication_failure_kind=landing_serialized, charged to the landing deadline only). A per-task advisory lock is taken first: when the sweep and the event-driven consumer both reach the land step for the same task, the second returns landing_in_progress and writes nothing to the task, because any task write would revoke the first land step's authority fence. That fence re-checks the task row immediately before the forge merge; if the row changed, the attempt raises publication_authority_revoked, which charges one landing attempt and backs off, and the next tick starts a fresh attempt. Inside the lock, ControlPlane._publish_git_target_attempt:

clones the current canonical tip and fetches the reviewed head;
runs git merge-tree (mac.merge_queue.validate_projected_merge, no tests) — a conflict sends the task back to its worker to rebase;
with required checks: opens or reuses the pull request, reads required_check_verdicts for the reviewed head, and requests the squash merge once they pass. Pending checks wait under the landing deadline; failed checks send the task back to its worker to fix them (below);
without required checks: lands only if the canonical tip is still the base the worker verified (it is an ancestor of the reviewed head) — through the pull request, or by a fast-forward --force-with-lease push on the direct-push path — re-validating the tip immediately before the merge. A tip that has moved sends the task back to its worker to rebase and retest.

Optimistic concurrency. If the tip moves between the land step's read and its merge (someone pushed outside mac), the guarded push or the re-validation refuses, and the step retries once at once; the retry then sees a tip the worker never verified and sends the task back.

Rebase and retest. A send-back moves the SAME task from review (needs_review, or a legacy reviewing row) to OPEN with a rebase_onto_tip directive in its metadata (the canonical tip, the verified base, the reviewed head and branch, any conflicted paths). The executor prompt renders it as a "Sent back to rebase" section: rebase onto the tip, resolve the named conflicts, keep the previous work. The next claim re-runs the worker, whose finalizer rebases onto the current tip before the verifier runs, so the new evidence is verified against the tip it will land on. The previous attempt's pull request (the agent's, or the one a land step was working on) is closed with a "Superseded" comment; a forge that refuses is logged (workflow.default_review.superseded_pr_close_failed) and never blocks the send-back. Send-backs are counted in metadata.landing.rebases and capped at two; the third blocks the task (landing_rebase_cap_exhausted). New evidence resets the rest of the landing budget but not this count.

Fix failing checks. When the pull request's required checks fail for the head that would land, the land step collects each failed check's name, conclusion, details URL and the tail of its GitHub Actions job log (up to its last ##[error], at most 150 lines, 8 KB across all checks, scrubbed of credentials). The SAME task goes back to OPEN with reason required_checks_failed and a fix_failed_checks directive in its metadata; the executor prompt renders it as a "Sent back to fix failing checks" section with the logs as an escaped, untrusted data block. The next attempt pushes from a fresh lease branch, so when it lands the land step moves the ORIGINAL pull request's head branch to the new reviewed head (--force-with-lease) and lands through that pull request: its checks re-run there, instead of the hub waiting on a pull request whose head never changed. A closed or merged pull request is not reused. Send-backs are counted in metadata.landing.check_fixes and capped at three; the fourth failure blocks the task (landing_check_fix_cap_exhausted) with the last failure summary. A failure that cannot be sent back (no pull request, no worker evidence) blocks at once as landing_non_retryable.

Reopened attempts land through the same pull request. The agent's pull-request lookup reuses the task's open pull request by task id, so any later attempt (after mac task reopen or a review send-back) names a pull request whose head branch is still an earlier attempt's. The land step moves that head branch to the new reviewed head (--force-with-lease) and lands through it, exactly as for a check-fix attempt, and records a reopened_task_pull_request command. It moves only a pull request that is open, unmerged, on the canonical branch, and whose body carries this task's own - task: marker; any other reference keeps the hub's fallback of opening a pull request for the attempt's own branch.

Never double-land. Before merging, the land step reads the pull request's state. A PR already merged (by the forge, a human, or an attempt of ours that died after the merge) is observed and recorded as landed, not merged again.

The mode and its gate are recorded in the publication evidence as a merge_serialization command (mode: serial_land_loop, test_gate: required_checks | worker_verifier) and on the canonical-integration proof.

mac's native speculative merge queue was removed on 2026-10-01 (it landed 4.7% of its entries); migration 0005_drop_native_merge_queue_tables drops its merge_queue_entries and merge_queue_windows tables. Approved tasks that were waiting on it are still reviewing and land through the loop on the next tick.

Checks still running. A merge the forge refuses because its own gates have not finished is not a failure. Publication defers with publication_failure_kind=pull_request_checks_pending and the existing publication-retry backoff re-attempts later; the pull request is reused rather than reopened, so retries are cheap. The task stays in needs_review (or a legacy reviewing) — approved but not completed — until the change is genuinely on the canonical branch.

Branches that must be up to date. A ruleset with "require branches to be up to date before merging" (strict_required_status_checks_policy, read by gitops.required_status_check_policy) will not merge a pull request that is behind the base, and nothing else updates it — it used to wait out the landing deadline. When the forge reports the PR behind (or the canonical tip is not in its head), the land step calls GitHub's update-branch (gitops.update_pull_request_branch, pinned to the observed head) and waits for the required checks on the updated head (publication_failure_kind=pull_request_branch_updated, charged to the deadline only). The updated head — the reviewed head plus merges of the canonical branch, nothing else — is what the checks verify and the merge is pinned to. If the forge reports a conflict, the task is sent back to rebase.

Squash and the integration proof. A squash merge lands the reviewed content under a new SHA, so the reviewed commit is deliberately not an ancestor of the canonical tip afterwards. The canonical-integration proof records this honestly (squash_merged: true, contains_reviewed_head: false, plus the pull-request number and URL) instead of asserting an ancestry that squashing destroys, and task completion accepts that proof shape.

Opting out. MAC_PUBLICATION_STRATEGY selects the strategy:

value	behaviour
`pull_request` (default)	push the branch, open a PR, let the forge squash-merge it
`direct_push`	the legacy path: fast-forward the canonical branch to the verified head in a disposable clone and push it under optimistic concurrency (`push_main_occ`)

Repositories with no forge. mac manages repositories that have no pull requests at all — a bare path, a file:// remote, or an http(s) remote for which no GH_TOKEN/GITEA_TOKEN is configured. Those fall back to direct_push automatically rather than failing, because refusing to publish them would strand the work. The fallback is never silent: the publication evidence carries a publication_strategy entry naming the strategy and the reason it was chosen, so a repository that should be using a pull request but is not shows up in mac task show's publication detail.

Migration. Nothing is stranded by the change. Publication is stateless per attempt (a fresh clone every time) and the worker side already pushes only its own branch, so a task that was mid-publication under the old path simply gets a pull request on its next attempt against the branch it had already pushed.

Repository credential learning and reviewer routing¶

Every completed remote review preparation writes an authoritative fleet_learning:repository_access memory record. The JSON content uses schema mac.fleet_learning.v1 and records only routing-safe facts: project, repository host, operation, agent, credential source name, outcome, failure class, bounded redacted error signature, recommendation, and task/review IDs. It never stores a credential value or authenticated URL.

For the task's repository host, a reviewer assigned through the review API is checked against these records (the default review workflow approves from worker evidence and selects no reviewer agent). An agent whose newest matching record is an authentication or authorization failure is ineligible during the configured cooldown. A newer success restores eligibility immediately; cooldown expiry returns the agent to unknown status. This lookup reads the authoritative PostgreSQL ledger directly, so routing changes immediately and does not wait for Qdrant vector backfill.

Review Jobs receive optional Git-host keys from the runner's configured Secret (default mac-api-config): GH_TOKEN, GITHUB_TOKEN, GITEA_TOKEN, and GITEA_USER. Missing keys are allowed for public repositories. Review Jobs do not receive MAC_SECRET_KEY. Host-mode workers also recognize MAC_TASK_GIT_TOKEN as a fallback. The worker injects an HTTPS token only into the Git command and restores the checkout's origin to the clean remote URL.

The pushed-ref evidence check uses the same environment-backed resolver. An authentication, authorization, or network failure on the control-plane host is indeterminate: it does not prove that the pushed ref is absent. A successful lookup with no matching ref still rejects phantom-push evidence, and the review verdict must still verify the executor's work.

Inspect the routing inputs and resulting review state:

mac --json admin memory search \
  --record-type fleet_learning:repository_access \
  --order desc --limit 50

mac --json admin memory search \
  --subject-type agent --subject-id agent_... \
  --record-type fleet_learning:repository_access \
  --order desc --limit 20

mac --json task show task_...

Do not hand-edit or invent success records to force routing. Repair the agent's credential, run a real repository operation, and let the resulting success supersede the failure. Multiple success records can exist when a review is retried; only the newest matching record controls eligibility.

Every mac-managed subprocess records a short-retention command audit event on the hub. The log captures command_id, agent, task, sanitized argv, cwd, start/end timestamps, return code, output byte counts, and output hashes. The default retention is 24 hours (MAC_COMMAND_AUDIT_RETENTION_SECONDS), and the latest rows are visible from /command-audit and the dashboard Observability view. This is operational telemetry for proving agents are doing work; a future security audit store can consume the same event shape externally.

The command audit is not a durable compliance archive. Its current job is to make the last day of worker behavior visible without relying on local shell history or unbounded per-host logs.

Executor success is not completion. A zero return code only means the executor reported without crashing. For the default workflow to auto-approve and publish, the evidence metadata must include a mac.worker_evidence.v1 verification manifest. Repo/code work must include a pushed git artifact (head_sha, remote_ref or PR URL, pushed=true, dirty=false, changed files) plus at least one passing test/check. Documentation or investigation work must include a pushed repo artifact or explicit artifacts plus passing checks. Deployment work must include deployment targets/services and passing checks. Thin reports, local-only diffs, missing manifests, failing checks, or unverifiable claims stay in needs_review/reviewing for manual handling.

During execution, the worker renews its active task lease. A successful renewal also refreshes the owning agent's last_seen_at, keeps it busy, and preserves current_task_id, so long-running work remains visible as live without allowing an invalid idle heartbeat while the lease is active.

For already-migrated or pre-upgrade rows that are stuck in needs_review, run the backlog tick against the hub:

curl -X POST 'http://hub.example.internal:8789/reviews/default/tick?limit=100&actor=operator'

Before enabling executor-backed claiming, use dry-run mode to record routing candidates without creating leases:

MAC_DEPLOY_WORKER_MODE=dry-run
MAC_DEPLOY_WORKER_CLAIM_ONLY_CANARY_TASKS=1
MAC_DEPLOY_WORKER_ALLOWED_PROJECTS=mac-canary

Dry-run mode emits worker.routing.policy, worker.routing.dry_run_candidate, or worker.routing.no_candidate events into /observability and /observability/stream. It must show only synthetic canary work before loop mode is enabled.

To enable executor-backed claiming from deploy config, set:

MAC_DEPLOY_WORKER_MODE=loop
MAC_DEPLOY_WORKER_CAPABILITIES=ops,python,hermes,review
MAC_DEPLOY_WORKER_CLAIM_ONLY_CANARY_TASKS=1
MAC_DEPLOY_WORKER_ALLOWED_PROJECTS=mac-canary

The generated service then runs mac-agent --register --loop against MAC_HUB_URL, using MAC_WORKER_TOKEN from ~/.mac/mac.env. The default executor wrapper is ~/.mac/bin/mac-hermes-task-executor, which calls the deployed upstream Hermes checkout in one-shot mode. The runtime proof treats that executable as a required session capability, so a deployed agent is not considered ready for Codex-like task work unless the executor path is present and executable.

Avoid printing the mac-agent process command line. On Linux, report mac-agent.service with systemctl show summary fields instead of systemctl status, because the service wrapper may pass the worker token to mac-agent as process argv. Logs should therefore show service state, PID, and restart count, but not the bearer token. Operators should continue to treat host-level process inspection as privileged access.

Workers advertise review by default so the default review workflow can pick real second-eye reviewers. During registration the worker persists its attestation key into ~/.mac/mac.env; if an older deploy missed that one-time key, the service rotates a replacement before it signs new evidence. Rotation is explicit recovery behavior and invalidates old signatures from that agent.

Loop mode claims ordinary eligible work by default. Set MAC_DEPLOY_WORKER_CLAIM_ONLY_CANARY_TASKS=1 only for a temporary qualification lane in which the worker must reject every task that is not explicitly canary-marked. Clear it immediately after qualification.

Legacy Beads Migration And Repository Registry¶

The live production issue authority is the MAC task ledger. Beads/Dolt round-trip sync is not part of the normal deployment path, and operators should not run bd for current MAC task lifecycle. Legacy Beads state is handled as a one-time import path:

mac task detect-beads <repo>
mac task migrate-beads <repo> --project <project>

Use --tickets-only only when creating local compatibility files during a migration audit. .tickets/<id>.md is ignored local operational state; it is not checked into this repository and is not a cross-host ledger.

Repository-backed execution uses the project repository registry instead of a legacy bridge poller. Register or onboard repositories through the current project/repository commands:

mac project register                                  # current checkout; creates contract task
mac project register <path-or-git-url>[#branch]       # local or remote-first onboarding
mac project update <project> --branch <branch>         # move the internal project fork
mac project unregister <project> --force               # detach history; disable checkout registration
mac admin bridge repository register <name> <path> --project <project>  # after .mac/project.yaml exists
mac admin bridge repository repos

Every registered repository must include a repository runtime contract at .mac/project.yaml. Registration and task creation use that contract to attach the bootstrap command, canonical test command, supported host families, canonical remote URL, and required evidence to new tasks. See Repository Runtime Contract.

The hub's repository-ref reconciler uses this registry as its complete workset. The hub therefore needs filesystem and GitHub access to each enabled checkout; automatic prune additionally requires the contract's canonical_remote_url. Monitor it with mac admin repo refs status and trigger a one-shot audit with mac admin repo refs reconcile --mode audit. See Managed Repository Ref Hygiene.

The hub advances the default review/publication workflow from heartbeat when MAC_REVIEW_TICK_ON_HEARTBEAT=1 and MAC_REVIEW_TICK_HUB_AGENT is set to the configured hub agent. The tick only moves tasks when required evidence, reviewer verdicts, and publication targets are present; otherwise it records explicit waiting reasons in observability.

AgentBus¶

/messages remains the constrained structured control bus and still rejects execution-shaped payloads. Agent-to-agent content exchange uses /agentbus instead:

POST /agentbus/streams opens a typed stream with content_type, topic, optional task linkage, and optional recipient.
POST /agentbus/streams/{id}/chunks appends ordered JSON, text, or base64 chunks. final=true closes the stream atomically after the chunk.
GET /agentbus/streams/{id}/chunks reads durable chunks after a sequence.
GET /agentbus/streams/{id}/events tails chunks as NDJSON for streaming consumers.

This is a typed transport channel, not an execution channel. Agents must still turn received content into explicit task/evidence/review actions through the normal API.

Roles and Workflows¶

Production mac includes an API-level organization model for coordinated work:

/roles stores the role catalog used to describe agent jobs, prompts, required/default capabilities, optional hardware requirements, and tenant scope. /roles/seed loads the built-in Loom-style role set.
/agents/{id}/role assigns a role to a registered agent. If the agent is bound to a Hermes persona, role assignment respects that persona's allowlist.
/workflows stores versioned DAG definitions. /workflows/import-yaml and /workflows/seed provide operator-friendly loading paths.
/workflows/{id}/start, /workflows/runs, and /workflows/runs/tick run and sweep workflows. Each node creates a normal mac task, so dispatch, evidence, review, publication, command audit, and Beads ledger behavior stay the same as single-task work.

/dashboard/state includes workflow-run summary data for UI clients. Full visual workflow authoring is still an operator-facing gap in the checked-in dashboard; use the API/CLI for workflow creation and editing until that UI is built.

Docker Engine / Moby¶

Docker on a darwin hub is supported for system services, including Qdrant, Firecrawl, test PostgreSQL, and telemetry collectors. It is never an OpenShell execution runtime on darwin, and its availability does not alter the node's macos_host posture. docker ps should therefore show only the expected service containers; investigate any container presented as a task sandbox.

docker build -t mac:latest .

docker run -d --name mac \
    -e MAC_SECRET_KEY="$(openssl rand -base64 48)" \
    -e MAC_API_TOKEN="$(openssl rand -hex 32)" \
    -v mac-data:/var/lib/mac \
    -p 127.0.0.1:8789:8789 \
    --restart unless-stopped \
    mac:latest

# Healthcheck is built into the image; `docker ps` shows (healthy) once up.
curl -fsS http://127.0.0.1:8789/health

For Kubernetes, ship the same image as a single-replica Deployment with a PVC mounted at /var/lib/mac. Use a ConfigMap for non-secret env and a Secret for MAC_SECRET_KEY / MAC_API_TOKEN.

Backups¶

Use PostgreSQL backup tooling against the configured authority. Keep credentials in a protected PostgreSQL service/password file or the process environment. A custom-format dump supports a deliberate restore into a separate target:

pg_dump --format=custom --file=/backups/mac.dump
pg_restore --list /backups/mac.dump

Set the connection environment for the intended database first. A dump listing is an inventory check, not a restore drill. Verify a restore into an isolated database, including the ordered migration ledger and application reads, before relying on a backup. Copying a data directory from a running server is not a supported application backup procedure.

Observability¶

GET /health is the liveness/readiness signal. Returns {"status":"ok"}.
GET /startup/hermes returns the redacted Hermes startup report: file existence/size/mtime metadata, warning strings, and Slack activation status.
GET /events is the unified audit stream — point a log shipper (vector, promtail, fluent-bit) at it with since= advancing every poll, or scrape the PostgreSQL tables through an authorized read connection.
mac admin events list --since <iso> is the operator's one-shot "what just happened" query.
POST /observability/metrics and POST /observability/logs ingest layer/source/name/level observations from workers, Hermes adapters, deploy scripts, and external monitors. POST requires agent scope when API tokens are enabled.
GET /observability lists persisted observations; GET /observability/summary returns dashboard aggregates and latest metric snapshots; GET /observability/stream tails observations as NDJSON for live dashboards or collectors.
GET /notifications lists the durable operator notification outbox for task lifecycle, review, publication, and bridge-stale events. POST /notifications/{id}/delivered marks entries delivered, failed, or skipped.

The FastAPI middleware records per-request http.request.duration_ms metrics and http.request logs. Control-plane task, agent, project, fleet, secret, environment, rollout, and eval events are mirrored into the observability stream with their original subject ids. The dashboard Observability tab uses URL-addressable filters, the summary endpoint, and an NDJSON subscription to visualize the live stream, unified events, command audit, and the operator notification outbox.

Upgrade procedure¶

Use scripts/fleet-update. It takes a verified PostgreSQL backup and applies the ordered schema migrations before the new hub starts; startup verifies rather than silently changing schema. Preserve the prior diagnostic artifacts.

Keep failed candidates and their affected workers held for diagnosis, then repair forward. Restoring an older source or database is an explicit break-glass operation, not an automatic response to a failed check. A release artifact alone does not authorize bypassing the cohort's fault-matrix gates.

Troubleshooting¶

Deploy aborts: "Qdrant did not become ready at :6333" (high-core nodes)¶

On a node where the container's CPU cgroup quota is small but the host exposes many cores (a common Kubernetes shape — e.g. a 4-CPU pod on a 192-core node), Qdrant sizes its actix/tokio thread pools from /proc/cpuinfo (the full node core count, not the cgroup quota) and tries to spawn far more threads than the container's PID cap allows. Thread creation then fails with EAGAIN and Qdrant panics at startup:

ERROR qdrant::startup: Panic ... Cannot spawn Arbiter's thread:
  "actix-rt|system:0|arbiter:NNN": Os { code: 11, kind: WouldBlock,
  message: "Resource temporarily unavailable" }

mac-qdrant-run (or the systemd unit) exits immediately, supervisord/systemd marks the program FATAL, and the deploy aborts because Qdrant is mandatory on the hub. The fix ships by default: the container PID cap is now QDRANT_PIDS_LIMIT (default 4096) instead of a hardcoded 512. If an even larger node still hits this, raise it in the fleet env, e.g. QDRANT_PIDS_LIMIT=8192, and redeploy. Quick check on the affected host: --pids-limit=512 → panic / curl :6333/collections returns 000; raised limit → 200.

supervisord pods: `supervisorctl` permission errors¶

On pods where supervisord runs as root (PID 1) with a root-only control socket (srwx------), supervisorctl as the deploy user gets PermissionError ... xmlrpc.py. The deploy already prefers sudo supervisorctl (run_supervisorctl); ensure the deploy user has passwordless sudo. A bare supervisorctl ... in a manual session needs sudo.

Private repository review is retracted or remains in `reviewing`¶

Start with the task and shared learning records:

mac --json task show task_...
mac --json admin memory search \
  --record-type fleet_learning:repository_access \
  --order desc --limit 50

Interpret the review reason before retrying:

reviewer_unavailable:reviewer_repository_access_authentication:<host> means MAC learned that the selected reviewer could not authenticate, retracted it, and will prefer a recent successful peer. Repair that agent's host-specific token before expecting it to be selected during the cooldown.
reviewer_unavailable:reviewer_repository_access_authorization:<host> means the credential was recognized but lacks access to the repository or organization. For GitHub organizations using SSO, authorize the token for the organization rather than rotating it blindly.
reviewer_unable_to_produce_verdict_after_<N>_attempts means repository access may have succeeded but the reviewer executor did not emit a valid signed verdict. Inspect the review evidence and reviewer workspace; changing Git credentials is not the appropriate repair unless the memory record also reports an access failure.

The delivered-nudge cap prevents an unparseable or crashing reviewer from spinning forever. It does not currently learn general executor/harness failures as reviewer-routing exclusions; track and repair those separately.

Per-task model pins¶

Per task, --model <name> pins a model by name. --model-strength 1..10 is still accepted and recorded in task metadata for compatibility, but it is advisory only: no strength ladder resolves it, so the task runs on the fleet default model unless --model also pins one.

Known limitations¶

Every topology requires PostgreSQL. Multi-replica deployments share one schema authority and must coordinate migrations before service startup.
No built-in TLS. Put a reverse proxy in front.
MAC_SECRET_KEY rotation is manual.
Operational success-first routing is currently enforced for repository access during autonomous review. General executor, sandbox, and verdict-production failures are bounded and visible but are not yet learned as reviewer-routing exclusions.
Existing SQLite authorities migrate offline with mac admin database migrate-sqlite-to-postgres. Stop every hub writer and worker supervisor first, then pass --hub-stopped; the command checkpoints WAL, holds an exclusive SQLite snapshot, requires an empty PostgreSQL target, copies each table atomically, and writes a resumable owner-only receipt. Completion requires matching deterministic full-row hashes for every table, a clean source integrity/FK check, clean destination FK checks, and reset PostgreSQL sequences. A count-only copy is never reported as successful. Example:

$ export MAC_MIGRATION_DATABASE_URL='postgresql:///mac'
$ mac admin database migrate-sqlite-to-postgres \
    --sqlite ~/.mac/mac.db \
    --report ~/.mac/migrations/sqlite-to-postgres.json \
    --hub-stopped

Use --resume only with that same receipt/source/target, or --verify-only for a later full verification. --restart explicitly truncates the target and is the only supported way to abandon a partial target. After a verified receipt, set MAC_DATABASE_URL, remove MAC_DB, restart the hub, and retain the SQLite file only as an immutable rollback/export artifact.