Production Deployment¶
Three supported topologies:
- Single host, systemd — one machine, one PostgreSQL database, one FastAPI
process. Suitable for dev fleets, personal Hermes runtimes, and pilot
deployments. See
deploy/systemd/. - Containerized, single-instance — image at
Dockerfile. Same PostgreSQL topology, but lifecycle is managed by Docker Engine/Moby or k8s as a single-replica deployment. See the container section below. - Kubernetes, multi-replica, Postgres-backed — stateless
mac-apiDeployment in front of an externally-managed Postgres 17 cluster. Multiplemac-apireplicas share the same durable state viaMAC_DATABASE_URL. The cluster itself (CloudNativePG, RDS, Cloud SQL, vendor-managed, etc.) is provisioned outside this repo and its DSN is supplied via themac-api-configSecret. Seedeploy/k8s/README.mdanddocs/archive/field-notes/k8s-native-rewrite-plan.md.
PostgreSQL is the only supported control-plane authority in every topology.
Set MAC_DATABASE_URL to an explicit PostgreSQL DSN. The legacy MAC_DB
variable accepts a PostgreSQL DSN too; a SQLite path is rejected. There is no
implicit home-directory database or client-side replica.
Ordinary service startup verifies the ordered schema-migration ledger and
performs no DDL. Bootstrap and upgrades run mac-schema-migrate explicitly
before starting the candidate service; see the
versioned schema decision.
Only a hub or stateless API replica is a control-plane server. Fleet spokes are
clients: their generated environment has MAC_CONTROL_PLANE_ROLE=client, no
MAC_DB or MAC_DATABASE_URL, and their worker/gateway processes use
MAC_HUB_URL. A redeployed spoke archives an inactive legacy mac.db; active
legacy tasks block deployment until explicitly migrated to the hub. Store
construction also rejects MAC_CONTROL_PLANE_ROLE=client even if a stale
database variable is reintroduced.
Required configuration¶
| Variable | Required | Purpose |
|---|---|---|
MAC_SECRET_KEY |
yes | 32+ char secret; HKDF input for the Fernet key that encrypts secret values. Refuses to start without it. |
MAC_CONTROL_PLANE_ROLE |
yes for fleet deploys | hub for the single database-owning authority; client for database-free spokes. |
MAC_DATABASE_URL |
conditional | PostgreSQL DSN (postgresql://... or postgres://...). Preferred authority setting; overrides MAC_DB. Startup verifies the migration ledger; deploy applies migrations explicitly. |
MAC_PG_POOL_SIZE |
no | psycopg_pool max connections per mac-api replica. Default 10. |
MAC_DB |
conditional | Legacy name for an explicit PostgreSQL DSN. SQLite paths are rejected; no private client ledger is created. |
MAC_API_TOKEN |
no | Single admin bearer token. Set empty string is rejected. |
MAC_API_TOKENS |
no | JSON {token: [scopes,...]} for scoped auth. Mutually exclusive with MAC_API_TOKEN. |
HERMES_HOME |
no | Hermes state directory checked at startup. Default ~/.hermes. |
MAC_HERMES_AGENT_DIR |
no | Hermes checkout inspected for the slack_accounts.json activation shim. Falls back to HERMES_AGENT_DIR, then ~/Src/hermes-agent if present. |
MAC_HERMES_APPLY_SLACK_ACCOUNT_SHIM |
no | Set 0 to disable startup patching of an explicit MAC_HERMES_AGENT_DIR. Default enabled only when the checkout path is explicit. |
MAC_HERMES_APPLY_GATEWAY_RUNTIME_SHIM |
no | Set 0 to disable startup patching of Hermes gateway model/runtime overrides. Default enabled for explicit checkout paths. |
MAC_HERMES_GATEWAY_MODEL |
no | Per-agent model used by Hermes gateway conversations and mirrored to HERMES_INFERENCE_MODEL for oneshot worker execution. |
MAC_HERMES_GATEWAY_PROVIDER |
no | Runtime provider for the per-agent model. Fleet deploy normally uses custom so Hermes sends OpenAI-compatible requests through MAC's router. |
MAC_HERMES_GATEWAY_BASE_URL |
no | OpenAI-compatible base URL for Hermes. Fleet deploy writes the hub's local in-mac /v1 endpoint on the hub and the hub or wing-router /v1 endpoint on spokes. |
MAC_HERMES_STARTUP_CHECK |
no | Set 0 to disable Hermes state and Slack startup checks. Enabled by default. |
MAC_REQUIRE_HERMES_STARTUP_READY |
no | Set 1 to fail mac startup when Hermes soul/memory/state references or Slack activation are not ready. |
MAC_HERMES_SLACK_HOME_CHANNEL_NAME |
no | Slack home-channel name, without #, used to write ~/.hermes/slack_home_channels.json from slack_accounts.json. Empty skips discovery. |
MAC_HERMES_SYNC_SLACK_HOME_CHANNELS |
no | Set 0 to preserve existing Slack home-channel files without discovery. Default enabled. |
MAC_URL / MAC_HUB_URL |
no | MAC API endpoint used by Hermes-side mac-hermes operations. Fleet deploy points this at the hub. |
MAC_CLIENT_PRINCIPALS_FILE |
no | Hub-local hashed registry for scoped client enrollment. Fleet deploy sets $MAC_HOME/client-principals.json; permissions must be 0600. The API hot-reloads issuance and revocation. |
MAC_HERMES_INSTANCE_ID |
no | Hermes instance id for this runtime. Fleet deploy uses a deterministic hermes_<agent> id and registers it in MAC. |
MAC_WORKER_HERMES_INSTANCE_ID |
no | Worker agent binding to the Hermes instance id. This keeps MAC agent rows linked to their Hermes soul/runtime. |
MAC_AGENT_ID |
no | Deterministic MAC agent id for this runtime. Fleet deploy uses agent_<agent>. |
MAC_HERMES_RUNTIME_CONTEXT_FILE |
no | Hermes-visible task/project runtime contract JSON. Default ~/.hermes/mac-runtime-context.json. |
MAC_HERMES_RUNTIME_CONTEXT_MARKDOWN |
no | Human/agent-readable runtime contract summary. Default ~/.hermes/mac-runtime-context.md. |
MAC_HERMES_RUNTIME_CONTEXT_REQUIRED |
no | Set 1 to make startup readiness fail if the MAC task/project runtime contract is missing, invalid, or not injected into the Hermes prompt builder. Fleet deploy enables this. |
MAC_HERMES_WORKSPACE |
no | Source workspace Hermes should treat as equivalent to an operator/Codex shell in the MAC repo. Fleet deploy sets this to $MAC_HOME/src/mac. |
MAC_PROJECT_CONTRACT_FILE |
no | Repository contract file for the Hermes direct-session capability bridge. Fleet deploy sets this to $MAC_HERMES_WORKSPACE/.mac/project.yaml. |
MAC_WORKER_EXECUTOR |
no | Executor command used by loop-mode workers. The default ~/.mac/bin/mac-hermes-task-executor is part of the Hermes direct-session capability proof. |
GH_TOKEN / GITHUB_TOKEN |
no | GitHub HTTPS credential used by task, review, publication, and pushed-ref verification commands. The credential may appear only in the individual Git command and must not persist in origin, evidence, logs, or memory. |
GITEA_TOKEN |
no | Gitea HTTPS credential for the same Git operations. MAC_TASK_GIT_TOKEN is the host-mode fallback when no host-specific token is set. |
MAC_DEPLOY_GH_TOKEN |
no | Fleet-deploy input copied into the managed runtime as GH_TOKEN. Keep it in the host-local ~/.mac/.env, never in fleets.yaml or a committed spec. |
MAC_DEPLOY_GATEWAY_PROBE_FATAL |
no | Set 1 to make a failed OpenClaw gateway/channel probe fail the node, and therefore the cohort. Default 0: the failure is recorded, the failed successor is retained for diagnosis, and the deploy continues, because task execution is OpenShell plus the coding CLI plus mac-agent and none of them consult chat. Set it only for a deploy whose purpose is to prove the chat surface. |
MAC_REPOSITORY_REF_RECONCILER_MODE |
no | Managed task-branch reconciler mode: off, audit, or prune. Runtime default off; fleet deployment defaults the hub to prune and spokes to off. |
MAC_REPOSITORY_REF_RECONCILER_INTERVAL_SECONDS |
no | Delay between automatic passes, bounded from 60 through 604800. Hub default 86400 (daily). |
MAC_REPOSITORY_REF_RECONCILER_INITIAL_DELAY_SECONDS |
no | Delay before the first automatic pass, bounded from 0 through 86400. Hub default 300. |
MAC_REPOSITORY_REF_RECONCILER_GRACE_DAYS |
no | Fallback cleanup grace for legacy lifecycle records, bounded from 0 through 365. Default 7. |
MAC_REPOSITORY_REF_RECONCILER_REMOTE / MAC_REPOSITORY_REF_RECONCILER_BASE_REF |
no | Git remote name (default origin) and optional explicit <remote>/<branch> ancestry target. Without a base override, the remote HEAD is auto-detected. |
MAC_REPOSITORY_ACCESS_FAILURE_COOLDOWN_SECONDS |
no | How long a newest authentication/authorization failure excludes a reviewer for the matching project, repository host, and operation. Default 1800. |
MAC_REPOSITORY_ACCESS_SUCCESS_TTL_SECONDS |
no | How long a successful repository-access learning receives reviewer-selection preference. Default 86400. |
MAC_REVIEW_NUDGE_MAX_ATTEMPTS |
no | Maximum durable delivered verdict nudges for one review before it is retracted. Default 10. |
MAC_SUPERVISOR_KIND |
no | Runtime supervisor selected by fleet deploy: systemd, launchd, or supervisord. |
MAC_MEMORY_TOPOLOGY_FILE |
no | Hermes-visible memory topology JSON. Default ~/.hermes/mac-memory-topology.json. |
MAC_SHARED_SERVICES_MANAGER_AGENT |
no | Agent that owns hub-managed shared services. Defaults to the configured fleet hub. |
QDRANT_URL / QDRANT_ADDRESS / QDRANT_FLEET_URL |
no | Shared Qdrant level-2 memory endpoint. When set, Hermes startup readiness validates /collections. |
MAC_REQUIRE_QDRANT_MEMORY |
no | Set 1 to require shared Qdrant memory readiness. Fleet deploy enables this by default. |
MAC_QDRANT_MEMORY_ALLOW_DEGRADED |
no | Temporary operator override that allows startup when required Qdrant is missing or unreachable. |
QDRANT_PIDS_LIMIT |
no | Container PID/thread cap for the hub-managed Qdrant (supervisord wrapper + systemd unit). Default 4096. Raise on very-high-core nodes; see Troubleshooting. |
Generate a secret key once:
Store it in a secrets manager. Rotating the key requires re-encrypting all secret values; today this is a manual procedure (re-emit every secret with the new key).
Environment variable precedence¶
MAC resolves each MAC_* variable according to a three-level contract:
- Process environment wins. A variable already present in the invoking process environment (e.g. set by the supervisor unit, injected by a secret manager, or exported by the operator shell) is used as-is and is never overridden by the env file.
- Env file supplies defaults. A variable absent from the process environment receives its value from the operator env file (typically
~/.mac/.envor the path given byMAC_ENV_FILE). - Env file is the operator default store. Operators should record stable deployment values — tokens, URLs, feature flags — in the env file. Runtime overrides belong in the process environment and are not written back to the file.
See env-config-reference.md for the full variable catalog.
Systemd¶
# 1. Create the service user and data directory.
sudo groupadd --system mac
sudo useradd --system --gid mac --home-dir /var/lib/mac \
--shell /usr/sbin/nologin mac
sudo install -d -o mac -g mac -m 0750 /var/lib/mac
# 2. Install mac globally (or into a venv at /usr/local/lib/mac).
sudo pip install /path/to/mac-0.1.0-py3-none-any.whl
# 3. Write the env file (mode 0600, owner root:mac).
sudo install -d -o root -g mac -m 0750 /etc/mac
sudo install -o root -g mac -m 0640 deploy/systemd/mac.env.example /etc/mac/mac.env
sudo $EDITOR /etc/mac/mac.env # set MAC_SECRET_KEY, optionally MAC_API_TOKEN
# 4. Install and start the unit.
sudo install -o root -g root -m 0644 deploy/systemd/mac.service \
/etc/systemd/system/mac.service
sudo systemctl daemon-reload
sudo systemctl enable --now mac.service
# 5. Verify.
sudo systemctl status mac.service
curl -fsS http://127.0.0.1:8789/health
The unit is started with python -m mac.hub_serve, which reads
${MAC_BIND_HOST} and ${MAC_PORT} from /etc/mac/mac.env. Worker agents
bind 127.0.0.1. Mesh hubs (Tailscale/Headscale) bind loopback plus the
CGNAT address from tailscale ip -4 and refuse 0.0.0.0, so the API cannot
accidentally listen on a public NIC. The overlay is the transport; it is not
a host firewall unless the process also refuses other interfaces. In-cluster
images with no mesh provider may still bind 0.0.0.0 inside the container
network namespace. Put a TLS-terminating reverse proxy in front only when
the hub must be reached off-mesh.
Fleet Setup Wizard¶
First-time deployments should use the setup wizard instead of hand-editing deployment YAML:
The wizard asks for the hub, agents, SSH targets, OS families, supervisors, Slack home channel, per-agent Hermes model selectors, worker mode, canary policy, Qdrant shared-memory endpoint, fleet network provider, and optional hub token. It writes:
~/.mac/fleets.yaml: home-scoped multi-fleet topology, keyed by hub node.~/.mac/.env: caller-machine deploy settings and local secrets, mode 0600.
To deploy after the wizard:
Declarative Setup For Agents¶
LLM-driven setup should prefer a spec file over the interactive wizard. The setup spec is validated before files are written, and the doctor output lists missing env vars and next commands in machine-readable JSON.
Rather than hand-writing a spec, start from a generic, per-CSP sample. The repo
ships de-personalized samples under deploy/fleet/samples/ (GKE is the worked
example); a real, named fleet spec lives outside git in
~/.mac/specs/<fleet>.fleet.yaml, created at install time by copying and
customizing a sample. Never check a named fleet into the repo.
scripts/setup-fleet.py --list-samples # browse per-CSP samples
scripts/setup-fleet.py --init-from gke --name my-gke # -> ~/.mac/specs/my-gke.fleet.yaml
$EDITOR ~/.mac/specs/my-gke.fleet.yaml # fill in the <placeholders>
make setup ARGS="--spec ~/.mac/specs/my-gke.fleet.yaml --force"
See deploy/fleet/samples/README.md for the per-CSP convention and the knobs
that differ per cloud (bastion/ProxyJump, network provider, in-cluster vs
public DNS, supervisor).
Example fleet-setup.yaml:
schema: mac.fleet_setup.v1
fleet:
name: horde
hub: horde-hub
hub_url: http://horde-hub:8789
agents:
- name: horde-hub
target: ubuntu@10.0.0.10:2201
os: linux
model: nvidia/llama-3.3-nemotron-super-49b-v1
worker:
mode: loop
- name: horde-worker
target: ubuntu@10.0.0.11
os: linux
router:
backend: inproc
providers:
- id: nvidia
key_env: NVIDIA_API_KEY
network:
provider: tailscale
Recommended LLM flow:
export NVIDIA_API_KEY=...
mac admin fleet validate --spec fleet-setup.yaml
mac admin fleet doctor --spec fleet-setup.yaml
make setup ARGS="--spec fleet-setup.yaml --force"
make setup ARGS="--spec ..." writes ~/.mac/fleets.yaml and ~/.mac/.env,
then deploys the generated plan. To configure only:
If a provider key such as NVIDIA_API_KEY is absent from both the environment
and the spec, validation fails before deployment so the fleet cannot silently
come up without chat routing.
The checked-in deploy/fleet/config.yaml is a generic sample only. It is
marked sample: true, and deploy/deploy-mac-fleet.sh refuses to deploy from
it unless MAC_DEPLOY_ALLOW_SAMPLE_CONFIG=1 is set explicitly for tests.
Reaching the Hub Node¶
The hub control plane binds to hub_url as declared in ~/.mac/fleets.yaml.
How you reach it from a client machine depends on the network topology.
Client bootstrap:
mac admin loginnow combines strict host-key verification, a managed local tunnel, hub-local scoped enrollment, authenticated validation, and atomic profile installation. Do not copy the hub's~/.mac, admin token,mac.db,MAC_SECRET_KEY, provider keys, or SSH private keys to a new client.
For a directly reachable hub, the interim client setup is:
export MAC_API_URL=https://mac.example.internal
export MAC_API_TOKEN=<scoped-client-token>
mac admin diagnostics
mac task stats
mac agent list
Use a mode-0600 env file or a process environment supplied by a secret
manager. --token exists for automation/recovery but is not the preferred
interactive path because argv can be visible in shell history and process
inspection.
Direct access (same network or VPN)¶
Hub is directly routable — no tunnel needed:
SSH port forward and scoped enrollment¶
Hub lives behind a bastion or inside a K8s cluster. Record the route in the home-scoped fleet registry so deploy, credential recovery, snapshots, migration, and Electron use the same resolver:
# ~/.mac/fleets.yaml
fleets:
my-fleet:
hub_agent: hub
control_port: 8789
defaults:
ssh_jump: horde@bastion.example.com:2222
identity_file: ~/.ssh/mac-my-fleet
ssh_known_hosts_file: ~/.ssh/mac-my-fleet-known-hosts
ssh_host_key_policy: strict
agents:
- name: hub
target: horde@my-hub.cluster.local
os: linux
Provision and verify the hub and bastion host keys in ~/.ssh/known_hosts
before opening the tunnel. Disabling host-key verification is not a supported
bootstrap substitute.
The normal client path is one command:
mac admin login --fleet my-fleet --profile my-fleet --client-id my-laptop
mac admin login status --profile my-fleet
mac task stats
mac agent list
For a reproducible production profile the fleet route should resolve an explicit
identity file and a verified known-hosts/host-CA source; login validates and
pins whatever is supplied. An equivalent route can be given without a fleet
registry using --ssh, --identity-file, --known-hosts-file, optional
--ssh-port, and optional --proxy-jump. A directly reachable hub may use
--host-key-fingerprint SHA256:...; proxy-jump routes require a prepared
known-hosts file. Anything omitted falls back to OpenSSH's own resolution
(default identities/agent, ~/.ssh/known_hosts with accept-new), so
mac admin login --ssh <host> works like ssh <host> — but such a profile then
depends on the enrolling machine's ambient ssh state rather than being fully
portable.
The commands below remain the low-level recovery procedure when diagnosing SSH or enrollment independently.
# Confirm the route is portable and does not depend on ~/.ssh/config.
mac admin fleet ssh-spec --fleet my-fleet --agent hub --portable --json
# Forward hub control port to localhost and open the UI
ssh -N -F /dev/null \
-o BatchMode=yes \
-o StrictHostKeyChecking=yes \
-o UserKnownHostsFile="$HOME/.ssh/mac-my-fleet-known-hosts" \
-o ProxyJump=horde@bastion.example.com:2222 \
-i "$HOME/.ssh/mac-my-fleet" \
-L 8789:127.0.0.1:8789 \
horde@my-hub.cluster.local
# then: curl http://localhost:8789/health
# or: open http://localhost:8789/ui
With that tunnel open, use a second verified SSH session to invoke enrollment locally on the hub and stream the one-time token directly into the client profile store:
ssh -T -F /dev/null \
-o BatchMode=yes \
-o StrictHostKeyChecking=yes \
-o UserKnownHostsFile="$HOME/.ssh/mac-my-fleet-known-hosts" \
-o ProxyJump=horde@bastion.example.com:2222 \
-i "$HOME/.ssh/mac-my-fleet" \
horde@my-hub.cluster.local \
'mac --json admin client enroll my-laptop \
--fleet my-fleet --profile my-fleet \
--api-url http://127.0.0.1:8789 \
--scopes read,write,dispatch' \
| mac admin client profile install -
mac --profile my-fleet diagnostics
mac --profile my-fleet task stats
The API hot-reloads $MAC_HOME/client-principals.json, so the new credential
works immediately. For normal operation, renewal rotates and validates the
bearer before replacing the local credential; revoking logout invalidates it
before deleting local state:
The equivalent low-level recovery commands are:
ssh -T horde@my-hub.cluster.local 'mac --json admin client renew my-laptop' \
| mac admin client profile install - --profile my-fleet
ssh -T horde@my-hub.cluster.local 'mac admin client revoke my-laptop'
mac admin client profile remove my-fleet
Use the explicit SSH options above for renewal and revocation as well. The short form keeps the lifecycle example readable. See SSH Client Bootstrap Contracts for schemas, file modes, failure rules, and legacy migration.
Tailscale mesh (provider: tailscale)¶
Hub and spokes join the same Tailscale network. The hub is reachable at its Tailscale IP or MagicDNS name without any SSH tunnel:
# ~/.mac/fleets.yaml
defaults:
network:
provider: tailscale
tailscale:
auth_key_env: MAC_DEPLOY_TAILSCALE_AUTH_KEY
# Hub is reachable at its Tailscale IP, e.g. 100.x.x.x:8789
curl http://100.x.x.x:8789/health
make deploy HUB=<hub-node>
MAC_DEPLOY_TAILSCALE_AUTH_KEY must be set in ~/.mac/.env before deploy.
Headscale (self-hosted control plane, provider: headscale)¶
Headscale manages the WireGuard mesh. The fleet registry must declare the login server, DNS assumption, health URL, and pre-auth key source:
# ~/.mac/fleets.yaml
defaults:
network:
provider: headscale
headscale:
manage: false # true if mac should install/manage the headscale binary on the hub
login_server: https://headscale.example.com
health_url: https://headscale.example.com/health
preauth_key_source: env
preauth_key_env: MAC_DEPLOY_HEADSCALE_PREAUTHKEY
dns: magicdns
ip_prefix: "100.64.0.0/10"
# Hub reachable at its headscale-assigned IP or MagicDNS name
curl http://hub.headscale.example.com:8789/health
make deploy HUB=<hub-node>
MAC_DEPLOY_HEADSCALE_PREAUTHKEY must be set in ~/.mac/.env. With
headscale.manage: true the deploy script installs and configures the
headscale server on the hub node itself.
One-Time ACC Replacement Deploy¶
For a configured fleet, use the Make deploy target:
Fleet deploy reads ~/.mac/fleets.yaml by default. Override
MAC_DEPLOY_FLEETS_CONFIG when a different registry path is required. The hub
node name selects the fleet. Host-local secret env files still own tokens and
provider credentials.
Fleet mesh networking is configured under defaults.network or per-agent
network overrides in ~/.mac/fleets.yaml. provider: tailscale is the
default and uses MAC_DEPLOY_TAILSCALE_AUTH_KEY from ~/.mac/.env when
automatic join is desired. provider: headscale is an explicit advanced mode:
the fleet registry must declare headscale.login_server,
headscale.health_url, headscale.preauth_key_source,
headscale.preauth_key_env, and the DNS assumption. Managed-hub Headscale is
available with headscale.manage: true, but it should be treated as a shared
service with backup, monitoring, and recovery expectations rather than an
implicit default.
Fleet deploy is supervisor-driven, not Linux-systemd-only. Set
MAC_DEPLOY_SUPERVISOR=auto unless a host needs an explicit override. Auto
selects launchd on macOS, systemd on systemd Linux, and supervisord when
that is the available process supervisor. The selected value is written to
MAC_SUPERVISOR_KIND and recorded in deploy manifests.
Fleet deploy mirrors each configured per-agent model into ACC_HERMES_GATEWAY_MODEL,
HERMES_INFERENCE_MODEL, and ACC_LLM_MODEL so upstream Hermes gateway turns
and mac-hermes-task-executor oneshot work use the same per-agent identity.
Upstream provider credentials remain centralized on the hub, resolved by the
in-mac router from MAC's encrypted vault or inherited host-local environment;
spokes receive only their hub-facing MAC token.
Git-host credentials are a separate execution concern. Fleet deploy resolves
MAC_DEPLOY_GH_TOKEN, GH_TOKEN, GITHUB_TOKEN, then the operator's existing
gh keychain login, and writes the result to each managed runtime as
GH_TOKEN. Only the source name is logged; the value travels over SSH stdin,
not in the remote command. Pure gateway_impl: none workers require successful
GitHub validation by default before drain or source replacement. The variable
is also included in OpenShell's private mode-0600 environment bundle so
confined tasks can clone and publish without copied host SSH keys. Do not put
the value in ~/.mac/fleets.yaml, a fleet spec, task metadata, or source
control. A vault record by itself does not populate a worker environment;
deploy or the Kubernetes runner Secret must inject the corresponding
environment key.
Deployment installs a reviewed MAC source bundle and locked service environment on each selected host. The configured gateway implementation determines the separate conversational runtime. Hermes retains its existing upstream service and active profile; MAC does not replace it with an assumed OpenClaw home. Systemd, launchd and supervisord adapters manage native services. Deprecated ACC and CCC databases are not part of the current deployment authority.
When the local Git remote is available, fleet deploy installs ~/.mac/src/mac
as a branch-tracking Git worktree and sets MAC_SELF_UPDATE_REPO to that path.
That lets the AgentBus repo-update control message pull future changes and
restart the listening mac-agent process without another manual deploy pass.
The fleet topology is hub-and-spoke, matching ACC. The configured hub exposes
the shared control plane URL from hub_url; spokes keep a host-local control
plane for local state and Hermes startup checks, but their mac-agent service
registers and heartbeats against the configured hub. By default the hub binds
0.0.0.0 and spokes bind 127.0.0.1.
Runtime lazy dependency installs are disabled after the preinstall step, and
HERMES_REDACT_SECRETS=false in inherited Hermes env files is corrected to
true because disabled redaction is treated as state drift.
The hub also owns the shared-services layer. Fleet deploy installs Qdrant on
the shared-services manager agent by default and configures every agent with
the same QDRANT_URL / QDRANT_FLEET_URL. Each agent receives a
Hermes-visible ~/.hermes/mac-memory-topology.json plus .env pointers that
describe local Hermes soul/conversation state, mac operational provenance, and
hub-managed shared level-2 memory. /startup/hermes reports
qdrant_level2 readiness using redacted endpoints only.
On a macOS hub, Docker containers for system services are expected. Qdrant,
Firecrawl, test PostgreSQL, and telemetry collectors are long-lived network
services whose container isolation is not part of an executor attestation.
This does not permit OpenShell execution sandboxes on darwin: those tasks run
as host applications and attest macos_host, because Docker Desktop's
LinuxKit kernel cannot enforce Landlock against the macOS host. Use docker ps
to confirm that observed containers are known system services; no container
may be treated as a sandbox or as evidence of execution confinement.
Deployment logs and migration reports are written under ~/.mac/logs/ on each
host:
deploy-*.logdeploy-manifest-*-pre.json,deploy-manifest-*-post.json, anddeploy-manifest-latest.jsonrollback-*.shandrollback-latest.shacc-migration-dry-run.jsonacc-migration-import.jsonacc-migration-status.jsonstartup-hermes.jsonhermes-messaging-deps.jsonhermes-home-channel-sync.jsonhermes-redaction-normalization.jsonhermes-log-summary.jsonmac-service-journal.txton Linux, ormac-service.logon macOShermes-gateway-journal.txton Linux, orhermes-gateway.logon macOSmac-agent-journal.txton Linux, ormac-agent.logon macOShub-agents.json
The activation shim for slack_accounts.json is intentionally applied by
mac startup, not by the deploy script, so this path exercises the startup
patch capability.
To roll back the most recent deployment on a host:
The rollback script restores the prior mac source tree, mac venv, Hermes checkout, and service definitions or launchd plists that existed before the deploy pass, then restarts the mac-managed services.
Worker Agents¶
The control-plane service does not execute tasks by itself. Each execution host
must run a worker process that registers or refreshes its machine/agent row,
heartbeats, then claims eligible open work with a real executor. Fleet deploy
installs that process as a service in heartbeat mode by default so hosts are
visible in the configured hub registry without claiming imported ACC work
prematurely:
mac-agent --url http://hub.example.internal:8789 --register \
--agent-name worker-1 --hostname worker-1.local \
--capabilities python,ops,review --resources '{"capacity":2}' \
--heartbeat-only
mac-agent --url http://hub.example.internal:8789 --register \
--agent-name worker-1 --capabilities python,ops,review \
--workspace ~/.mac-agent/workspaces --loop \
--executor ~/.mac/bin/mac-hermes-task-executor
Use --heartbeat-only during deploy validation when you want fleet visibility
without claiming migrated ACC work. Start the --loop form only after the
executor command is the intended production worker. Successful executions write
log evidence, move tasks to needs_review, and ask the control plane to run
the default review workflow. The default workflow prefers a healthy reviewer
that has never owned the task. If no independent reviewer is currently
eligible, it may assign the least-conflicted healthy review-capable agent and
records reviewer_independence=fallback plus the reason in task history and
observability. A newly available independent reviewer supersedes a pending
fallback review. Set task metadata review.require_independent_reviewer: true
(or review.allow_independence_fallback: false) for work that must wait instead.
Every path still requires a separate signed review_verdict; agent-generated
work still requires a reviewer LLM different from the executor LLM. The
workflow publishes/completes the task only when executor evidence and reviewer
verdict are verifiable.
Failed executions fail the task with evidence attached.
For high-risk work, set metadata.review.risk_level to high or critical.
Approval then fails closed unless the signed executor and verdict manifests
identify different model families and different upstream providers; merely
using two versions of Claude, GPT, or another single lineage is insufficient.
Unknown lineage/provider metadata also blocks approval, and reviewer
independence fallback is disabled. The same constraints can be enabled
individually with review.require_different_model_family: true and
review.require_different_model_provider: true.
How approved work lands: pull requests, not pushes to main¶
Publication is how an approved task reaches the canonical branch. The default is a pull request, matching the worktree-and-PR discipline the repository already asks of humans:
- The agent works in its own worktree on its own branch and pushes only that
branch (the executor finalizer's
guarded_push, which pushes an explicitHEAD:refs/heads/<task-branch>refspec and refuses if the canonical tip is not already contained in the branch). - The agent opens its own pull request, onto the task's canonical branch
(from the repository contract — never assumed to be
main), and records it in the worker evidence manifest asrepo.pull_request. - On approval, the hub clones the exact canonical tip, runs the merge gate,
makes sure the reviewed commit is on the task branch on the remote, and
reuses the agent's pull request. It opens one itself only when the
evidence carries none — an older worker, or a forge the agent could not
reach — and that fallback is recorded (
opened_by: hub_fallback) rather than being indistinguishable from the normal path. - The forge lands that pull request, pinned to the reviewed head SHA (the
pull-request equivalent of
--force-with-lease: if the branch moved underneath us, the merge is refused rather than landing something nobody reviewed) — through the branch's merge queue when it has one, and otherwise through a plain squash merge, see below. - The hub verifies the resulting canonical tip and records it as the canonical-integration proof.
The hub never pushes the canonical branch on this path, and it does not author pull requests: opening one is the agent's decision about how its own work lands, and the hub's job is to record that it happened and to gate completion.
The agent's forge credential. The agent uses the credential already in its
own process environment (GH_TOKEN/GITHUB_TOKEN/GITEA_TOKEN — the same
variable guarded_push authenticates with, written into ~/.mac/mac.env at
deploy time and mounted as an optional secretKeyRef on K8s workers). When it
has none, it resolves github.token from the hub's secret store by name,
at the moment of use — audited, never cached, never written to evidence, never
carried in a bus message or task metadata. A resolved secret that does not
claim the forge's capability is refused rather than sent to the API.
Verify, do not assume. The party requesting the merge confirms the required
contexts actually passed for the reviewed head SHA before asking, rather than
trusting the forge to refuse if they have not. This is not belt-and-braces
pedantry: the fleet authenticates as the repository owner, and the ruleset
carries a RepositoryRole bypass with mode always (deliberate operator
break-glass), so the merge request inherits that bypass and can land straight
past required checks — the first publication under this flow merged two seconds
after the pull request was opened, with every required context reported
SKIPPED. Nothing gated it: not the forge (bypassed) and not the hub (which
skips its own contract re-projection precisely because the forge reports
required checks).
A context that has not reported, is still running, or reported skipped is
not a pass; publication defers with
publication_failure_kind=pull_request_checks_pending. A context that reported
a real failure raises pull_request_checks_failed. An empty required-contexts
list is a different case from a required list that is still pending, and the
two are recorded separately in the publication evidence
(required_check_verification.case: none_configured, pending, failed,
unverifiable, or verified) — an unprotected repository is gated by the
hub's own contract run instead, and a repository whose checks simply have not
started is not mistaken for one.
How the merge is serialized. merge_queue.py validates against the
projected post-merge state — the "Not Rocket Science Rule": test the tree
that will actually land, serialize the merges, and post-merge testing is then
redundant. A plain forge squash merge does not preserve that: if the
canonical branch advances between the required checks completing and the merge
executing, the tree that lands was never tested as such. Required status checks
alone do not close this; a merge queue does, and unlike
strict_required_status_checks_policy it serializes the merges without
serializing the test runs (which here take ~2 hours, so strict rebasing never
converges with several open pull requests).
| branch has | what mac does | the guarantee |
|---|---|---|
a forge merge queue (merge_queue) |
enqueues the PR pinned to the reviewed head; the queue tests the projected merge and lands it in order | what was tested is what lands |
no forge merge queue (mac_native_queue) |
orders the change in mac's own merge queue, tests it against the tree it will land on, and refuses the merge unless the canonical tip's tree is still that exact tree | what was tested is what lands |
Both branches are recorded in the publication evidence as a
merge_serialization entry naming the mode and its guarantee, and on the
canonical-integration proof as merge_serialization.
mac's own merge queue¶
Why it exists. GitHub merge queues are available only on
organization-owned repositories, and GitHub has said it does not plan to
open them to personal accounts. Adding a merge_queue rule to a User-owned
repository's ruleset returns HTTP 422 Invalid rule 'merge_queue' even with no
parameters. So on every personal repository mac manages there is no forge queue
to borrow serialization from, and mac_native_queue is not a rare fallback —
it is the only path. src/mac/native_merge_queue.py provides the queue itself.
What it does. Approved changes awaiting land are ordered per (repository,
canonical branch) in the merge_queue_entries table. Entry N is projected on
top of entries 1..N-1 (Zuul's speculative merge train) so several entries can
be tested in parallel, and they land in order. If entry K fails it is
evicted, and every speculative result behind it is discarded — those
entries were green against a state that will never exist — and the survivors are
re-planned in a new speculation epoch without K.
The invariant. Never land an untested tree. Each entry records the tree it was tested against; the land gate refuses the merge unless the canonical tip's tree is byte-identical to it. Comparing trees rather than commit SHAs is what makes speculation safe and what survives squash merges, which change the commit but not the tree. Every ambiguous state — an unreadable tip, a lost lease, a PR whose state the forge will not report — defers through the existing publication retry backoff. None of them can reach "merge anyway".
Never double-land. Before merging, the queue reads the pull request's state. A PR already merged (by the forge, a human, or an attempt of ours that died after the merge) is observed and recorded as landed, not merged again. Landing is idempotent in the ledger, so a hub restart mid-flight cannot credit one land twice.
Bounded. An AIMD window — the same control law as TCP congestion control,
which is where Zuul got it — caps how many entries may speculate at once. It
starts at the floor (so a fresh queue is strictly serial), grows by
MAC_MERGE_QUEUE_WINDOW_INCREMENT on each successful land up to
MAC_MERGE_QUEUE_WINDOW_CEILING, and halves on any failure down to
MAC_MERGE_QUEUE_WINDOW_FLOOR. Entries outside the window defer; they keep
their place in line.
| knob | default | what it bounds |
|---|---|---|
MAC_MERGE_QUEUE_WINDOW_FLOOR |
1 |
the narrowest window; 1 is a strictly serial queue |
MAC_MERGE_QUEUE_WINDOW_CEILING |
4 |
the most entries that may speculate at once, and therefore the most workers speculation can occupy. Set to 1 to disable speculation without disabling the queue |
MAC_MERGE_QUEUE_WINDOW_INCREMENT |
1 |
how fast the window recovers after a failure |
MAC_MERGE_QUEUE_LEASE_SECONDS |
5400 |
how long a slot may be held before a dead hub's slot is reclaimable. Deliberately longer than a full contract run (~45 min) |
MAC_MERGE_QUEUE_CAPABILITY_TTL_SECONDS |
86400 |
how long a resolved forge capability is trusted before it is re-probed |
What is observable. Every publication records a merge_serialization
command carrying a queue snapshot: queue_depth, window_size,
window_floor/window_ceiling, entries_testing/entries_tested,
landed_count, failure_count, speculation_discarded, and the last ten
evictions with their reasons. The same numbers are emitted as metrics under
merge_queue.* (GET /observability/metrics?name=merge_queue.evicted), and the
per-attempt commands name each decision: merge_serialization_capability,
merge_queue_slot, merge_queue_speculative_base, merge_queue_tested,
merge_queue_observe_pull_request, merge_queue_land_gate,
merge_queue_landed, merge_queue_eviction.
Which mechanism applies is a stored project attribute, not a per-merge probe.
mac.merge_capability resolves the forge's capability once and stores it on the
project's repository record in project_repositories.metadata under
merge_serialization_capability. It records supported and enabled
separately (they differ: an org repo can have a queue and may not have turned it
on), the forge kind, whether a credential resolved, and when and by what it was
determined — so an operator can see that an answer is six weeks old rather than
trusting it silently. The existing GitHub ingest poller refreshes it on its
normal pass, behind MAC_MERGE_QUEUE_CAPABILITY_TTL_SECONDS, and reports the
outcome in its run report under merge_queue_capability. To force a refresh
now, run the poller: mac admin fleet github-ingest run (POST /github-ingest/run).
A missing or expired answer is re-resolved at publication time. Unknown is
never permission to do an unserialized squash — it routes to mac's queue,
which serializes correctly regardless of what the forge does.
Live-hub note.
schema.sqlisCREATE TABLE IF NOT EXISTSwith no migration framework, andPostgresStore.initialize()only creates missing tables.merge_queue_entriesandmerge_queue_windowstherefore appear on a hub the next time the schema is applied; on an already-running hub, apply the DDL fromsrc/mac/data/postgres/schema.sql(the block headed "mac's own merge queue") once by hand. Until they exist, publication on a repository without a forge queue will fail rather than fall back to an unserialized squash — which is the correct direction to fail.
Queued, not merged yet. A pull request accepted into the merge queue has
not landed. Publication defers with
publication_failure_kind=pull_request_queued through the same retry backoff
pending checks use, and a later attempt observes the merge the queue performed
(it asks the forge for the pull request's state before doing anything, so a PR
the queue already landed is never merged twice).
Who gates the merge. Both, at different moments. mac's own reviewer verdict plus the merge gate decide whether a pull request is opened and a merge is requested at all; the forge's required status checks decide whether that merge is permitted. When the forge reports required status checks for the canonical branch, the hub skips its own local re-projection of the contract suite — GitHub runs those checks against the merge result it will actually produce, which is a better question than the hub's approximation and does not cost the 15–45 minutes that previously made approved tasks time out unpublished. When the forge reports no required checks, the hub keeps its own contract gate: an unprotected repository must not silently lose its gate.
Checks still running. A merge the forge refuses because its own gates have
not finished is not a failure. Publication defers with
publication_failure_kind=pull_request_checks_pending and the existing
publication-retry backoff re-attempts later; the pull request is reused rather
than reopened, so retries are cheap. The task stays in reviewing — approved
but not completed — until the change is genuinely on the canonical branch.
Squash and the integration proof. A squash merge lands the reviewed
content under a new SHA, so the reviewed commit is deliberately not an
ancestor of the canonical tip afterwards. The canonical-integration proof
records this honestly (squash_merged: true, contains_reviewed_head: false,
plus the pull-request number and URL) instead of asserting an ancestry that
squashing destroys, and task completion accepts that proof shape.
Opting out. MAC_PUBLICATION_STRATEGY selects the strategy:
| value | behaviour |
|---|---|
pull_request (default) |
push the branch, open a PR, let the forge squash-merge it |
direct_push |
the legacy path: merge locally in a disposable clone and push the canonical branch under optimistic concurrency (push_main_occ) |
Repositories with no forge. mac manages repositories that have no pull
requests at all — a bare path, a file:// remote, or an http(s) remote for
which no GH_TOKEN/GITEA_TOKEN is configured. Those fall back to
direct_push automatically rather than failing, because refusing to publish
them would strand the work. The fallback is never silent: the publication
evidence carries a publication_strategy entry naming the strategy and the
reason it was chosen, so a repository that should be using a pull request but
is not shows up in mac task show's publication detail.
Migration. Nothing is stranded by the change. Publication is stateless per attempt (a fresh clone every time) and the worker side already pushes only its own branch, so a task that was mid-publication under the old path simply gets a pull request on its next attempt against the branch it had already pushed.
Repository credential learning and reviewer routing¶
Every completed remote review preparation writes an authoritative
fleet_learning:repository_access memory record. The JSON content uses schema
mac.fleet_learning.v1 and records only routing-safe facts: project,
repository host, operation, agent, credential source name, outcome, failure
class, bounded redacted error signature, recommendation, and task/review IDs.
It never stores a credential value or authenticated URL.
For the task's repository host, reviewer selection prefers agents with a
recent successful review_clone, then agents with no recent matching record.
An agent whose newest matching record is an authentication or authorization
failure is ineligible during the configured cooldown. A newer success restores
eligibility immediately; cooldown expiry returns the agent to unknown status.
This lookup reads the authoritative PostgreSQL ledger directly, so routing changes immediately and does not
wait for Qdrant vector backfill.
Review Jobs receive optional Git-host keys from the runner's configured Secret
(default mac-api-config): GH_TOKEN, GITHUB_TOKEN, GITEA_TOKEN, and
GITEA_USER. Missing keys are allowed for public repositories. Review Jobs do
not receive MAC_SECRET_KEY. Host-mode workers also recognize
MAC_TASK_GIT_TOKEN as a fallback. The worker injects an HTTPS token only into
the Git command and restores the checkout's origin to the clean remote URL.
The pushed-ref evidence check uses the same environment-backed resolver. An authentication, authorization, or network failure on the control-plane host is indeterminate: it does not prove that the pushed ref is absent. A successful lookup with no matching ref still rejects phantom-push evidence, and the review verdict must still verify the executor's work.
Inspect the routing inputs and resulting review state:
mac --json admin memory search \
--record-type fleet_learning:repository_access \
--order desc --limit 50
mac --json admin memory search \
--subject-type agent --subject-id agent_... \
--record-type fleet_learning:repository_access \
--order desc --limit 20
mac --json task show task_...
Do not hand-edit or invent success records to force routing. Repair the agent's credential, run a real repository operation, and let the resulting success supersede the failure. Multiple success records can exist when a review is retried; only the newest matching record controls eligibility.
Every mac-managed subprocess records a short-retention command audit event on
the hub. The log captures command_id, agent, task, sanitized argv, cwd,
start/end timestamps, return code, output byte counts, and output hashes. The
default retention is 24 hours (MAC_COMMAND_AUDIT_RETENTION_SECONDS), and the
latest rows are visible from /command-audit and the dashboard Observability
view. This is operational telemetry for proving agents are doing work; a future
security audit store can consume the same event shape externally.
The command audit is not a durable compliance archive. Its current job is to make the last day of worker behavior visible without relying on local shell history or unbounded per-host logs.
Executor success is not completion. A zero return code only means the executor
reported without crashing. For the default workflow to auto-approve and publish,
the evidence metadata must include a mac.worker_evidence.v1 verification
manifest. Repo/code work must include a pushed git artifact (head_sha,
remote_ref or PR URL, pushed=true, dirty=false, changed files) plus at
least one passing test/check. Documentation or investigation work must include a
pushed repo artifact or explicit artifacts plus passing checks. Deployment work
must include deployment targets/services and passing checks. Thin reports,
local-only diffs, missing manifests, failing checks, or unverifiable claims stay
in needs_review/reviewing for manual handling.
During execution, the worker renews its active task lease. A successful renewal
also refreshes the owning agent's last_seen_at, keeps it busy, and preserves
current_task_id, so long-running work remains visible as live without allowing
an invalid idle heartbeat while the lease is active.
For already-migrated or pre-upgrade rows that are stuck in needs_review, run
the backlog tick against the hub:
Before enabling executor-backed claiming, use dry-run mode to record routing
candidates without creating leases:
MAC_DEPLOY_WORKER_MODE=dry-run
MAC_DEPLOY_WORKER_CLAIM_ONLY_CANARY_TASKS=1
MAC_DEPLOY_WORKER_ALLOWED_PROJECTS=mac-canary
Dry-run mode emits worker.routing.policy,
worker.routing.dry_run_candidate, or worker.routing.no_candidate events
into /observability and /observability/stream. It must show only synthetic
canary work before loop mode is enabled.
To enable executor-backed claiming from deploy config, set:
MAC_DEPLOY_WORKER_MODE=loop
MAC_DEPLOY_WORKER_CAPABILITIES=ops,python,hermes,review
MAC_DEPLOY_WORKER_CLAIM_ONLY_CANARY_TASKS=1
MAC_DEPLOY_WORKER_ALLOWED_PROJECTS=mac-canary
The generated service then runs mac-agent --register --loop against
MAC_HUB_URL, using MAC_WORKER_TOKEN from ~/.mac/mac.env. The default
executor wrapper is ~/.mac/bin/mac-hermes-task-executor, which calls the
deployed upstream Hermes checkout in one-shot mode. The runtime proof treats
that executable as a required session capability, so a deployed agent is not
considered ready for Codex-like task work unless the executor path is present
and executable.
Fleet deploy deliberately avoids printing the mac-agent process command line.
On Linux it reports mac-agent.service with systemctl show summary fields
instead of systemctl status, because the service wrapper currently passes the
worker token to mac-agent as process argv. Deployment logs should therefore
show service state, PID, and restart count, but not the bearer token. Operators
should continue to treat host-level process inspection as privileged access.
Workers advertise review by default so the default review workflow can pick
real second-eye reviewers. During registration the worker persists its
attestation key into ~/.mac/mac.env; if an older deploy missed that one-time
key, the service rotates a replacement before it signs new evidence. Rotation
is explicit recovery behavior and invalidates old signatures from that agent.
Loop mode claims ordinary eligible work by default. Set
MAC_DEPLOY_WORKER_CLAIM_ONLY_CANARY_TASKS=1 only for a temporary
qualification lane in which the worker must reject every task that is not
explicitly canary-marked. Clear it immediately after qualification.
Legacy Beads Migration And Repository Registry¶
The live production issue authority is the MAC task ledger. Beads/Dolt
round-trip sync is not part of the normal deployment path, and operators should
not run bd for current MAC task lifecycle. Legacy Beads state is handled as a
one-time import path:
Use --tickets-only only when creating local compatibility files during a
migration audit. .tickets/<id>.md is ignored local operational state; it is
not checked into this repository and is not a cross-host ledger.
Repository-backed execution uses the project repository registry instead of a legacy bridge poller. Register or onboard repositories through the current project/repository commands:
mac project register # current checkout; creates contract task
mac project register <path-or-git-url>[#branch] # local or remote-first onboarding
mac project update <project> --branch <branch> # move the internal project fork
mac project unregister <project> --force # detach history; disable checkout registration
mac admin bridge repository register <name> <path> --project <project> # after .mac/project.yaml exists
mac admin bridge repository repos
Every registered repository must include a repository runtime contract at
.mac/project.yaml. Registration and task creation use that contract to attach
the bootstrap command, canonical test command, supported host families,
canonical remote URL, and required evidence to new tasks. See
Repository Runtime Contract.
The hub's repository-ref reconciler uses this registry as its complete workset.
The hub therefore needs filesystem and GitHub access to each enabled checkout;
automatic prune additionally requires the contract's canonical_remote_url.
Monitor it with mac admin repo refs status and trigger a one-shot audit with
mac admin repo refs reconcile --mode audit. See
Managed Repository Ref Hygiene.
The hub advances the default review/publication workflow from heartbeat when
MAC_REVIEW_TICK_ON_HEARTBEAT=1 and MAC_REVIEW_TICK_HUB_AGENT is set to the
configured hub agent. The tick only moves tasks when required evidence,
reviewer verdicts, and publication targets are present; otherwise it records
explicit waiting reasons in observability.
AgentBus¶
/messages remains the constrained structured control bus and still rejects
execution-shaped payloads. Agent-to-agent content exchange uses /agentbus
instead:
POST /agentbus/streamsopens a typed stream withcontent_type,topic, optional task linkage, and optional recipient.POST /agentbus/streams/{id}/chunksappends ordered JSON, text, or base64 chunks.final=truecloses the stream atomically after the chunk.GET /agentbus/streams/{id}/chunksreads durable chunks after a sequence.GET /agentbus/streams/{id}/eventstails chunks as NDJSON for streaming consumers.
This is a typed transport channel, not an execution channel. Agents must still turn received content into explicit task/evidence/review actions through the normal API.
mac-agent also listens for one constrained AgentBus control topic before it claims tasks:
- Topic:
mac.repo.update.v1 - Content type:
application/vnd.mac.repo-update+json - Payload schema:
mac.agentbus.repo_update.v1
The listener runs git pull --ff-only in MAC_SELF_UPDATE_REPO and exits for
service-manager restart only when the worktree HEAD changes. Dirty worktrees,
non-git source trees, invalid remotes/branches, and pull failures are reported
as result streams instead of being forced. Result streams use topic
mac.repo.update.result.v1 and content type
application/vnd.mac.repo-update-result+json.
To broadcast a source update from the hub:
Roles, Workflows, and Provisioning¶
Production mac includes an API-level organization model for coordinated work:
/rolesstores the role catalog used to describe agent jobs, prompts, required/default capabilities, optional hardware requirements, and tenant scope./roles/seedloads the built-in Loom-style role set./agents/{id}/roleassigns a role to a registered agent. If the agent is bound to a Hermes persona, role assignment respects that persona's allowlist./provisioning/requestsrecords missing capacity requests when the fleet has no suitable agent for a role/capability requirement./workflowsstores versioned DAG definitions./workflows/import-yamland/workflows/seedprovide operator-friendly loading paths./workflows/{id}/start,/workflows/runs, and/workflows/runs/tickrun and sweep workflows. Each node creates a normal mac task, so dispatch, evidence, review, publication, command audit, and Beads ledger behavior stay the same as single-task work.
/dashboard/state includes workflow-run summary data for UI clients. Full
visual workflow authoring is still an operator-facing gap in the checked-in
dashboard; use the API/CLI for workflow creation and editing until that UI is
built.
Docker Engine / Moby¶
Docker on a darwin hub is supported for system services, including Qdrant,
Firecrawl, test PostgreSQL, and telemetry collectors. It is never an OpenShell
execution runtime on darwin, and its availability does not alter the node's
macos_host posture. docker ps should therefore show only the expected
service containers; investigate any container presented as a task sandbox.
docker build -t mac:latest .
docker run -d --name mac \
-e MAC_SECRET_KEY="$(openssl rand -base64 48)" \
-e MAC_API_TOKEN="$(openssl rand -hex 32)" \
-v mac-data:/var/lib/mac \
-p 127.0.0.1:8789:8789 \
--restart unless-stopped \
mac:latest
# Healthcheck is built into the image; `docker ps` shows (healthy) once up.
curl -fsS http://127.0.0.1:8789/health
For Kubernetes, ship the same image as a single-replica Deployment with a
PVC mounted at /var/lib/mac. Use a ConfigMap for non-secret env and a
Secret for MAC_SECRET_KEY / MAC_API_TOKEN.
Backups¶
Use PostgreSQL backup tooling against the configured authority. Keep credentials in a protected PostgreSQL service/password file or the process environment. A custom-format dump supports a deliberate restore into a separate target:
Set the connection environment for the intended database first. A dump listing is an inventory check, not a restore drill. Verify a restore into an isolated database, including the ordered migration ledger and application reads, before relying on a backup. Copying a data directory from a running server is not a supported application backup procedure.
Observability¶
GET /healthis the liveness/readiness signal. Returns{"status":"ok"}.GET /startup/hermesreturns the redacted Hermes startup report: file existence/size/mtime metadata, warning strings, and Slack activation status.GET /eventsis the unified audit stream — point a log shipper (vector, promtail, fluent-bit) at it withsince=advancing every poll, or scrape the PostgreSQL tables through an authorized read connection.mac admin events list --since <iso>is the operator's one-shot "what just happened" query.POST /observability/metricsandPOST /observability/logsingest layer/source/name/level observations from workers, Hermes adapters, deploy scripts, and external monitors. POST requiresagentscope when API tokens are enabled.GET /observabilitylists persisted observations;GET /observability/summaryreturns dashboard aggregates and latest metric snapshots;GET /observability/streamtails observations as NDJSON for live dashboards or collectors.GET /notificationslists the durable operator notification outbox for task lifecycle, review, publication, and bridge-stale events.POST /notifications/{id}/deliveredmarks entries delivered, failed, or skipped.
The FastAPI middleware records per-request http.request.duration_ms metrics
and http.request logs. Control-plane task, agent, project, fleet, secret,
environment, rollout, and eval events are mirrored into the observability
stream with their original subject ids. The dashboard Observability tab uses
URL-addressable filters, the summary endpoint, and an NDJSON subscription to
visualize the live stream, unified events, command audit, and the operator
notification outbox.
Upgrade procedure¶
Use the synchronized cutover runbook and its typed transaction protocol. Preserve an independently restorable PostgreSQL backup and the prior diagnostic artifacts. Deploy applies the ordered schema migrations before candidate services start; startup verifies rather than silently changing schema.
Keep failed candidates and their affected workers held for diagnosis, then repair forward. Restoring an older source or database is an explicit break-glass operation, not an automatic response to a failed check. A release artifact alone does not authorize bypassing the cohort's fault-matrix gates.
Kubernetes (K8s-native topology)¶
For multi-replica mac-api, deploy the manifests under deploy/k8s/.
The Postgres cluster itself is not managed from this repo — bring
your own (CloudNativePG, RDS, Cloud SQL, vendor-managed, etc.) and
supply the DSN via the mac-api-config Secret. Likewise, ArgoCD
Application manifests are not shipped here; point one Application
per kustomize tree from your platform-config repo if you sync with ArgoCD.
# 1. Create the namespace + operator-supplied Secret carrying the DSN
# and bearer tokens (or apply your ExternalSecret).
kubectl create namespace mac
kubectl -n mac create secret generic mac-api-config \
--from-literal=MAC_DATABASE_URL='postgresql://user:pass@host:5432/mac' \
--from-literal=MAC_SECRET_KEY="$(openssl rand -base64 48)" \
--from-literal=MAC_WORKER_TOKEN="$(openssl rand -hex 32)"
# 2. mac-api Deployment + Service (replicas: 2, no PVC).
kubectl apply -k deploy/k8s/mac-api
# 3. mac-k8s-orchestrator — claims ready tasks, creates one batch/v1
# Job per claim, and reconciles stuck Jobs against mac-api lease state.
kubectl apply -k deploy/k8s/mac-runner
The full apply order and ExternalSecret wiring are documented in
deploy/k8s/README.md. The persistence layer uses PostgreSQL in every supported topology. The
legacy SQL compatibility helpers do not constitute a second supported engine.
The archived Kubernetes rewrite plan records migration history, not current
backend selection.
Troubleshooting¶
Deploy aborts: "Qdrant did not become ready at :6333" (high-core nodes)¶
On a node where the container's CPU cgroup quota is small but the host exposes
many cores (a common Kubernetes shape — e.g. a 4-CPU pod on a 192-core node),
Qdrant sizes its actix/tokio thread pools from /proc/cpuinfo (the full node
core count, not the cgroup quota) and tries to spawn far more threads than the
container's PID cap allows. Thread creation then fails with EAGAIN and Qdrant
panics at startup:
ERROR qdrant::startup: Panic ... Cannot spawn Arbiter's thread:
"actix-rt|system:0|arbiter:NNN": Os { code: 11, kind: WouldBlock,
message: "Resource temporarily unavailable" }
mac-qdrant-run (or the systemd unit) exits immediately, supervisord/systemd
marks the program FATAL, and the deploy aborts because Qdrant is mandatory on
the hub. The fix ships by default: the container PID cap is now
QDRANT_PIDS_LIMIT (default 4096) instead of a hardcoded 512. If an even
larger node still hits this, raise it in the fleet env, e.g. QDRANT_PIDS_LIMIT=8192,
and redeploy. Quick check on the affected host: --pids-limit=512 → panic /
curl :6333/collections returns 000; raised limit → 200.
supervisord pods: supervisorctl permission errors¶
On pods where supervisord runs as root (PID 1) with a root-only control socket
(srwx------), supervisorctl as the deploy user gets
PermissionError ... xmlrpc.py. The deploy already prefers sudo supervisorctl
(run_supervisorctl); ensure the deploy user has passwordless sudo. A bare
supervisorctl ... in a manual session needs sudo.
Private repository review is retracted or remains in reviewing¶
Start with the task and shared learning records:
mac --json task show task_...
mac --json admin memory search \
--record-type fleet_learning:repository_access \
--order desc --limit 50
Interpret the review reason before retrying:
reviewer_unavailable:reviewer_repository_access_authentication:<host>means MAC learned that the selected reviewer could not authenticate, retracted it, and will prefer a recent successful peer. Repair that agent's host-specific token before expecting it to be selected during the cooldown.reviewer_unavailable:reviewer_repository_access_authorization:<host>means the credential was recognized but lacks access to the repository or organization. For GitHub organizations using SSO, authorize the token for the organization rather than rotating it blindly.reviewer_unable_to_produce_verdict_after_<N>_attemptsmeans repository access may have succeeded but the reviewer executor did not emit a valid signed verdict. Inspect the review evidence and reviewer workspace; changing Git credentials is not the appropriate repair unless the memory record also reports an access failure.
The delivered-nudge cap prevents an unparseable or crashing reviewer from spinning forever. It does not currently learn general executor/harness failures as reviewer-routing exclusions; track and repair those separately.
Dynamic model selection (opt-in, hub only)¶
The hub can periodically propose the fleet's "powerhouse" models from web-search
mentions, filtered against the configured providers' models.dev catalogs. It is
opt-in and does not control the default deployed router today: explicit
MAC_ROUTER_DEFAULT_MODEL and MAC_ROUTER_WILDCARD_MODELS values still win.
The first successful selection is stored as active because there is no incumbent;
later dynamic changes remain pending unless an enabled eval gate approves them or
an operator runs mac admin fleet model-selection promote.
Do not describe the current selection as proof that the router can serve a model. The catalog namespace is not yet reconciled with the router's exact model allowlist, and the per-worker strength ladder is not distributed from the hub. These gaps are why deployment leaves both selection and its automated swap evaluator disabled by default.
Environment variables (hub):
| Variable | Default | Meaning |
|---|---|---|
MAC_MODEL_SELECT_ENABLED |
off | Run the weekly selection refresher. |
MAC_MODEL_SELECT_INTERVAL_SECONDS |
604800 | Refresh cadence. |
MAC_MODEL_SELECTION_FILE |
$MAC_HOME/model-selection.json |
Where the active/pending selection + strength ladder are persisted. |
MAC_MODEL_SWAP_EVAL_ENABLED |
off | Evaluate later swaps through the configured router and automatically adopt an approved candidate; otherwise swaps stay pending for mac admin fleet model-selection promote. |
MAC_MODEL_SWAP_EVAL_GOLDEN_SET |
built-in floor set | Path to a JSON or JSONL golden set of eval cases. |
The built-in floor includes paired benchmark-labelled and production-shaped
agentic-integrity cases for fabricated work, test tampering, score
falsification, and tool-result concealment. Every pair uses the same behavioral
requirement with pair_id plus presentation (benchmark or realistic). The
swap gate records realism_gap, the mean absolute score gap within complete
pairs, and blocks candidates whose gap increases by more than the configured
drift threshold. Integrity requirements belong in safety_required_points;
each entry may be a string or a list of acceptable phrasings. Missing one is a
safety violation and forces that case's correctness to zero.
The built-in cases are a smoke-test floor, not a deployment-quality corpus. Configure a version-controlled, rotating, production-shaped holdout that covers the fleet's actual tool, repository, evidence, and reviewer workflows. Keep a private portion out of model-facing prompts, preserve benchmark/realistic pair IDs, and review both aggregate quality and the realism gap before promotion.
Opaque hosted-model APIs expose outputs, not residual-stream activations, so MAC cannot honestly run a Jacobian lens or any other model-internal activation audit through the current router. MAC's optional external activation probe only classifies tensors supplied by a model runtime that the operator owns and instruments; it does not recover hosted-model states. If a future open-weight backend exposes compatible residuals, the probe must remain advisory and use a separately validated classifier plus a held-out calibration set. Deterministic evidence, test, review-diversity, and publication gates remain authoritative; an activation classifier must never approve work by itself. See External activation-probe prototype for its exact data boundary and non-goals.
Per task, --model <name> pins a model by name and --model-strength 1..10
pins by capability (resolved via the strength ladder; hub agent only until
ladder distribution lands).
Autonomous scientific optimization¶
The hub can continuously test bounded execution-policy changes against task
quality, rework, latency, tokens, and cost. The durable experiment registry,
database-backed singleton scheduler, mandatory delayed-quality guardrails, and
promotion/rollback workflow are documented in
scientific-optimizer.md. New systemd deployments
leave the scheduler disabled in mac.env.example. Enable
MAC_SCIENTIFIC_OPTIMIZER_ENABLED=1 only after a manual optimizer tick and
ordinary task-flow queries have both met the deployment's latency budget.
Known limitations¶
- Dynamic model selection is opt-in and does not override the explicit router defaults installed by fleet deployment. Catalog/allowlist reconciliation and ladder distribution must land before it becomes a fleet routing control.
- Every topology requires PostgreSQL. Multi-replica deployments share one schema authority and must coordinate migrations before service startup.
- No built-in TLS. Put a reverse proxy in front.
MAC_SECRET_KEYrotation is manual.- Operational success-first routing is currently enforced for repository access during autonomous review. General executor, sandbox, and verdict-production failures are bounded and visible but are not yet learned as reviewer-routing exclusions.
- Existing SQLite authorities migrate offline with
mac admin database migrate-sqlite-to-postgres. Stop every hub writer and worker supervisor first, then pass--hub-stopped; the command checkpoints WAL, holds an exclusive SQLite snapshot, requires an empty PostgreSQL target, copies each table atomically, and writes a resumable owner-only receipt. Completion requires matching deterministic full-row hashes for every table, a clean source integrity/FK check, clean destination FK checks, and reset PostgreSQL sequences. A count-only copy is never reported as successful. Example:
$ export MAC_MIGRATION_DATABASE_URL='postgresql:///mac'
$ mac admin database migrate-sqlite-to-postgres \
--sqlite ~/.mac/mac.db \
--report ~/.mac/migrations/sqlite-to-postgres.json \
--hub-stopped
Use --resume only with that same receipt/source/target, or --verify-only
for a later full verification. --restart explicitly truncates the target
and is the only supported way to abandon a partial target. After a verified
receipt, set MAC_DATABASE_URL, remove MAC_DB, restart the hub, and retain
the SQLite file only as an immutable rollback/export artifact.