Skip to content

Hub Availability

The fleet is hub-and-spoke on purpose: the ledger's atomicity, auditability, and credential centralization all come from having one authority. This document covers the three mechanisms that keep that choice from meaning "one process on one machine is the fleet's single point of failure", and the promote procedure for replacing a failed hub.

1. Inference is decoupled from the ledger process

MAC_DEPLOY_ROUTER_BACKEND=standalone runs the LLM router as its own process (mac-router, listening on MAC_DEPLOY_ROUTER_PORT, default 8790) instead of mounting /v1 inside the hub ledger API:

  • Hub API restarts no longer drop in-flight model streams.
  • Ledger lock pressure and model traffic stop sharing one event loop.
  • Provider credentials stay centralized: on the hub host the router resolves secret: refs from the co-located control-plane store, exactly like the in-process mount did. llm.route observations still land in the ledger.

Run it under the host's supervisor next to the hub API, e.g. systemd:

[Service]
EnvironmentFile=%h/.mac/mac.env
ExecStart=%h/.mac/venv/bin/mac-router --host 127.0.0.1 --port 8790
Restart=on-failure

(or the equivalent launchd plist on a macOS hub). The deploy writes the gateway base URLs to point at the router port; the hub API's own /v1 mount is a no-op for any backend other than inproc, so the two can never both serve model traffic.

Per-site replica. A wing whose network path to the hub is unreliable (e.g. NAT/DERP-relayed pods) can run its own mac-router and set MAC_DEPLOY_ROUTER_URL=http://<replica>:8790/v1 for the agents in that wing. Only inference moves — task claims, evidence, and messaging still go to the hub. The replica authenticates callers with the same hub-facing bearer tokens (MAC_ROUTER_TOKENS); its upstream providers use env keys or key=none private endpoints — hub vault material is never shipped to it.

Riding out provider outages. When no upstream provider answers (all unreachable, overloaded or rate-limited, or all breakers open), the router holds the request and routes it again with backoff (2, 4, 8, then every 15 seconds) for up to MAC_ROUTER_OUTAGE_WAIT_SECONDS (default 120) before it returns 503 all_providers_unavailable with Retry-After: 30. At most MAC_ROUTER_OUTAGE_WAITERS requests (default 8) wait at once, because the in-process router holds a hub request thread while it waits; the rest get the 503 straight away. An upstream 401 or 403 means the provider refused the hub's own key, so the router fails over to the next provider. When every provider refuses, it returns that refusal and does not wait.

If a task's attempt still fails while the router logged all_providers_unavailable for it, the dispatcher reopens the task after the usual retry backoff without charging the attempt, up to MAC_ROUTER_OUTAGE_REQUEUES times per task (default 3).

Infrastructure failures on the worker. Two failures belong to the host, not the task:

  • The finalizer's fetch or push could not reach the git remote: DNS, connection refused or timed out, TLS or RPC transport errors.
  • No coding agent passed its in-sandbox preflight, and every preflight failed in a way a later run can clear: timeout, rate limit, provider 5xx, or no inference token.

The executor records the cause from its own git stderr and preflight classes, never from the agent's output, as infrastructure_failure in the evidence. The worker then blocks the attempt as executor_infrastructure_failure with the cause in its diagnosis, and it names the unpushed head when there is one.

The dispatcher reopens the task without charging the attempt, up to MAC_INFRASTRUCTURE_REQUEUES times per task (default 3). After that, the failure consumes attempts like any transient one, and an identical repeat stops the task. Some failures still go through their normal gates:

  • evidence that claims publication is never covered by this
  • a failed repository test gate
  • missing binaries, refused credentials and policy denials

2. Ledger bytes are bounded: evidence blobs live outside the DB

With MAC_EVIDENCE_BLOB_DIR set (the deploy defaults it to ~/.mac/evidence-blobs on the hub), evidence artifact bytes above MAC_EVIDENCE_INLINE_MAX_BYTES (default 64 KiB) are written to a content-addressed, integrity-verified blob directory; the ledger row keeps digest, size, metadata, and a content_uri. The secret-scoped artifact GET materializes content transparently, so nothing downstream changes. This keeps ledger snapshots small and replication fast — the availability story below gets cheaper because the DB carries coordination state, not payload bytes.

Blob files are deduplicated by digest and never rewritten; reclaim space with an age-based sweep of the directory. Back the directory up alongside ledger snapshots if artifact bytes must survive a hub loss (rsync -a in the same sync hook works).

3. The ledger itself: two availability tiers

Postgres is a first-class store backend (MAC_DATABASE_URL; schema, trigger, and translation layers are in-tree with live tests). Set MAC_DEPLOY_DATABASE_URL=postgresql://... for the hub and the deploy writes MAC_DATABASE_URL and drops MAC_DB — the node still declares exactly one durable authority. Availability then becomes a database problem with off-the-shelf answers (streaming replica + managed failover: CloudNativePG, Patroni, RDS/Cloud SQL). Hub processes are stateless in this mode — the k8s manifests already run replicas: 2 against one Postgres — so hub failover is "start the API pointing at the same DSN".

In-tree verified backups (mac-pg-backup). Streaming replication is the RTO answer, but the hub also keeps its own restore-verified artifact so a break-glass recovery does not depend solely on the database operator. mac-pg-backup (also python -m mac.pg_backup) takes a consistent logical dump and proves it is restorable:

  • pg_dump -Fc (custom archive, single snapshot-isolated transaction) → owner-only 0600 artifact in a 0700 <out>/postgres/ directory → sha256 manifest → retention (--keep-last, default 14) → off-box ship hook (MAC_PG_BACKUP_SYNC_CMD, with MAC_PG_BACKUP_PATH / _SHA256 / _MANIFEST exported, mirroring the ledger path's hook contract).
  • Restore-to-scratch drill. Each backup (or every MAC_PG_BACKUP_VERIFY_EVERY runs) restores the dump into a throwaway mac_restore_verify_* database via pg_restore and proves the schema plus representative row counts (tasks, agents, events) came back, then drops the scratch database. A backup that cannot be restored fails closed — it is not counted as a backup.
MAC_PG_BACKUP_SYNC_CMD='rsync -a "$MAC_PG_BACKUP_PATH"* standby:~/.mac/backups/postgres/' \
  mac-pg-backup --dsn "$MAC_DATABASE_URL" --out ~/.mac/backups

The PgBackupScheduler runs this on an interval inside the hub process, enabled by default for the PostgreSQL authority and disabled with MAC_PG_BACKUP_ENABLED=0. A dump/verify/ship failure is a loud ledger observation (pg.backup.failed) plus an operator notification. Verify an artifact's integrity offline with mac-pg-backup --verify-manifest <dump>.

No SQLite fallback. A PostgreSQL failure — connection loss, a failed dump, a failed restore drill — is surfaced loudly and never downgraded to a SQLite backup or authority. PostgreSQL is the only supported live authority. The immutable 2026-07-28 SQLite cutover archive (mac admin migrate archive: mode-0600, sha256 manifest, verified at creation) is preserved as recovery evidence — a frozen snapshot of the pre-cutover authority for forensic/legal recovery — and is explicitly not a live fallback authority. Nothing restarts it as a second live ledger.

Tier B — retired

There is no SQLite tier. mac-ledger-backup, mac.ledger_backup, and their scheduler have been deleted, and the only backup path is the PostgreSQL one above.

They were removed rather than left disabled. On 2026-08-02 the scheduler was snapshotting the 0-byte ~/.mac/mac.db left behind by the Postgres migration every 15 minutes: an empty database passes integrity_check, so each snapshot was "verified", manifested, retained 14 deep, and shipped to the standby as the recovery artifact — while the real Postgres backup was failing. A disabled scheduler is one environment variable away from doing that again; a deleted one is not.

Promote procedure (standby takes over)

Split-brain rule first: the old hub must be fenced before the standby serves a single write. Two live authorities silently diverge; nothing reconciles them after the fact.

  1. Fence the old hub. Stop its service (systemctl stop <fleet> / launchctl bootout ...). If the host is unreachable, ensure agents can't reach it either (tailnet ACL or shut the node down) before proceeding.
  2. Restore the ledger on the standby.
  3. Tier A: promote the Postgres replica (or let the operator/managed failover do it) and skip to step 3. If no replica survived, restore the newest shipped mac-pg-backup artifact into an empty cluster: mac-pg-backup --verify-manifest <dump> (integrity), then createdb mac && pg_restore --no-owner --no-privileges --dbname=mac <dump>, and point MAC_DATABASE_URL at it. Never restore into a database that is still serving — a Postgres failure is repaired in Postgres, not by falling back to SQLite. Restore the evidence-blob directory alongside it. There is no Tier B fallback: if no replica and no verified dump survived, the authority is gone, and that is what the verified-dump discipline above exists to prevent.
  4. Make the standby the hub. Deploy it as the fleet's hub (MAC_CONTROL_PLANE_ROLE=hub — i.e. it is the hub_agent / shared-services manager in the deploy), same MAC_API_TOKEN/worker-token material (restore mac.env from the standard deploy backups).
  5. Repoint the fleet. Update hub_url for the fleet in ~/.mac/fleets.yaml and redeploy the agents (the --hub-url / MAC_DEPLOY_NEW_HUB_URL path), or — much better — give the hub a stable tailnet DNS name from day one and rebind that name to the standby so no per-agent env changes are needed. hub_url accepts a DNS name anywhere an IP works today.
  6. Old hub stays fenced. When the old host returns, redeploy it as a spoke (its MAC_CONTROL_PLANE_ROLE becomes client, and the store factory refuses to let a client own a DB). Any tasks written to its ledger while partitioned are recovered by an operator reading that database directly under --local-authority and re-filing what matters against the new hub — they do not merge automatically, and no tooling merges them.

What this deliberately does not do

  • No automatic failover: promoting is an operator action precisely because fencing (step 1) cannot be safely inferred by the standby from "I can't reach the hub".
  • No hub-to-hub replication protocol: replication is delegated to Postgres, with verified dumps shipped off-box as the break-glass artifact. That keeps the single-authority invariant the rest of the system (dispatch guards, task authority, attestation) is built on.
  • No fallback authority: the SQLite backup path is deleted, so a Postgres authority cannot silently degrade to a stale ledger, and the 2026-07-28 SQLite cutover archive stays immutable recovery evidence rather than a live authority. Restoring is an explicit operator action gated by the fence in step 1.