Skip to content

Historical field note

This record preserves prior investigation or planning evidence. It is not a current operating contract; use the numbered book and current runbooks for instructions.

Kubernetes-native rewrite plan

Set 2026-05-28 via /goal. This is a separate roadmap from docs/scaling-plan.md. The phase numbers DO NOT match. scaling-plan.md Phases 1-7 cover functional scaling (tenancy, dispatch, review, rollout, workflows). The K8s-native rewrite Phases 3-5 below cover the operational rearchitecture for running mac in Kubernetes.

Architectural separation

Role What it owns
Hermes User conversation
mac-api Durable coordinator (stateless, multi-replica)
mac-agent-worker Role-specific workers
Task Jobs Isolated per-task execution
Postgres (externally managed) Source of truth
ExternalSecrets Credential injection
ArgoCD (operator-supplied) Deployment authority

"Kubernetes-native" does NOT mean baking everything into the Hermes pod. It means each role above runs as its own workload with its own scaling and failure boundary.

Phase 3 — Postgres

Make mac-api survivable as multiple stateless replicas by replacing the single-writer SQLite store with an externally-managed Postgres 17 cluster.

3.1 — Extract Store protocol (zero behavior change)

Goal: introduce a backend abstraction without altering observable behavior.

  • Define Store as a typing.Protocol (structural — no inheritance churn) in src/mac/store.py, covering the 6 public methods: close, transaction, execute, executemany, query_one, query_all, plus the path attribute and an ensure_column(table, column, definition) migration hook.
  • SQLiteStore stays as-is and naturally satisfies the protocol. _conn stays private (sqlite3-specific).
  • Add StoreError exception in the same module; SQLiteStore wraps sqlite3.Error on raise. api.py:1897 imports StoreError instead of importing sqlite3 directly.
  • Type-annotate ControlPlane and downstream services as Store, not SQLiteStore, so a Postgres backend can be substituted.

Acceptance: all existing tests pass; grep -r "import sqlite3" src/mac/ returns only store.py and migration.py (ACC migration reader).

3.2 — PostgresStore with dialect shim

Goal: a second backend that implements Store over psycopg[binary] (sync, matching current call style) without rewriting service-layer SQL.

Driver: psycopg v3 sync + psycopg_pool.ConnectionPool sized from MAC_PG_POOL_SIZE (default 10).

Translation shim (the central design choice — the alternative of rewriting ~50+ raw SQL sites was rejected as too risky):

  • ? placeholders are rewritten to %s inside PostgresStore.execute/executemany/query_*. Unit test covers ≥20 SQL fixtures pulled from services.py, including strings containing ? inside quoted literals (the regex must respect quotes).
  • Result rows are wrapped to support both row['col'] and row[0] indexing, matching sqlite3.Row. Use psycopg's dict_row row factory plus a thin wrapper class.
  • transaction() opens an explicit psycopg transaction; commit on context exit, rollback on exception. Default isolation is acceptable.
  • psycopg.Error is re-raised as StoreError.

JSON dialect helpers (json_extract, json_set, json_object, json) are NOT solved in PostgresStore — they belong in Phase 3.3 (SQL functions defined inside the Postgres schema). PostgresStore just passes those calls through.

3.3 — Schema port (57 tables, indexes, triggers, view)

Goal: a single idempotent db/postgres/schema.sql that creates the same logical schema as SQLiteStore._initialize() on a fresh postgres17 database, plus additive migrations under db/postgres/migrations/00X_*.sql.

Type mappings:

SQLite Postgres
TEXT primary keys TEXT (unchanged)
TEXT JSON columns JSONB (text input auto-casts)
*_at TEXT (ISO strings) TEXT — do not change to TIMESTAMPTZ in this phase; would force call-site rewrites
AUTOINCREMENT (observability_events) GENERATED BY DEFAULT AS IDENTITY
CHECK constraints + triggers on tasks Direct port (syntax is close)

Dialect shim functions defined inside schema.sql so service-layer SQL stays untouched:

  • json_extract(j jsonb, path text) RETURNS text — translates SQLite '$.foo' / '$[0]' paths into Postgres jsonb_path_query_first.
  • json_set, json_object, json() — wrap jsonb_set, jsonb_build_object, to_jsonb.

The events view (store.py:996-1107) is the highest-risk piece — json_set/json_object inside a UNION ALL across ~10 tables. Plan: rewrite it natively using Postgres jsonb_build_object/jsonb_set rather than relying on the shims, so the view compiles cleanly.

_ensure_column for additive migrations: PostgresStore.ensure_column queries information_schema.columns and emits ALTER TABLE … ADD COLUMN IF NOT EXISTS ….

3.4 — Backend factory + env wiring

src/mac/store.py (or src/mac/stores/factory.py) exposes make_store_from_env() -> Store:

  • MAC_DATABASE_URL=postgresql://… → PostgresStore
  • otherwise, explicit MAC_DB → SQLiteStore(MAC_DB)
  • neither configured → startup error; no implicit client-side database

Update all 4 construction sites identified in memory/project_store_surface.md (or equivalent session context) to call the factory:

  • services.ControlPlane.__init__ default
  • services.ControlPlane.in_memory() test factory
  • cli.py entry
  • api.py FastAPI startup

production-deployment.md's env table gets MAC_DATABASE_URL and MAC_PG_POOL_SIZE.

3.5 — Stateless mac-api Deployment

deploy/k8s/mac-api/ (new directory):

  • deployment.yaml — multi-replica Deployment, no PVC, env from the operator-supplied mac-api-config Secret which carries MAC_DATABASE_URL (DSN of an externally-managed Postgres 17 cluster), MAC_SECRET_KEY, MAC_WORKER_TOKEN.
  • service.yaml, namespace.yaml, secret.example.yaml (generic v1/Secret template; operators replace placeholder values and apply out-of-band, or convert into their preferred secret-store object).

The Postgres cluster itself is not managed from this repo. Bring your own (CloudNativePG, RDS, Cloud SQL, vendor-managed, etc.); the DSN is supplied via Secret. ArgoCD Application manifests are operator-supplied for the same reason — sync from your platform-config repo, one Application per kustomize tree.

3.6 — Tests

  • All current tests continue to pass against SQLiteStore (default).
  • New pytest.mark.postgres marker; tests run against a real Postgres (testcontainers-python or MAC_TEST_PG_URL). Same test bodies — fixtures pick the backend via the factory.
  • Placeholder-translation unit test exercises ≥20 representative SQL strings.
  • events view parity test inserts representative rows into each underlying table and compares projected JSON between SQLite and Postgres.

3.7 — Live SQLite → Postgres data migration

One-shot mac db migrate-to-postgres CLI: opens a SQLite file, opens a MAC_DATABASE_URL, copies row-for-row in dependency order. Idempotent on re-run (UPSERT by primary key). For operators upgrading existing deployments.

3.8 — Docs

Update docs/production-deployment.md: remove the "single-writer SQLite" caveat for the Postgres topology, add a "Kubernetes + Postgres" section, document MAC_DATABASE_URL. Update README.md topology summary.

Phase 4 — Job-per-task execution

Each claimed lease becomes a Kubernetes Job. The worker process inside the Job handles exactly one task and records evidence before exiting. Crash recovery is delegated to the Job controller + the existing lease-expiry retry path already implemented in mac-api.

Chosen approach: new mac-k8s-runner Deployment

Recommended over extending mac-agent because:

  • mac-agent is designed to run tasks itself (executor + loop in one process). Repurposing it as a Job-launcher is a deeper rewrite than starting fresh.
  • K8s-idiomatic: one workload type per responsibility. Runner watches + creates Jobs; Jobs run executors; controllers reconcile failures.
  • Permissions surface is smaller: runner needs batch.Job create/list; individual mac-agent images don't need cluster API rights.
  • Independent scaling: a single runner Deployment serves many concurrent Jobs; runner replicas only need to grow with API/dispatch throughput.

4.1 — mac-k8s-runner Deployment (skeleton)

New binary entry point mac-k8s-runner that:

  1. Polls mac-api /dispatch/candidates for ready tasks the runner is configured to handle (filtered by capability label).
  2. For each candidate, calls POST /tasks/{id}/claim to atomically acquire the lease (the same call workers use today).
  3. Creates a batch/v1 Job in the configured namespace:
  4. One container per Job
  5. Image: mac:<runtime-env-digest> (resolved from runtime_environments.manifest for the task)
  6. Env: MAC_TASK_ID, MAC_LEASE_ID, MAC_URL, MAC_WORKER_TOKEN (short-lived bearer token minted per-Job via POST /tokens/mint)
  7. backoffLimit: 0 — retries are owned by mac-api's lease-expiry + max_attempts, not the Job controller, so Job restart semantics stay simple.
  8. activeDeadlineSeconds: matches the lease TTL so a stuck Job is killed before its lease expires.
  9. Records the Job UID in task.metadata.k8s_job_uid so reconciliation can correlate.

Runner is multi-replica. Each replica leases tasks independently; the existing uniq_leases_active_per_task partial unique index prevents two runners from launching duplicate Jobs.

4.2 — In-Job executor: mac task run single-shot

Add a CLI subcommand mac task run --lease <id> that:

  1. Reads MAC_TASK_ID, MAC_LEASE_ID, MAC_WORKER_TOKEN from env.
  2. Renews the lease while running (every lease_ttl/3).
  3. Invokes the task executor (existing mac-hermes-task-executor path or a per-role override).
  4. Submits evidence + transitions the task to needs_review / failed before exiting (the K8s Job controller treats a 0 exit as success; evidence must be persisted before that signal so we never lose work on a clean exit race).
  5. Exits 0 on successful submission, non-zero on failure to submit (lease-expiry path then reopens the task).

This is the "single-shot" mode of the existing worker loop. Most of the code already exists in src/mac/worker.py; the change is to make the loop wrapper optional so a Job can call the inner one-task function directly.

4.3 — Token minting endpoint

New endpoint POST /tokens/mint that issues a short-lived bearer token scoped to one task/lease. Used by the runner to give each Job a least-privilege credential. Token includes:

  • task_id and lease_id claims
  • exp matching the Job's activeDeadlineSeconds
  • agent_id of the runner replica (for command-audit attribution)

Validated by mac-api's existing auth middleware with a new scope task:execute that allows only evidence/lease/transition writes for the named task.

4.4 — Manifests

  • deploy/k8s/mac-runner/deployment.yaml — runner Deployment with the smallest ServiceAccount that has batch.jobs create/get/list/watch in its own namespace.
  • deploy/k8s/mac-runner/rbac.yaml — Role + RoleBinding.
  • deploy/k8s/mac-runner/serviceaccount.yaml.
  • ArgoCD Application entry — supplied by operators from their platform-config repo, not shipped here.

4.5 — Tests

  • Unit: runner's "candidate -> claim -> Job spec" pure function tested with a fake K8s client + mocked mac-api.
  • pytest.mark.k8s integration test using kind/k3d that spins up the full stack (Postgres, mac-api, mac-runner) and runs one canary task.

Open questions for Phase 4

  • Per-tenant Job namespaces, or one shared namespace with labels? Likely shared namespace + app.kubernetes.io/instance labels — simpler RBAC, and tenant isolation already exists at the mac-api auth layer.
  • How are role-specific images selected? Today the runtime_environments table holds a manifest JSON with the image digest; the runner reads that field. Pre-Phase-4 work: ensure every role has a runtime_environment_id pointing at a manifest with a pinned @sha256:... digest.

Phase 5 — Operator/controller (shipped MVP)

Implemented as mac-k8s-controller: a singleton-replica Deployment in the mac namespace running a periodic reconciliation loop. Two reconcilers:

  • reconcile_stuck_jobs lists active K8s Jobs labelled app.kubernetes.io/managed-by=mac-k8s-runner. For each Job it asks mac-api for the corresponding task's current lease and state, and deletes the Job when ANY of: the task is in a terminal state (completed/failed/cancelled); the task's active lease id differs from the Job's mac.lease.id label (re-claimed by another runner); the task has no active lease; or the lease's leased_until is in the past. This is the cleanup path for evicted/OOMKilled Jobs whose parent task already moved on.

  • reconcile_provisioning_requests lists open agent_provisioning_requests and dispatches each one to an injected WorkerPoolScaler. The reference K8sDeploymentScaler bumps the replica count of a Deployment named mac-worker-<role-slug> by delta_per_request (default 1), capped by max_replicas_per_role. Disabled by default (MAC_CONTROLLER_SCALER_ENABLED=0) so operators opt in once their worker-pool Deployments follow the naming convention.

CRDs are intentionally NOT introduced in the MVP — the goal text says "optionally introduce CRDs" and the controller ships against any cluster without custom resources. A follow-up can promote agent_provisioning_requests and rollouts to first-class CRDs once the API surface stabilises.

RBAC (deploy/k8s/mac-controller/rbac.yaml): scoped Role on the mac namespace with batch.jobs delete + apps.deployments/scale patch. No cluster-wide rights.

Phase 5 status

Goal item Status
Watch provisioning requests Done (poll-based)
Scale worker pools Done (reference scaler, opt-in)
Reconcile stuck Jobs Done
Optionally introduce CRDs Deferred (out of scope per "optionally")

Risk register (Phase 3)

Risk Mitigation
? translation breaks on strings containing ? inside literals Quote-aware translation + ≥20 fixture unit tests
events view JSON shape drifts between backends Parity test that round-trips each underlying row type
Service code accidentally calls _conn directly grep for _conn references; assertion in Phase 3.1 PR
Test suite slows down with real Postgres Marker-gated; SQLite remains default
Live-data migration corrupts rows on column mismatch Migrate-script reads schema from each side, fails loud on shape drift