Skip to content

Crash diagnosis

MAC fleet agents run beneath a supervisor-independent crash observer. The observer is copied to ~/.mac/bin/mac-crash-observer and deliberately uses the managed standalone Python 3.14.7 standard library instead of the MAC virtual environment. Native installation refuses an absent or mismatched managed interpreter. A broken MAC package environment therefore cannot disable its own crash reporter; the observer does not depend on the system Python selection.

The native service manager remains responsible for restart:

  • systemd uses Restart=always and launches the observer with --supervisor systemd;
  • launchd uses KeepAlive and --supervisor launchd;
  • supervisord uses autorestart=true and --supervisor supervisord;
  • the Kubernetes orchestrator Deployment uses --supervisor kubernetes and a pod-local spool volume that survives container restart.

Evidence lifecycle

Before starting the worker, the observer enables PYTHONFAULTHANDLER, unbuffered stderr, and core dumps where the host permits them. It forwards termination signals to the child, tees a bounded stderr tail, and distinguishes intentional supervisor shutdown from unexpected exit.

For an unexpected exit it records:

  • agent, process, supervisor, PID, exit code, and signal;
  • source commit and tree;
  • active task and lease when the hub remains reachable;
  • fatal Python/native stack output and a bounded stderr tail;
  • core path or systemd-coredump metadata/backtrace;
  • disk, load, CPU, and child-resource observations.

Filesystem cores are hard-linked or copied into ~/.mac/crashes/<event-id>/core with mode 0600. Retention defaults to the three newest cores and refuses to copy a core larger than 512 MiB. Configure those bounds with MAC_CRASH_CORE_RETAIN_COUNT and MAC_CRASH_CORE_MAX_BYTES.

Every report is first written atomically to a mode-0600 JSON file beneath MAC_CRASH_SPOOL_DIR (default ~/.mac/crash-spool). Successful hub delivery removes it; the next observer invocation replays any files retained during a hub outage. event_id makes replay idempotent.

Hub API

An agent-bound token submits its own evidence:

POST /agents/{agent_id}/crash-reports

Operator projections are:

GET  /crash-reports
GET  /crash-reports/{report_id}
POST /crash-reports/{report_id}/resolve

The list projection requires read. Full occurrence content requires secret because stderr can contain private application output. Resolution requires admin.

The hub redacts common credentials before persistence and computes the fingerprint itself from the source revision, process, termination class, and a normalized stack signature. Every occurrence remains durable while matching events share one active incident.

Operator response

The hub records crashes; it does not repair them. The first occurrence of a fingerprint opens an incident and records one agent.crash.observed operator notification. Further occurrences of an open incident are retained and counted without another notification. An occurrence after the operator resolved the incident reopens it and notifies again.

No task is filed. Earlier releases filed a P0 repair task per incident under the crash-observer actor and re-filed it after each failed attempt; that autonomous repair loop was removed on 2026-09-30. Decide what to do from the report and POST /crash-reports/{report_id}/resolve once the fix is verified.

Verification

A deployment is conformant only when its installed service command contains mac-crash-observer, the observer can post a synthetic non-zero child exit, and the hub retains the occurrence. Restart policy alone is not crash-diagnosis conformance.