Audit: prove deploy prerequisites before phase-1 mutation, preserve Python diagnostics¶
Investigation-only ground truth for the repair task "Prove all deploy prerequisites before phase-1 mutation and preserve Python diagnostics". No production behavior is changed by this document; it specifies the exact edits and test assertions the follow-on repair must make.
Finding A — phase-1 worker mutation precedes prerequisite proof¶
In deploy/deploy-mac-fleet.sh, run_typed_cohort() runs the phases in this
order:
run_bounded_node_phase "$selected_specs_file" phase1-prepare typed_phase1_prepare_workerrun_bounded_node_phase "$selected_specs_file" prerequisites typed_prerequisite_workerrun_bounded_node_phase "$selected_specs_file" stage-bundle ...
typed_phase1_prepare_worker calls prepare_remote_phase1_restore_contract,
which acquire_remote_deployment_lock, uploads the phase-1 quiesce helper /
daemon-functions files to the worker, and runs bash "$helper" prepare on the
remote worker. That is a real per-worker mutation (deployment lock acquisition
plus remote helper execution that arms the restore contract).
typed_prerequisite_worker (prepare_remote_prerequisite_bundle) is the
read-only prerequisite proof. It runs AFTER the phase-1 prepare mutation.
Therefore phase-1 mutation currently begins before all read-only prerequisites are proven, violating the requirement to "prove all deploy prerequisites before phase-1 mutation".
Additionally, hub_dispatch_hold_transition_available() (deploy-mac-fleet.sh
~line 3406) is defined — the comment at ~line 3409 states the intent to "prove
the POST operation itself before phase 1 mutates any worker" — but it has NO
call site anywhere in the script. hub_dispatch_hold_cas_available() is the
only availability gate actually invoked (in prepare_remote_mac_agent_deployment).
So the transition-batch prerequisite is never proven before phase-1 mutation.
Required source edits (Finding A)¶
- Invoke the read-only prerequisite proof before any phase-1 mutation. In
run_typed_cohort(), move theprerequisitesbounded phase (and the never-invokedhub_dispatch_hold_transition_availablegate) ahead of thephase1-preparebounded phase / journalphase1-prepare-startmutation so that read-only prerequisite proof and the transition-batch POST availability check both complete beforetyped_phase1_prepare_workeracquires any remote deployment lock or runs the remotepreparehelper. - Wire
hub_dispatch_hold_transition_availableinto the pre-phase-1 gate (mirroring howhub_dispatch_hold_cas_availableis checked), failing closed when the hub only exposesrelease-batchand lacks the distinctPOST /agents/dispatch-hold/transition-batchoperation. - Keep
assert_prerequisite_remaining_budget(usingMAC_DEPLOY_PREREQUISITE_PHASE_BUDGET_SECONDS, default 2400, andMAC_DEPLOY_PREREQUISITE_APPLY_GUARD_SECONDS, default 120) applied so the reordered prerequisite phase still enforces the remaining-time budget before mutation.
Required test edits (Finding A)¶
tests/test_deploy_fleet_parallel_staging.py::test_parallel_typed_barriers_keep_wal_parent_owned_and_orderedcurrently codifies the buggy order: itsorderedtuple placesrun_bounded_node_phase ... phase1-prepareandcohort_journal_mutate phase1-armedBEFORErun_bounded_node_phase ... prerequisites. Update this tuple soprerequisitesprecedesphase1-prepare/phase1-prepare-start, and keep thepositions == sorted(positions)assertion.- Add an assertion (here or in
tests/test_deploy_hold_adoptions.py) thathub_dispatch_hold_transition_availablehas a real call site inrun_typed_cohort/mainand that its position precedes the first phase-1 mutation (phase1-prepare-startjournal write /typed_phase1_prepare_worker). tests/test_deploy_hold_adoptions.pycurrently only asserts the availability helper's body inspects/agents/dispatch-hold/transition-batchand itspost; extend it to assert the helper is invoked before phase-1 mutation.
Finding B — Python helper diagnostics swallowed on unexpected failure¶
deploy/fleet-node-phase1-quiesce.sh embeds Python blocks whose top-level
except Exception handlers discard the exception message and traceback,
printing only the class name:
- ~line 2952:
print("phase-1 quiescence failed unexpectedly: %s" % type(exc).__name__, ...) - ~line 3744:
print("phase-1 quiescence failed unexpectedly: %s" % type(exc).__name__, ...)
On an unexpected failure an operator sees e.g. "phase-1 quiescence failed
unexpectedly: KeyError" with no message and no traceback, making remote
diagnosis impossible. The curated QuiescenceFailure / ReceiptFailure
branches already print a useful message and must stay; only the unexpected
fallback loses information.
Note: the raw child-process (systemctl / launchctl / manager) stderr captured by
run_bounded() is intentionally suppressed to avoid leaking secret-bearing
command output (see reject_secret_or_raw_output and the
test_systemd_inspection_error_fails_closed_without_raw_output contract). The
fix must preserve the Python-level diagnostic (message + traceback.format_exc()
of the helper itself) WITHOUT reintroducing raw child stderr.
Required source edits (Finding B)¶
- In both
except Exception as exc:fallbacks, print the exception message and a full Python traceback (e.g.traceback.format_exc()), not justtype(exc).__name__, to stderr beforeraise SystemExit(1). Addimport tracebackwhere needed. Keep the existing best-effortoutput_path.unlink()cleanup in the receipt block. - Do NOT surface raw child-process stdout/stderr; only the helper's own Python message/traceback.
Required test edits (Finding B)¶
- Add a test in
tests/test_fleet_node_phase1_quiesce.pythat forces an unexpected (non-QuiescenceFailure) exception in each Python block and asserts the helper stderr contains the exception message and traceback frames (e.g. the exception type AND aTraceback (most recent call last)line and the helper file/function), while still asserting no secret / raw child output leaks (mirroringtest_systemd_inspection_error_fails_closed_without_raw_output).
Verification¶
- Canonical command:
scripts/run-contract-tests.sh. - Targeted while iterating:
.venv/bin/python -m pytest tests/test_deploy_fleet_parallel_staging.py tests/test_deploy_hold_adoptions.py tests/test_fleet_node_phase1_quiesce.py -q.
Out of scope / not implicated¶
src/mac/diagnostics.pyis the read-only control-plane diagnostics registry (mac admin diagnostics); it already isolates a raising check into anerrorFinding and is unrelated to the phase-1 helper stderr loss. No change needed.src/mac/deploy_service.pyrecords deploy/dependency metadata; it does not run the phase-1 helper or handle its stderr. No change needed.