HGX elastic capacity¶
MAC has an operator-side controller for adding fungible HGX capacity without weakening the machine-onboarding trust boundary:
$ mac admin hgx capacity status --pending-requests 2 --max-sessions 6
$ mac admin hgx capacity plan --pending-requests 2 --headroom 1 --max-sessions 6
$ mac admin hgx capacity execute --pending-requests 2 --headroom 1 --max-sessions 6
$ mac admin hgx capacity mark-onboarded <session-id> --agent-id <agent-id>
status and plan are read-only. They run hgx list through
mac.hgx_provider.HgxProvider, inspect the durable controller receipt, and do
not create, stop, resume, or delete provider sessions. The explicit execute
command creates capacity. On an HGX-enabled hub, the optional background
autoscaler invokes the same bounded controller from durable provisioning
demand; provider work never runs on a dispatcher or HTTP thread.
HGX authentication¶
HGX is MAC's external capacity provider, not a token-configured MAC provider.
There is no HGX API token to mint, paste, or store in a MAC environment file.
The operating account that will run hgx completes the one-time interactive
browser bounce:
After that login succeeds, HGX commands are ready to use in that account; run
the capacity status, plan, or explicit execute command normally. If an
HGX session later expires or is revoked, repeat hgx login once and retry the
operation. The adapter recognizes the usual unauthenticated HGX errors and
reports this recovery without exposing provider stderr.
Enable the optional autoscaler only where its operating context can already
run the configured MAC_HGX_BINARY using that established HGX login session:
MAC_HGX_AUTOSCALE_ENABLED=1
MAC_HGX_AUTOSCALE_MAX_SESSIONS=10
MAC_HGX_AUTOSCALE_SCALE_UP_STABILIZATION_SECONDS=120
MAC_HGX_AUTOSCALE_SCALE_UP_STEP=1
MAC_HGX_AUTOSCALE_COOLDOWN_SECONDS=300
MAC_HGX_AUTOSCALE_SCALE_DOWN_STABILIZATION_SECONDS=3600
MAC_HGX_AUTOSCALE_SCALE_DOWN_STEP=1
MAC_HGX_AUTOSCALE_SPARE_MIN_AGE_SECONDS=3600
The autoscaler does not perform an interactive login itself. A missing or expired HGX login is observable as a provider error and never falls back to unverified capacity.
Readiness contract¶
Every created session uses the current HGX CLI contract for an explicit
standard-dind instance. It also passes explicit placement and resource
arguments instead of inheriting provider defaults:
The CLI exposes --cluster, --gpu, --memory-gib, and --cpu; cluster
identifiers are restricted to a safe argv shape, GPU is bounded to 0..8,
memory to 8..256 GiB, and CPU to 1..64. The controller records the immutable
provider session ID immediately after the provider accepts the create request.
That request is not readiness evidence.
Before a session is reported as attested capacity, the controller:
- addresses it only by immutable session ID;
- waits for
hgx status <session-id>to stop failing; and - requires
hgx ssh <session-id> -- ...to return the unpredictable nonce generated byHgxProvider.attest_ssh.
An SSH failure is recorded by failure class without raw provider output. A
failed session never counts toward min-ready.
Bounds and cooldown¶
The desired ready count is:
max-sessions bounds all live provider inventory, including unrelated live
HGX sessions, so the controller does not evade a provider-wide safety limit.
However, unrelated sessions are not assumed idle: only controller-created,
nonce-attested, not-yet-onboarded sessions satisfy pending provisioning
demand. Five existing busy workers plus one pending request therefore plans
one new session when max-sessions is greater than five. Existing untracked
workers are not re-attested by the capacity controller.
One execution creates at most max-create-per-run sessions (default one).
cooldown-seconds prevents a later invocation from immediately starting
another step. Provider HTTP 429/quota errors are reduced to the
secret-free provider_quota_exhausted failure class rather than being
misreported as satisfied demand.
Autoscaling curve¶
The automatic curve is intentionally asymmetric:
- a task-bound
dispatch.no_eligible_agentrequest is durable immediately, but it must remain actionable forscale-up-stabilization-secondsbefore it contributes to desired capacity; - each reconciliation creates at most
scale-up-stepsessions; - the controller cooldown spaces later scale-up steps;
- demand returning to zero starts a separate, longer
scale-down-stabilization-secondswindow; and - one scale-down pass retires at most
scale-down-stepold surplus sessions.
Task-bound requests are reconciled before capacity math. Requests for terminal
tasks or tasks already assigned are cancelled instead of becoming phantom
demand. Legacy dispatch/review rows without a task ID are also cancelled
because their liveness cannot be proved. Reviewer shortages and service-role
requests remain available to their matching provisioners but do not create a
generic HGX coding worker. The raw, sustained, ignored-by-reason, and
zero-demand-age counts are emitted as hgx.autoscaler.* observability metrics.
The receipt defaults to ~/.mac/hgx-elastic-capacity.json. Provider mutation
commands create or update it using an fsynced, mode-0600 atomic replacement. It
contains immutable session IDs, timestamps, secret-free outcome classes, and
the next required action. A non-blocking process lock rejects overlapping
execute invocations before either can create capacity.
Provisioning requests and onboarding¶
The durable signal store remains ProvisioningService.list_pending_requests()
(or GET /provisioning/requests?status=pending), but the automatic HGX demand
count is the subset of task-bound dispatch.no_eligible_agent rows that still
refer to an open, unowned task. Do not pass the total store count to the manual
controller: it also contains reviewer and specialized service-role requests
that a generic coding worker cannot satisfy.
The autoscaler registers a wake-only request listener and then polls the
durable rows on its own background thread. The older synchronous
register_provisioner hook remains an auto-fulfillment extension point, but it
must return None until onboarding has registered a real agent. An HGX session
ID is not a MAC agent ID and must never be used to fulfill a provisioning
request.
After nonce SSH attestation, the controller stops and persists a
prepare_fungible_onboarding next action. The operator or a separately
reviewed automation path must supply:
- the fleet agent name;
- the hub agent;
- a reviewed draining/degraded fungible placeholder; and
- endpoint-bound worker credentials.
Those inputs feed deploy/deploy-mac-fleet.sh --prepare-fungible-onboarding. The capacity
controller does not invent credentials or silently mark provisioning requests
fulfilled.
Once onboarding has registered the real agent and the corresponding provisioning request is fulfilled, consume the capacity receipt explicitly:
$ mac admin hgx capacity mark-onboarded <immutable-session-id> \
--agent-id <registered-mac-agent-id>
This is an idempotent receipt-only mutation; it performs no provider action.
The onboarded session continues to consume the provider-wide
max-sessions/quota bound, but it no longer counts as available supply for a
future pending request.
Retirement¶
Automatic retirement is restricted to controller-created sessions that never
became registered MAC agents. They must be surplus after the scale-down
stabilization window and older than spare-min-age-seconds; deletion proceeds
one bounded step at a time using the immutable session ID.
An onboarded session is never automatically deleted by this controller.
Retiring a registered worker also requires draining its leases, revoking its
worker credential, tombstoning the agent, and updating the authoritative
fleets.yaml record. Until that multi-resource lifecycle transaction exists,
the autoscaler fails closed at this boundary instead of leaving MAC believing a
deleted provider instance is still a live worker.