Repository documentation

0.7.0 release evidence closure — one hierarchical ledger over every stage and host dialog

• Status: historical
• Owning queue item: RELEASE-EVIDENCE-001
• Completion / archival evidence: Stage 5 implementation commits

Why this document exists

The attempted 0.5.2 release failed and left no usable record of why. Reconstructing it afterwards found only unrelated leftovers: pip traceback bytecode and litai-driver-failure-*.log files written ~1:51pm for RebuildSourceCacheDerivationManifest.lifecycle_plan_identity. The actual ~3:12pm host-E2E samples crash was unrecoverable, and the sample scratch that held its generated source, build output, and runtime transcripts was already deleted.

That is not bad luck; it is the current design. Every layer of the release stack deliberately discards or anonymizes its evidence:

scripts/fanout_samples.py is the one component that already behaves correctly: it writes a per-worker log, always retains failed workspaces, and has --retain-workspaces for successful ones. It is the prior art this plan generalizes.

The consequence is the one Jordan named: an LLM (or a human) triaging a failed release has to troll a large filesystem for correlated timestamps, and usually loses. The fix is not more logging. It is a closure: one hierarchical, pointer-based record where release is the outermost node and every sub-workflow, stage, gate, sample, variant, and host dialog beneath it records its own pins and the paths to its own retained output.

The model

One ledger per outermost operation

A single evidence run is created by the outermost litai operation (normally release check, but equally rebuild, build, or samples when invoked directly) and inherited by everything it spawns:

_build/evidence/
  20260825T211200Z-9f3ac1/          # <utc>-<short digest of the root operation identity>
    ledger.ndjson                   # append-only, flushed per record: the authority
    index.json                      # derived closure, rewritten on completion *and* on failure
    n0001/ … n0042/                 # one short directory per node (Windows MAX_PATH safe)
      stdout.log stderr.log         # retained transcripts, redacted
      pins.json outputs.json        # this node's exact identities and pointers
    latest -> ../20260825T211200Z-9f3ac1   # pointer file on Windows, symlink elsewhere

ledger.ndjson is append-only and flushed per record precisely because the 0.5.2 run died mid-gate: any format that is only valid once written completely is a format that is absent exactly when it matters. index.json is derived and rebuildable (litai release evidence index).

Node schema

New module src/literate_ai/evidence_ledger.py, new schemas literate-ai/evidence-ledger-node@1 and literate-ai/evidence-ledger@1 registered in schemas/v2 and schema_catalog (with compatibility.json and the migration characterization baseline updated in the same change):

{
  "schema": "literate-ai/evidence-ledger-node@1",
  "node_id": "n0031",
  "path": "release/gate/release-check/samples/hello-component/python/host-e2e",
  "parent_id": "n0030",
  "operation": "sample.host-e2e",
  "state": "failed",
  "host": {"kind": "local", "platform": "linux-x86_64"},
  "started_at": "2026-08-25T21:12:03Z",
  "ended_at": "2026-08-25T21:12:47Z",
  "duration_ms": 44118,
  "pins": {
    "plan_identity": "urn:…", "policy_identity": "urn:…",
    "prepared_revision": "sha256:…", "lifecycle_driver_identity": "sha256:…",
    "component_lock_identities": ["urn:…"], "source_cache_key": "urn:…",
    "flavor_selectors": ["flavor://literate-ai/lang-python"],
    "coding_cli": "claude", "model": "…", "wheel_digest": "sha256:…"
  },
  "outputs": [
    {"role": "stdout", "path": "n0031/stdout.log", "bytes": 91233,
     "digest": "sha256:…", "media_type": "text/plain", "retention": "retained"},
    {"role": "scratch", "path": "/tmp/litai-samples-9f3ac1/hello-component/python",
     "bytes": 20481223, "digest": null, "media_type": "inode/directory",
     "retention": "retained"},
    {"role": "diagnostic", "path": "n0031/driver-failure.log", "…": "…"}
  ],
  "error": {"code": "sample.host_e2e_failed", "message": "…", "excerpt": "…"},
  "children": ["n0032"]
}

States are pending, running, passed, failed, skipped (checkpoint-resumed or sentinel-current), and unavailable (the node could not be observed — a crashed child, a lost worker). unavailable is load-bearing: a run that dies must still say where it died. Pointer retention is retained, pruned, host-only (still on a worker), or imported (transferred into coordinator custody).

Attachment without a central registry

Two environment variables carry the ledger across every process boundary: LITAI_EVIDENCE_RUN (the run root) and LITAI_EVIDENCE_PARENT (the parent node id). Any child that knows about the ledger opens the same run and appends beneath its declared parent; recursive make gates, run_samples.py, the lifecycle driver, and worker launchers all inherit it for free.

Crucially, coverage does not depend on the child cooperating. A single wrapper — evidence_ledger.record_subprocess(...), built on the existing diagnostics.trace_subprocess redaction — creates the node, tees the child's stdout/stderr to both the terminal and n####/std*.log, and records argv, exit status, and duration. An uninstrumented step is therefore still represented by a node with a complete transcript from its parent's point of view. Deeper instrumentation adds pins and children; it is never a prerequisite for having evidence at all.

Triage surface

One command is the LLM's entry point:

$ litai release evidence explain            # defaults to the latest run
release … failed
  gate release-check (target: local, worker w-linux-01) … failed
    samples … failed
      hello-component/python/host-e2e … failed  sample.host_e2e_failed
        pins: lifecycle_driver sha256:2022e513…  lock urn:…  cli claude  model …
        stderr  _build/evidence/…/n0031/stderr.log (91 KiB)
        scratch /tmp/litai-samples-9f3ac1/hello-component/python (20 MiB, retained)
        driver  _build/evidence/…/n0031/driver-failure.log (412 KiB)

explain prints the root-to-failure path only, bounded to roughly one page: passing siblings collapse to a count. Companion subcommands are index (rebuild index.json from the NDJSON), show <node> (one node's complete record), and prune (retention-budget reclamation). litai release check prints the evidence root when it starts through the existing Python progress/stderr mechanism and renders explain on failure before returning the existing CLI failure; no Makefile shell logic is needed. Exit codes and existing stdout contracts are unchanged.

Retention: the 0.5.2 fix

Retention is the part that actually recovers the crash Jordan lost, and it is a change in policy, not just in recording:

• Delete on success only, and only after the ledger records the success. Every TemporaryDirectory / rmtree(ignore_errors=True) in the release path becomes a run-scoped directory removed on the passing path exclusively: sample_runner.main, installed_project_smoke.py, installed_e2e_gate.py, installed_roundtrip.py, wheel_smoke.py, and fanout_samples.py's local run root (already correct for failures).
• Sample scratch keeps its own root. _external_runtime_root requires sample scratch to live outside the repository, so it cannot move under _build/evidence/. It instead gets a deterministic, run-scoped location (LITAI_SAMPLE_RUNTIME_ROOT, defaulting to <tmp>/litai-samples-<run>/…), is retained on failure, and is referenced from the ledger by absolute path with retention: retained.
• The driver failure log stops being anonymous. _retain_driver_diagnostic writes into the active node's directory under a deterministic name; the OS-temp mkstemp path remains only as the no-ledger fallback. This alone converts the "litai-driver-failure-*.log from ~1:51pm" archaeology into a named child pointer.
• In-memory diagnostics get flushed at the raise. standard_local.py and its siblings write failure_diagnostics to n####/lifecycle-failures.json where they are currently only stored in a dict, keyed by the same component revision URIs.
• Digests keep their bytes. Wherever evidence today records stdout_digest / stderr_digest alone, it records a pointer and the digest.
• Bounded disk. litai release evidence prune plus a configurable retention budget (oldest runs first, failed runs last) keeps _build/evidence/ from growing without limit; make clean never removes a failed run's evidence, make really-clean may.

Host and target dialogs

Host dialogs are first-class nodes, not a footnote, because in the 0.5.2 shape the failure was a host dialog:

• legacy-local / local host: the gate subprocess node with full retained transcripts, plus per-gate child nodes from run_checkpointed_gates.py.
• --target local (SSH worker fleet): one node per worker recording the worker identity, the verified workspace revision, the connection/probe dialog, the bounded control envelope verbatim, and the custody bundle. This preserves the existing boundary from skills/agent/bind-lifecycle-evidence/SKILL.md exactly: SSH stdout stays a bounded ≤1 MiB control plane, complete evidence travels in the digest-bound bundle. Remote files appear first as retention: host-only pointers naming the worker, then flip to imported with local paths once transferred and verified — and worker attempt custody is deleted only after the ledger records the acknowledged import.
• --target github: in addition to run_id/run_url/conclusion, the failing job's logs are captured (gh run view --log-failed) into the node so a CI failure is not a URL-only dead end from a machine that may not be able to reach it later.
• samples-platform-regression fan-out: fanout_samples.py's existing per-worker logs and retained workspaces become ledger pointers, one node per target, carrying source_identity / materialized_source_identity / transport_identity as pins.
• Endpoint hygiene: worker endpoints follow the existing privacy rule — the ledger records the configured worker_id and a digest of the endpoint, never a raw private hostname, and every transcript passes through the diagnostics redaction before being written.

Blind steps: shell and Make surfaces become instrumented Python

A step that runs outside a Python harness cannot record pins or retain output, so the remaining shell/Make surfaces are where evidence coverage stops. Scoped for 0.7.0 (tracked as RELEASE-EVIDENCE-002):

• Makefile gate recipes stay the user-facing entry points but delegate to python -m literate_ai.harness <gate> so each gate is a ledger node with pins, rather than a bare recipe whose only artifact is $?. Recipes remain readable one-liners; the instrumentation lives in Python where it is testable.
• scripts/find-compatible-python.sh / .cmd — invoked from $(shell …) at Makefile parse time, so a resolution failure today yields an empty PYTHON and a confusing downstream error. The interpreter choice is a pin and belongs in the ledger; the POSIX/Windows pair is replaced by one Python resolver with a minimal shell shim retained only for the bootstrap case where no interpreter is known yet.
• scripts/litai-launcher / .cmd — 6 and 5 lines of blind exec; folded into the same harness entry point.
• .github/workflows/ci.yml run: steps — the shell steps that are not already a Python script get the same treatment, and CI uploads the evidence run as an artifact so a CI-side release failure is triageable with the identical explain command.
• Out of scope: docs/presentations/**/regenerate*.sh and .claude/hooks/*.sh are not in the release path.

Correlation with existing records

The ledger indexes existing evidence; it does not replace or re-implement any of it, and it never becomes attestation.

• prepared.json gains a required evidence object (run_id, root, index_path, node_id), which is a schema bump to literate-ai/prepared-release@2 with @1 still accepted for read. It deliberately has no ledger_digest: the append-only ledger changes when the root closes after the prepared record is written, so a digest in that record would be unverifiable.
• The publication receipt references the same run.
• PerformanceSpan.extra gains evidence_node_id, so timing telemetry joins to the node that produced it. Perf spans stay best-effort diagnostics.
• diagnostics.operation_log sinks are opened per node directory, so operation/subprocess NDJSON arrives already partitioned by stage instead of as one blob.
• RELEASE_CHECKPOINT records the node id of each completed gate so a resumed run keeps one ledger and resumed gates keep their original evidence, recorded as state: skipped with a pointer to the earlier node. The checkpoint stays disposable recovery state, never release attestation.

Stages

Each stage is independently landable and leaves the tree green.

1. Foundation. evidence_ledger.py, both schemas, the environment handoff, the record_subprocess wrapper, and litai release evidence explain|index|show|prune. No existing gate changes behavior.
2. Release spine. check_release emits the root node and the target/gate/worker nodes for legacy-local, local, and github; transcripts are retained on both paths; prepared-release@2; run_checkpointed_gates.py node mapping; Python surfaces the evidence root and renders the failure explanation on stderr rather than changing Makefile shell logic.
3. Host dialogs and retention. Sample runner scratch retention with per-sample / per-variant / per-phase nodes; _retain_driver_diagnostic relocation; failure_diagnostics flush; installed-e2e / install-check / installed-project-e2e / wheel-check / roundtrip-host retention; fan-out pointer wiring.
4. Blind-step harness. RELEASE-EVIDENCE-002's Make/shell replacement and the CI evidence artifact.
5. Acceptance. The 0.5.2 regression: inject a host-E2E sample failure under make release and assert that a single explain invocation names the failing sample, variant, and phase, its pins, and retained scratch plus complete transcript paths that still exist after the run exits — the precise thing that did not exist on the 0.5.2 branch.

Implementation evidence

• Foundation and correction work landed in dca9144a and aa5a3357.
• The release spine and prepared-release compatibility landed in c61abd4b.
• Host-dialog retention and fan-out evidence landed in 4c34708b and e26c0b65.
• The blind-step harness and CI evidence artifact landed in f62e04ac.
• Stage 5 default scratch lifetime landed in 1bd3c235; post-exit acceptance, signal normalization, checkpoint wrapping, and CI explanation invocation landed in b310fc74.

Acceptance contract

• A failed release is triageable from one litai release evidence explain output, bounded to roughly one page, with no filesystem search.
• Every node records the exact pins it used and pointers to its own outputs; no stage in the release closure lacks a node.
• No release-path failure deletes its scratch, transcripts, or diagnostics.
• Digest-only evidence no longer exists in the release path.
• Remote evidence still honors the bounded control envelope and digest-bound custody rules, with worker cleanup gated on acknowledged import.
• No transcript contains a credential value or a raw private endpoint.
• A run killed mid-gate still yields a valid ledger whose deepest node is running or unavailable.

Out of scope for 0.7.0

Any new remote transport; GitLab release dispatch (still fail-closed per RELEASE-005); an external log-aggregation or tracing service; replacing perf.py; making the ledger load-bearing authority for build authorization or acceptance. The ledger is a triage index over evidence that other records already own.

Risks

• Disk growth from retained scratch — mitigated by the retention budget, prune, and retaining failures preferentially over successes.
• Windows path length (WINDOWS-PATH-001) — mitigated by short n#### node directories, with the human-readable node path stored inside the record.
• Secret leakage into newly persisted transcripts — mitigated by routing every write through the existing diagnostics redaction, with an explicit test.
• Self-hosting churn: these changes touch src/literate_ai/ and the declared lifecycle-driver implementation paths, so each stage refreshes the TCB pin (make driver-review-record) and the documentation marker (make documentation-review).