Dreaming, rewritten¶
mac.dreaming replaces dream_scanner, dream_cycle_classifier and
dream_repair_tasks. This note records why, so the next person does not
rebuild the thing that was removed.
What the old cycle did, measured¶
Live hub ledger, 2026-07-28, after 40 days of production:
| metric | value |
|---|---|
dream:failure_pattern rows |
154,273 |
| distinct summaries among them | 4,414 (a 35:1 duplication ratio) |
nap_summary rows |
154,324 (4,540 distinct bodies) |
| share of the entire memory store | 97.6% |
| artifacts recording something that worked | 0 |
| raw evidence that was a success | 1,443 of ~7,020 (21%) |
findings at high confidence |
111,031 — none of which triggered any action |
| repair tasks filed | 1,259 |
| repair tasks completed | 4 |
| dream artifacts actually embedded (i.e. retrievable) | 253 |
Five investigations in this directory triaged individual findings. All five returned NO LIVE DEFECT or NOT ACTIONABLE; two failed specifically because the sole evidence was a success record that the pipeline had labelled a failure.
The root cause¶
The upstream feature this was modelled on is memory curation: read a memory store plus past transcripts, emit a new, smaller store with duplicates merged and stale entries replaced, leave the input untouched, and attach the output to future sessions.
What got built was a defect scanner: regex over rows, append findings to the
live store, file bug tickets. The word "failure" does not appear anywhere in
the upstream design. That single substitution explains every symptom — wins are
invisible to a defect scanner, defect signatures are repo-specific so nothing
generalised beyond mac, and a scanner has no reason to ever remove a row.
The deepest inversion: dreams exist to fix a store that accumulates duplicates and stale entries. The old cycle became, by a wide margin, the largest producer of exactly that.
What the rewrite does¶
Pipeline (mac.dreaming.engine.dream):
Five things are different in kind, not degree:
-
Copy-on-write.
save_runonly ever writesdream_runs/dream_candidate_entries.memory_recordsis touched by exactly one function,promote_run, and only after gates pass and someone asks. A bad dream is discarded, not cleaned out of live memory afterwards. -
Shrinking is a hard gate.
compression_gatequarantines any run whose output exceedsmax_output_ratio(default 0.75) of its input. A dream that grows the store cannot be promoted. Promotion also retires the rows each candidate supersedes, so adoption is net-negative on row count. -
Wins are first-class.
MemoryKind.PRACTICEis a peer ofPITFALL.balance_gatereports the mix on every run so a regression to failure-only output is visible immediately rather than after 40 days. -
Confidence means corroboration.
MemoryCandidate.source_countcounts distinct origins, so three references to one session is one source. The old scorer counted rows, and the consolidator wrote a fresh row per pass, so re-reading one observation three times scoredhigh. -
Conversations are judged as wholes.
SessionReflectionrecords the inferred objective and whether it was met, abandoned, derailed or left unresolved — the question the old cycle had no representation for.
Plus: extraction is a model call through the existing router seam
(resolve_model_caller), the heuristic fallback reads only structured
outcome fields and never keyword-scans prose, project comes from the data
rather than a hardcoded mac.* table, the pipeline's own output is excluded
from its input so it cannot feed on itself, and promoted memories are rendered
into the executor prompt by recall_deployment_lessons — the loop the old
artifacts never closed.
Gates¶
provenance_coverage, contradiction_reduction, privacy,
retrieval_quality, compression, win_balance (advisory).
The old implementation built exactly one of these — the privacy filter — and
built it well. Its regexes are carried over near-verbatim in
mac.dreaming.redact. The three quality gates it skipped are the three that
would have caught the 154,273 duplicates.
Bounded history¶
The nap cycle runs a dream per agent per nap, so the run tables would grow
without limit — the same failure this rewrite exists to fix, one level up.
prune_runs keeps the most recent N per state (200 promoted, 50
ready-for-review, 50 quarantined, 20 discarded) and deletes their candidate
entries with them, and run_dream_cycle calls it every pass. Promoted runs are
kept longest because they are the provenance for memories that are live.
Verified against real data¶
1,200 live deployment_learning rows through the heuristic extractor:
1200 in -> 171 out (14%) all six gates pass
kinds: 5 practice / 166 pitfall
supersessions queued: 326 (promotion would be net -155 rows)
The heuristic path produces thin practice statements because it can only read structured fields; substantive wins need the model path. That is a known limit of the fallback, not of the design.
Promotion¶
run_dream_cycle auto-promotes by default: a run that cleared every gate is
adopted immediately, writing dream_memory:* records and retiring the rows they
supersede. A quarantined run is never promoted — it stays readable for
inspection. Set MAC_DREAM_AUTO_PROMOTE=0 and restart to turn this off on a
live fleet without a redeploy.
Retirement is the only irreversible step in the pipeline, and under
auto-promotion it runs unattended, so promote_run caps deletions at
MAX_RETIRE_PER_RUN (500). Hitting the cap halts retirement, reports
retire_capped: true, and still completes the promotion — one malformed run
cannot quietly empty the store.
Auto-promotion is what closes the loop: promoted memories are rendered into the
executor prompt by recall_deployment_lessons. Without it the pipeline produces
candidates that nothing ever reads — which is where it sat for its first 50
production runs.
Operating it¶
mac admin dream run --project mac --limit 2000 # curate; writes a candidate store
mac admin dream show <run_id> # gates, stats, candidates
mac admin dream promote <run_id> # adopt; retires superseded rows
mac admin dream discard <run_id> --reason "..." # throw it away
The nap cycle calls run_dream_cycle automatically and never auto-promotes.