Trace Review Bot — Status

Session-trace miner + weekly lesson review for the Hermes fleet. Created 2026-10-02 by jimmy (Hermes profile jimmy, Windows 11). Purpose: turn raw session traces into evidence-backed lesson proposals and deliver them to Tony's Telegram — instead of leaving failures buried in chat history.

What it is

Miner (deterministic, no LLM): C:\Users\fight\AppData\Local\HermesHost\diagnostics\trace_review.py
- Scans state.db plus every profiles/*/state.db over a --days window.
- Signals: S1 repeated-error, S2 retry-loop, S3 user-correction, S4 assistant self-correction, S5 repeated-request.
- Every finding carries evidence (session, message_id, timestamp, snippet) and a falsifier; anything it cannot evidence is recorded NOT_AVAILABLE, never filled in.
Reports: ...\diagnostics\trace-review\<UTC-stamp>\ → trace-review.md, trace-review.json, findings.jsonl (ledger used for NEW / RECURRING / ESCALATED).
Lesson pass: the trace-lessons-review skill reads the findings and writes proposals.md into the same folder. It proposes; it never applies.

Jobs

Job	ID	Schedule	Mode	Script / skill
Trace miner (daily)	`6b57b2ed2847`	`30 7 * * *`	no_agent	`profiles\jimmy\scripts\trace_review_run.py`
Trace lessons review (weekly)	`9fb3bd266e71`	`0 8 * * 0`	agent	skill `trace-lessons-review`
Trace digest sender (weekly)	`743bc93e1ff5`	`30 8 * * 0`	no_agent	`profiles\jimmy\scripts\trace_digest_sender.py`

All three run with native cron delivery (deliver: telegram:5789896722) — the gateway delivers the job's stdout. See 01_Wiki/hermes-cron-delivery-gateway-config for the day the delivery looked broken and what it actually was.

743bc93e1ff5 is paused: it was a fallback sender built while delivery looked broken. Kept on disk, resumable if a report ever needs a guaranteed push.

Delivery path

profiles\jimmy\scripts\send_telegram_digest.py — reads the bot token and home channel from the profile .env at runtime, redacts keys/tokens, chunks to Telegram's limit, posts via the Bot API, prints ['ok'] per chunk.
Digests are deliberately snippet-free (signatures + counts only): error lines can carry credentials. Full evidence quotes stay on disk in the report.
trace_digest_sender.py is the safety net: it sends any proposals.md / trace-review.md newer than its .last-delivered.json marker, so a report cannot be written and then never delivered.

Findings so far

2026-10-02, 7-day window: 39 sessions / 4,225 messages / 13 findings — S3 user-correction 10 across 5 sessions; S1 browser_exec timeouts (5 sessions); S1 terminal 180 s timeouts (5); S4 self-correction 5; S1 ripgrep search failures (4); approval-gate BLOCKED (3); plus retry-loop, command not found and path-not-found repeats.

Weekly review 2026-10-02 (...trace-review\20261002T202641Z\proposals.md): 5 proposals covering 6 findings, 7 findings dropped as noise/low-trust.

#	Proposal	State
P1	Miner defect: S3 counted diagnostic questions ("why did X break?") as user corrections	Applied 2026-10-02
P2	`browser_exec` hangs to the 420 s batch deadline ignoring `timeout_s` (7× in one week)	Open — needs Tony
P3	Shell `timeout N` cannot extend the terminal tool's 180 s default	Open — needs Tony
P4	`search_files` can fail wholesale and still report `total_count: 0`	Open — needs Tony
P5	`execute_code` kernel does not inherit the shell environment	Open — needs Tony
P6	`agent-reach` references call `gh` / `mcporter`, neither installed here (exit 127)	Open — needs Tony

Lessons (generalizable)

Hermes delivers notices in the user role. Tool timeouts, delegation results and gateway envelopes arrive as role=user. Any signal computed over user messages must exclude them, or [Command timed out after 180s] counts as a user correction.
Anchored regexes only match the start of a message. ^\s*(…|again\b) never caught "why did telegram break again?"; mid-sentence recurrence complaints needed a separate unanchored pattern.
A signal's own precision is falsifiable. P1 was applied only after checking the 12 matching messages by hand (9 were questions, not corrections) — 44 markers → 40, all user-authored.
A long-running process goes stale across an update. This session started 08:26:41 and the update landed 08:45:11; every from <module> import <name> for a name the update added then failed with "cannot import name", citing a file that does define it. Restart the session (and the gateway) after an update — the update receipt's "mixed sys.modules" warning is the signal.
An agent job can write its report and still skip its own send step. When delivery is native that no longer matters; while it looked broken, delivery had to be a separate deterministic job.

Open items

P2–P6 awaiting accept/reject.
Miner gap: "that is wrong" (no apostrophe) still misses that'?s (wrong|not).

Owner / status

Owner: jimmy (Hermes profile jimmy). Status: active. Created 2026-10-02.

5.2 KiB Raw Permalink Blame History Unescape Escape