vault-sync: 2026-10-06 00:00 (jimmy)
This commit is contained in:
jimmy
@@ -0,0 +1,9 @@
|
||||
# 2026-10-05 — RUDR9 installed on Hermes hub
|
||||
|
||||
- Installed RUDR9 (9-role AI engineering org) at commit `e69e16e` on the real hub `C:\Users\fight\AppData\Local\hermes` (HERMES_HOME overridden; installer default `~/.hermes` would be phantom).
|
||||
- 7 new profiles: planner, architect, vcm, builder, security, performance, reviewer. Default = CTO. Guard plugin enabled; long-task hook installed; kanban configured (orchestrator=default).
|
||||
- Default SOUL backed up (`SOUL.md.rudr9-backup`); config snapshot `config.yaml.rudr9-preinstall-20261005-223958.bak`.
|
||||
- Model: all 8 profiles → OpenRouter (`or`) `deepseek/deepseek-v4-flash-0731` — one model for the whole team (all local endpoints were down). Smoke test passed (builder → SMOKE-OK).
|
||||
- Context7 MCP enabled on architect+builder (verified).
|
||||
- **Blocked:** GitHub MCP + VCM need a GitHub PAT (none on machine). Add to `Documents\Vault\tokens_system.txt` as `GITHUB_PERSONAL_ACCESS_TOKEN=...` then wire to default/vcm/reviewer .env + enable.
|
||||
- Full record: `04_projects/rudr9/status.md`.
|
||||
@@ -5,6 +5,7 @@
|
||||
|
||||
| Date | Lesson | Context |
|
||||
|------|--------|---------|
|
||||
| 2026-10-05 | Bash-based Hermes installers (RUDR9) default `HERMES_HOME=$HOME/.hermes` -> silent phantom tree on Windows; always export `HERMES_HOME="C:/Users/fight/AppData/Local/hermes"` before running. Long installers with many `hermes config set` calls exceed the 180s terminal timeout - resume after timeout (phases are idempotent) and verify every artifact afterwards. v0.21 `profile create --clone` clones from the ACTIVE profile (v0.18 cloned default) - pin with `HERMES_PROFILE=default`. One model serves all role profiles (roles != model subscriptions); if a cloned profile's model endpoint is down, re-point via `hermes -p X config set model.provider/model.default`. Full record: `04_projects/rudr9/status.md`. | RUDR9 install (2026-10-05) |
|
||||
| 2026-09-25 | OMH (oh-my-hermes) install on Windows: the PowerShell remote-installer pattern (fetch install.ps1, run iex) is flagged by the command-safety gate and just times out on Telegram (no approval UI). Use the documented agent path: pin the SHA with `git ls-remote https://github.com/rlaope/oh-my-hermes.git refs/heads/main` (never `main`), then `uv tool install "git+https://github.com/rlaope/oh-my-hermes@$SHA"` (local build, no remote-exec, does not trip the gate), then `omh --hermes-home "C:\Users\fight\AppData\Local\hermes\profiles\leonard" setup --no-interactive --yes --no-omh-tui`, then `omh doctor` (target 54/54 passing, 0 blocking). `omh` is a native binary: pass Windows `C:\...` paths, never MSYS `/c/...`. Non-interactive setup does not touch model aliases. | OMH install (2026-09-25) |
|
||||
| 2026-09-25 | Native-tool path discipline + pre-delete verification (near-miss during OMH setup): I passed an MSYS path `/c/Users/...` to the **native** `omh.exe`, which read it literally as a phantom `C:\c\Users\...`; that phantom root pre-existed and also held a `models/` subtree (git research repo, caches, GGUF weights) that I did not recognize when I `rm -rf`'d the phantom. Real data survived only because the authoritative `C:\models` (git clean at remote HEAD; all weights present; public remote sudoingX/bonsai2-small-gpu) was a separate tree. Rules: (a) pass `C:\Users\...` to native tools, never MSYS `/c/...`; (b) before deleting a dir, list its contents and confirm no pre-existing real files; (c) after any write, verify the intended target actually gained the expected files; (d) if a phantom root (`C:\c`) appears, check whether it shadows a real `models/` clone first. | OMH install side-effect (2026-09-25) |
|
||||
| 2026-09-19 | Bonsai 2 27B fast build (sudoingX fork: kernel PR #218 + Hadamard #217, MTP head graft) runs ~2x stock: fresh tg128 39.3 tok/s (vs 26.3), streaming ~49 tok/s (code 53.4), greedy output BYTE-IDENTICAL to spec-off (lossless). Windows-native CUDA build BLOCKED on this PC: MSVC 14.44/14.51 crashes CUDA 12.4 cudafe++, no admin for CUDA 13/old MSVC, pip nvcc wheels ship no nvcc.exe on Windows. Build+serve in WSL2 (gcc-13 + CUDA 12.4 apt; GPU passthrough native) at 127.0.0.1:8899; model in WSL ext4 /opt/models, bind --host 0.0.0.0 (127.0.0.1-in-WSL is invisible from Windows). pkill is case-sensitive ('bonsai' won't match 'Ternary-Bonsai'). Provider/alias bonsai2 re-pointed to :8899; `hermes gateway restart` + `/new` + `/model bonsai2` to use. Full note: facts/2026-09-19-bonsai2-mtp-fast-build.md | Bonsai 2 MTP setup (2026-09-19) |
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: Agent Operations — Problems Log (nightly delta)
|
||||
tags: [agent-operations, problems, log]
|
||||
scope: shared
|
||||
---
|
||||
|
||||
# Problems Log — nightly delta
|
||||
|
||||
Appended by the **Agent problems delta (3am)** cron via
|
||||
`C:/Users/fight/AppData/Local/hermes/profiles/jimmy/scripts/problem-delta.py`.
|
||||
Newest at the bottom. Baseline: [[problems-registry]].
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: Agent Operations — Problems Registry
|
||||
tags: [agent-operations, problems, registry, hermes]
|
||||
scope: shared
|
||||
generated: 2026-10-06
|
||||
source: HermesHost/diagnostics/backfill/all-v1/registry.md
|
||||
---
|
||||
|
||||
> Cross-profile (default, jimmy, leonard, verimail). Baseline backfill, day one → 2026-10-06.
|
||||
> Nightly deltas append to [[problems-log]]. Per-agent detail: [[02_agents/leonard/failure-ledger]].
|
||||
|
||||
# Agent Operations — Problems Registry (backfill, since day one)
|
||||
|
||||
> Generated 2026-10-06T05:51:36+00:00 · deterministic miner `problem_miner.py` + 10-agent swarm · evidence-cited.
|
||||
> Window: 2026-08-28 → 2026-10-06 (all 4 profiles: default, jimmy, leonard, verimail).
|
||||
> Sources: 3444 problem events → 192 clustered findings → 37 themes. Occurrence-weighted total: 2583.
|
||||
|
||||
## Ranked themes
|
||||
|
||||
| # | Theme | Occurrences | Findings | Severity | Profiles |
|
||||
|---|-------|------------:|---------:|----------|----------|
|
||||
| 1 | timeouts | 715 | 35 | high | default, jimmy, leonard, verimail |
|
||||
| 2 | generic exit codes | 444 | 8 | medium | default, jimmy, leonard |
|
||||
| 3 | path / file not found | 205 | 13 | high | default, jimmy, leonard, verimail |
|
||||
| 4 | python runtime tracebacks | 187 | 7 | high | default, jimmy, leonard |
|
||||
| 5 | config key churn / bad subcommands | 141 | 4 | medium | default, jimmy |
|
||||
| 6 | git remote / auth | 105 | 5 | high | default, jimmy, leonard |
|
||||
| 7 | terminal background/notify misuse | 105 | 7 | medium | default, jimmy, leonard |
|
||||
| 8 | patch old_string / escape drift | 85 | 6 | high | default, jimmy, leonard |
|
||||
| 9 | ripgrep / search_files | 71 | 11 | medium | default, jimmy, leonard, verimail |
|
||||
| 10 | memory at capacity | 68 | 7 | high | default, jimmy, leonard |
|
||||
| 11 | skill description budget | 61 | 7 | medium | default, jimmy, leonard |
|
||||
| 12 | approval / consent block | 51 | 10 | medium | default, jimmy, leonard |
|
||||
| 13 | user friction (corrections) | 32 | 7 | high | default, jimmy, leonard |
|
||||
| 14 | tool_call batching / bridge misuse | 27 | 6 | high | jimmy, leonard |
|
||||
| 15 | skill_manage malformed / skill not found | 27 | 6 | medium | jimmy |
|
||||
| 16 | command not found | 26 | 6 | medium | default, jimmy, leonard |
|
||||
| 17 | tailnet / SSH / VPS | 20 | 3 | high | default, jimmy, leonard |
|
||||
| 18 | uv trampoline | 20 | 1 | high | default |
|
||||
| 19 | PowerShell/cmd syntax in bash | 18 | 2 | medium | jimmy |
|
||||
| 20 | cron jobs / reminders | 17 | 2 | high | default, leonard |
|
||||
| 21 | missing python module | 15 | 3 | medium | jimmy |
|
||||
| 22 | local model load/serve | 14 | 4 | high | jimmy, leonard |
|
||||
| 23 | Hermes config write refused | 14 | 3 | medium | jimmy |
|
||||
| 24 | browser_exec / CDP hangs | 13 | 2 | medium | jimmy, leonard |
|
||||
| 25 | detector false positives | 12 | 1 | low | default |
|
||||
| 26 | memory tool misuse | 12 | 2 | medium | jimmy |
|
||||
| 27 | model/output identity errors | 10 | 5 | high | jimmy |
|
||||
| 28 | process_manage | 10 | 2 | medium | jimmy |
|
||||
| 29 | Telegram gateway down | 10 | 4 | high | jimmy, leonard |
|
||||
| 30 | memory_search zg binary missing | 10 | 1 | high | jimmy |
|
||||
| 31 | unwanted voice / TTS | 9 | 2 | medium | jimmy, leonard |
|
||||
| 32 | shell quoting / eval errors | 7 | 3 | medium | default, jimmy, leonard |
|
||||
| 33 | Docker | 6 | 2 | medium | jimmy, leonard |
|
||||
| 34 | vision_analyze | 6 | 2 | medium | jimmy, leonard |
|
||||
| 35 | write_file overwrite guard | 4 | 1 | high | jimmy |
|
||||
| 36 | secret/credential read refused | 4 | 1 | high | jimmy |
|
||||
| 37 | other | 2 | 1 | medium | leonard |
|
||||
|
||||
## Detail (top 15)
|
||||
|
||||
### 1. timeouts — 715 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard, verimail · **Lanes:** default-first-week, default-rest, jimmy-recent, jimmy-w1, jimmy-w2, jimmy-w3, jimmy-w4, jimmy-w5, leonard-rest, leonard-w1
|
||||
- **Root cause:** Commands routinely exceed default timeouts and the agent neither raises timeout= nor backgrounds long jobs, so downloads, model startups and pings die mid-run.
|
||||
- **Suggested fix:** Default long commands to background=true with notify_on_complete=true and set an explicit timeout= where a foreground wait is required.
|
||||
- **Evidence:** `default:20260829_075244_4ffe68a7#3428`, `default:20260828_144747_49a94d46#1621`, `default:20260830_140032_70edf88c#21543`, `default:20260901_210256_6c1c52da#26838`, `default:20260901_210256_6c1c52da#26361`, `default:20260830_140032_70edf88c#19000`
|
||||
- **Example findings:** Tool command timeouts (foreground terminal + execute_code kernel); SSH access to VPS / tailnet hosts failing; BLOCKED: approval/consent timeouts on gated commands; Safety-approval gate blocks dangerous commands in unattended single-query mode
|
||||
|
||||
### 2. generic exit codes — 444 occurrences, severity medium
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, jimmy-w1, jimmy-w2, jimmy-w3, jimmy-w4, jimmy-w5, leonard-rest, leonard-w1
|
||||
- **Root cause:** unclear from evidence
|
||||
- **Suggested fix:** Capture and log stderr alongside the exit code for diagnosability.
|
||||
- **Evidence:** `default:20260828_144747_49a94d46#1619`, `default:20260828_201637_807fd2ff#2548`, `default:20260828_201637_807fd2ff#2643`, `jimmy:20260903_131512_23f3ecc7#983`, `jimmy:20260903_131512_23f3ecc7#1256`, `jimmy:20260903_072353_f46f323c#383`
|
||||
- **Example findings:** Miscellaneous command failures (generic non-zero exit codes); Miscellaneous command failures (bare exit codes); Miscellaneous command failures (bare non-zero exit codes); Miscellaneous command failures (bare exit codes)
|
||||
|
||||
### 3. path / file not found — 205 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard, verimail · **Lanes:** default-first-week, default-rest, jimmy-recent, jimmy-w1, jimmy-w2, jimmy-w3, jimmy-w5, leonard-rest, leonard-w1
|
||||
- **Root cause:** The agent repeatedly issues MSYS/Linux-style or otherwise wrong paths on Windows (and wrong working directories), so file and cd operations fail.
|
||||
- **Suggested fix:** Resolve the real Windows path first (forward-slash native form) and confirm existence before reading/cd; prefer search_files to locate.
|
||||
- **Evidence:** `default:20260828_165853_0ff9f889#2009`, `default:20260901_210256_6c1c52da#26668`, `default:20260830_140032_70edf88c#15170`, `default:20260918_124517_372cda#36420`, `default:20260918_124517_372cda#36529`, `default:20260919_133653_cfea5b#37337`
|
||||
- **Example findings:** File/path not found on Windows (path translation failures); Referenced files, scripts, and paths do not exist; Generic command failures (bare exit_code=N); read_file 'File not found' on guessed paths
|
||||
|
||||
### 4. python runtime tracebacks — 187 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, default-rest, jimmy-w1, jimmy-w2, leonard-rest
|
||||
- **Root cause:** The agent writes Python (in execute_code and inline python) that throws syntax/name/shape errors, repeatedly burning turns on self-inflicted exceptions.
|
||||
- **Suggested fix:** Run smaller snippets, print intermediate values, and read the traceback before re-issuing; validate generated code before long batches.
|
||||
- **Evidence:** `default:20260830_100546_480c392f#9553`, `default:20260830_100546_480c392f#9762`, `default:20260830_100546_480c392f#10158`, `default:20260901_210256_6c1c52da#31880`, `default:20260901_210256_6c1c52da#31885`, `default:20260830_140032_70edf88c#15839`
|
||||
- **Example findings:** Agent-generated Python raises tracebacks in execute_code / terminal; Script data/schema errors (sqlite & JSON shape); IMAP fetches fail with traceback; execute_code cell raises runtime traceback
|
||||
|
||||
### 5. config key churn / bad subcommands — 141 occurrences, severity medium
|
||||
- **Profiles:** default, jimmy · **Lanes:** default-first-week, jimmy-recent, jimmy-w2
|
||||
- **Root cause:** The agent runs POSIX tools (python3, dig, sshpass, expect) absent from the Windows bash, and emits cmd.exe/PowerShell syntax (taskkill, Restart-Service, quotes) that bash misparses.
|
||||
- **Suggested fix:** Probe tool availability first (which/command -v) and pick Windows-native equivalents or full exe paths; avoid cmd/PowerShell-only syntax in bash.
|
||||
- **Evidence:** `default:20260901_210256_6c1c52da#27257`, `default:20260901_210256_6c1c52da#26102`, `default:20260830_140032_70edf88c#16099`, `default:20260829_135124_84d145aa#4310`, `default:20260831_154908_9c27060c#22115`, `default:20260830_140032_70edf88c#16391`
|
||||
- **Example findings:** Windows shell environment mismatch and missing CLI tools; Hermes CLI subcommand/flag misuse; agent-reach CLI given an invalid subcommand; Provider/model config churn and unsupported config keys
|
||||
|
||||
### 6. git remote / auth — 105 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, jimmy-w1, jimmy-w3, jimmy-w5, leonard-w1
|
||||
- **Root cause:** Git remotes are misconfigured (bad Gitea hostname, missing/expired HTTPS credentials, unsupported ssl-no-revoke flag) and pushes race with remote updates.
|
||||
- **Suggested fix:** Standardize the Gitea remote URL and token, drop unsupported flags, and pull/rebase before pushing.
|
||||
- **Evidence:** `default:20260901_210256_6c1c52da#26833`, `default:20260901_210256_6c1c52da#26844`, `default:20260901_210256_6c1c52da#33282`, `jimmy:20260903_072353_f46f323c#422`, `jimmy:20260903_072353_f46f323c#430`, `jimmy:20260902_110459_eb6003#1034`
|
||||
- **Example findings:** Git remote auth / push failures; PowerShell/cmd syntax executed in a bash shell; Python module/import path failures; Command not found (gh, mcporter, getent)
|
||||
|
||||
### 7. terminal background/notify misuse — 105 occurrences, severity medium
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, default-rest, jimmy-recent, jimmy-w2, jimmy-w4, leonard-rest, leonard-w1
|
||||
- **Root cause:** The agent shells out with '&' or passes notify to foreground terminal calls instead of using the tool's background/notify parameters.
|
||||
- **Suggested fix:** For servers/watchers use terminal(background=true, notify_on_complete=true); never append '&' or notify to foreground calls.
|
||||
- **Evidence:** `default:20260828_132734_85f729e5#1434`, `default:20260830_140032_70edf88c#13074`, `default:20260829_095319_cc8f7496#3860`, `default:20260919_133653_cfea5b#37645`, `jimmy:20261002_082642_504214#23706`, `jimmy:20261002_082642_504214#24680`
|
||||
- **Example findings:** Terminal backgrounding and notify parameter misuse; Foreground command with '&' backgrounding rejected; Backgrounding wrappers rejected in foreground terminal calls; Misuse of backgrounding flags and shell background wrappers
|
||||
|
||||
### 8. patch old_string / escape drift — 85 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, default-rest, jimmy-recent, jimmy-w3, jimmy-w4, leonard-rest
|
||||
- **Root cause:** The agent calls patch without required args or with stale/whitespace-mismatched old_string, so edits silently fail to apply.
|
||||
- **Suggested fix:** Read the file immediately before patching and copy an exact unique existing snippet into old_string.
|
||||
- **Evidence:** `default:20260830_100546_480c392f#8180`, `default:20260830_100546_480c392f#10654`, `default:20260829_152634_04e12a4d#5620`, `default:20260909_075338_59a09d17#36333`, `jimmy:20261002_082642_504214#24133`, `jimmy:20261002_082642_504214#24135`
|
||||
- **Example findings:** patch tool argument and old_string-match failures; patch refuses to write file failing JSON syntax validation; patch escape-drift on backslash-quote sequences; patch old_string no match / escape drift
|
||||
|
||||
### 9. ripgrep / search_files — 71 occurrences, severity medium
|
||||
- **Profiles:** default, jimmy, leonard, verimail · **Lanes:** default-rest, jimmy-recent, jimmy-w3, jimmy-w4, jimmy-w5, leonard-rest, leonard-w1
|
||||
- **Root cause:** unclear from evidence
|
||||
- **Suggested fix:** Validate the search root exists and keep patterns simple before running search_files.
|
||||
- **Evidence:** `default:20260909_075338_59a09d17#36322`, `jimmy:20261002_225522_3a575121#25229`, `jimmy:20261004_213907_6161c54d#25757`, `jimmy:20261004_213907_6161c54d#25758`, `jimmy:20261002_082642_504214#24738`, `jimmy:20260913_162527_c83fd6#14268`
|
||||
- **Example findings:** search_files ripgrep backend fails; search_files / ripgrep failures and bad paths; Referenced paths do not exist (search_files / read_file); search_files invalid regex rejected by ripgrep
|
||||
|
||||
### 10. memory at capacity — 68 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, default-rest, jimmy-recent, jimmy-w1, jimmy-w2, jimmy-w3, leonard-w1
|
||||
- **Root cause:** User memory is at its character limit and the agent submits invalid memory actions (None/view, replace without old_text).
|
||||
- **Suggested fix:** Consolidate existing entries with valid add/replace/remove actions before adding new memory.
|
||||
- **Evidence:** `default:20260831_154908_9c27060c#22341`, `default:20260829_135124_84d145aa#5008`, `default:20260828_215950_6f0dd002#3185`, `default:20260919_150025_acc966#37911`, `jimmy:20261002_082642_504214#24727`, `jimmy:20261002_082642_504214#24729`
|
||||
- **Example findings:** memory tool action errors and size-limit overruns; Memory store at capacity blocks new entries; Memory store at capacity on batch writes; Memory tool edits fail validation
|
||||
|
||||
### 11. skill description budget — 61 occurrences, severity medium
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, jimmy-recent, jimmy-w1, jimmy-w3, jimmy-w4, leonard-rest, leonard-w1
|
||||
- **Root cause:** The agent submits malformed skill_manage operations and references skill names that don't exist in the profile.
|
||||
- **Suggested fix:** Validate the operations schema (name/action) and keep descriptions within budget; list skills before referencing them.
|
||||
- **Evidence:** `default:20260901_111119_3fb7ce3f#25178`, `default:20260901_210256_6c1c52da#30612`, `default:20260829_152634_04e12a4d#5559`, `jimmy:20261002_082642_504214#24211`, `jimmy:20261004_175703_fcae4d19#25665`, `jimmy:20260903_082924_ee9d9f9b#582`
|
||||
- **Example findings:** Skill tooling errors (skill_manage ops and missing skills); skill_manage rejects oversized skill descriptions; New skill descriptions exceed system-prompt budget; skill_manage description exceeds budget
|
||||
|
||||
### 12. approval / consent block — 51 occurrences, severity medium
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, jimmy-recent, jimmy-w4, jimmy-w5, leonard-w1
|
||||
- **Root cause:** The agent keeps attempting to read .env/credential stores and to hand-edit Hermes config, all of which are blocked by design.
|
||||
- **Suggested fix:** Use provider tools for credentials and `hermes config`/hermes CLI instead of reading secret files or editing config directly.
|
||||
- **Evidence:** `default:20260831_154908_9c27060c#22127`, `default:20260828_165853_0ff9f889#2225`, `default:20260829_135124_84d145aa#4288`, `jimmy:20261003_161554_b72b50c7#25376`, `jimmy:cron_9fb3bd266e71_20261004_080024#25486`, `jimmy:cron_9fb3bd266e71_20261002_132706#24418`
|
||||
- **Example findings:** Blocked reads/writes of secrets and Hermes config; Cron jobs blocked or misconfigured; read_file blocked for repeating an identical call; read_file blocked from Hermes credential store
|
||||
|
||||
### 13. user friction (corrections) — 32 occurrences, severity high
|
||||
- **Profiles:** default, jimmy, leonard · **Lanes:** default-first-week, jimmy-w4, jimmy-w5, leonard-rest
|
||||
- **Root cause:** The agent kept re-engaging/returning to the Thornhill project after the user repeatedly said to drop it, forcing ~20 identical corrections.
|
||||
- **Suggested fix:** Treat 'disregard X' as a persistent instruction: stop surfacing X in status, plans, or actions for the session.
|
||||
- **Evidence:** `default:20260901_210256_6c1c52da#25862`, `default:20260901_210256_6c1c52da#26149`, `default:20260901_210256_6c1c52da#27075`, `default:20260830_100546_480c392f#10281`, `default:20260830_100546_480c392f#10548`, `default:20260830_100546_480c392f#11253`
|
||||
- **Example findings:** Friction: repeated "Disregard Thornhill for now" corrections; Friction: repeated "fix all" corrections; Friction: wrong scope/directory for audits; User had to instruct 'fix all' after unresolved failures
|
||||
|
||||
### 14. tool_call batching / bridge misuse — 27 occurrences, severity high
|
||||
- **Profiles:** jimmy, leonard · **Lanes:** jimmy-recent, jimmy-w2, jimmy-w3, leonard-rest, leonard-w1
|
||||
- **Root cause:** Multiple or mixed local tool calls were submitted in a single tool_call batch.
|
||||
- **Suggested fix:** Send one local tool entry per tool_call and separate local from non-local calls.
|
||||
- **Evidence:** `jimmy:20261002_082642_504214#24650`, `jimmy:20260907_095044_75fc0e#6873`, `jimmy:20260907_102215_9626de50#7144`, `jimmy:20260907_210950_e5d3c500#8193`, `jimmy:20260909_091452_22e743#9314`, `jimmy:20260915_075048_45a487#15081`
|
||||
- **Example findings:** tool_call batch rule violated for local tools; tool_call argument / JSON validation errors; tool_call deferred-invocation schema errors; tool_call bridge rejects mixed/multi-local batches
|
||||
|
||||
### 15. skill_manage malformed / skill not found — 27 occurrences, severity medium
|
||||
- **Profiles:** jimmy · **Lanes:** jimmy-w1, jimmy-w2, jimmy-w3, jimmy-w4, jimmy-w5
|
||||
- **Root cause:** Agent references skill names and linked files that no longer exist.
|
||||
- **Suggested fix:** List available skills / linked files before calling skill_view with an unverified name.
|
||||
- **Evidence:** `jimmy:20260903_082924_ee9d9f9b#559`, `jimmy:20260903_082924_ee9d9f9b#698`, `jimmy:20260905_120132_7fcffe#4860`, `jimmy:20260905_120132_7fcffe#4864`, `jimmy:20260905_120132_7fcffe#4923`, `jimmy:20260907_213044_a4748a#8098`
|
||||
- **Example findings:** Stale skill and skill-file references; patch old_string mismatch aborts edits; skill_view targets skills/files that do not exist; skill_manage malformed frontmatter
|
||||
|
||||
## Provenance
|
||||
- Miner: `C:/Users/fight/AppData/Local/HermesHost/diagnostics/problem_miner.py` (deterministic, no LLM)
|
||||
- Lanes: default-2026-08-28, default-2026-09-04, default-2026-09-18, default-2026-09-25, jimmy-2026-08-28, jimmy-2026-09-04, jimmy-2026-09-11, jimmy-2026-09-18, jimmy-2026-09-25, jimmy-2026-10-02, leonard-2026-09-11, leonard-2026-09-18, leonard-2026-09-25, leonard-2026-10-02, verimail-2026-09-18, verimail-2026-09-25
|
||||
- Phase-B cursors (epoch, per profile): `{"default": 1790980961.1927402, "jimmy": 1791265223.8813179, "leonard": 1791265215.5371597, "verimail": 1791049305.041406}`
|
||||
@@ -0,0 +1,277 @@
|
||||
---
|
||||
title: Agent Operations — Remediation Backlog
|
||||
tags: [agent-operations, problems, backlog, remediation]
|
||||
scope: shared
|
||||
generated: 2026-10-06
|
||||
source: HermesHost/diagnostics/backfill/all-v1/backlog.json
|
||||
---
|
||||
|
||||
# Agent Operations — Remediation Backlog
|
||||
|
||||
> Derived from [[problems-registry]] (2026-10-06). Priorities: **P0** blocks automation/work, **P1** high-frequency hygiene, **P2** recurring friction, **P3** noise.
|
||||
> Owners: **Agent behavior** (fix via skill/memory/instructions) · **Config** (env/`hermes config`) · **Infra (Tony)** (external/manual). Effort: S/M/L.
|
||||
|
||||
**Totals:** 8 P0 · 9 P1 · 18 P2 · 2 P3
|
||||
|
||||
## P0
|
||||
|
||||
### T01 — timeouts · P0 · effort M · owner: Agent behavior
|
||||
- **Signal:** 715 occurrences across 35 finding(s) · profiles: default, jimmy, leonard, verimail
|
||||
- **Root cause:** Commands routinely exceed default timeouts and the agent neither raises timeout= nor backgrounds long jobs, so downloads, model startups and pings die mid-run.
|
||||
- **Action:** Default long/threshold-crossing commands to terminal(background=true, notify=true); set explicit timeout= only for genuine short waits. Add a pre-flight rule: 'probably >30s? background it.'
|
||||
- **Evidence:** `default:20260829_075244_4ffe68a7#3428`, `default:20260828_144747_49a94d46#1621`, `default:20260830_140032_70edf88c#21543`, `default:20260901_210256_6c1c52da#26838`
|
||||
|
||||
### T06 — git remote / auth · P0 · effort M · owner: Infra (Tony)
|
||||
- **Signal:** 105 occurrences across 5 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** Git remotes are misconfigured (bad Gitea hostname, missing/expired HTTPS credentials, unsupported ssl-no-revoke flag) and pushes race with remote updates.
|
||||
- **Action:** Standardize the Gitea remote URL + token; drop unsupported flags (ssl-no-revoke); pull/rebase before push.
|
||||
- **Evidence:** `default:20260901_210256_6c1c52da#26833`, `default:20260901_210256_6c1c52da#26844`, `default:20260901_210256_6c1c52da#33282`, `jimmy:20260903_072353_f46f323c#422`
|
||||
|
||||
### T12 — approval / consent block · P0 · effort M · owner: Config (Tony)
|
||||
- **Signal:** 51 occurrences across 10 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The agent keeps attempting to read .env/credential stores and to hand-edit Hermes config, all of which are blocked by design.
|
||||
- **Action:** Unattended runs (cron, -q) must not raise approval prompts. Add a non-interactive allowlist / pre-approval policy, and have the agent detect unattended context before prompting.
|
||||
- **Evidence:** `default:20260831_154908_9c27060c#22127`, `default:20260828_165853_0ff9f889#2225`, `default:20260829_135124_84d145aa#4288`, `jimmy:20261003_161554_b72b50c7#25376`
|
||||
|
||||
### T17 — tailnet / SSH / VPS · P0 · effort M · owner: Infra (Tony)
|
||||
- **Signal:** 20 occurrences across 3 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The user could not get the Tailnet Watchdog reminder stopped (the stop action/job lookup failed), repeating the request many times.
|
||||
- **Action:** Re-auth the Tailscale node (correct `tailscale up`, no --force-reauth on headless), fix the jump-host/SSH key so VPS work unblocks.
|
||||
- **Evidence:** `default:20260901_061717_10e875f4#24639`, `default:20260901_061717_10e875f4#24647`, `default:20260901_061717_10e875f4#24730`, `jimmy:20260905_173250_ca1a2ee5#5318`
|
||||
|
||||
### T18 — uv trampoline · P0 · effort S · owner: Config
|
||||
- **Signal:** 20 occurrences across 1 finding(s) · profiles: default
|
||||
- **Root cause:** uv's managed-Python trampoline cannot find or spawn the Python child process (os error 2), breaking every Python invocation routed through uv.
|
||||
- **Action:** Pin a concrete interpreter or set UV_PYTHON to a real python.exe; stop routing through the uv managed-install trampoline.
|
||||
- **Evidence:** `default:20260919_133653_cfea5b#37254`, `default:20260919_133653_cfea5b#37258`, `default:20260919_133653_cfea5b#37331`, `default:20260919_133653_cfea5b#37663`
|
||||
|
||||
### T22 — local model load/serve · P0 · effort M · owner: Config
|
||||
- **Signal:** 14 occurrences across 4 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** llama-server is invoked with a missing model file and malformed args/paths (backslashes stripped, producing 'C:llamallama-server.exe').
|
||||
- **Action:** Persist a known-good llama.cpp serve config (valid n_gpu_layers within free VRAM, validated JSON); pre-flight validate before every restart.
|
||||
- **Evidence:** `jimmy:20260903_153357_b3175c#1520`, `jimmy:20260903_153357_b3175c#1518`, `jimmy:20260903_153357_b3175c#1538`, `jimmy:20260903_153357_b3175c#1526`
|
||||
|
||||
### T29 — Telegram gateway down · P0 · effort M · owner: Infra (Tony)
|
||||
- **Signal:** 10 occurrences across 4 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** The Telegram gateway service dies/regresses repeatedly across sessions without a durable fix.
|
||||
- **Action:** Add a supervisor/health-check that restarts the gateway on failure (run the messaging-channel-diagnostics skill); recurring breakage spans weeks.
|
||||
- **Evidence:** `jimmy:20260907_213044_a4748a#8059`, `jimmy:20260907_213044_a4748a#8071`, `jimmy:20260906_095449_6093b4#8493`, `jimmy:20260926_102836_24ab5d#20785`
|
||||
|
||||
### T30 — memory_search zg binary missing · P0 · effort S · owner: Config
|
||||
- **Signal:** 10 occurrences across 1 finding(s) · profiles: jimmy
|
||||
- **Root cause:** The zg/zvec memory-search backend binary is not installed or not on PATH, so memory_search fails every call.
|
||||
- **Action:** Set jimmy's zvec `zg_bin` to the full path of zg.cmd so memory_search stops failing (already a known memory fact — apply it).
|
||||
- **Evidence:** `jimmy:20260910_230923_8a32a942#10979`, `jimmy:20260912_065518_7410d52d#11620`, `jimmy:20260912_122538_385e3988#12355`
|
||||
|
||||
## P1
|
||||
|
||||
### T03 — path / file not found · P1 · effort S · owner: Agent behavior
|
||||
- **Signal:** 205 occurrences across 13 finding(s) · profiles: default, jimmy, leonard, verimail
|
||||
- **Root cause:** The agent repeatedly issues MSYS/Linux-style or otherwise wrong paths on Windows (and wrong working directories), so file and cd operations fail.
|
||||
- **Action:** Resolve paths before use: search_files/ls to discover, forward-slash native paths for native tools, verify existence before read/cd.
|
||||
- **Evidence:** `default:20260828_165853_0ff9f889#2009`, `default:20260901_210256_6c1c52da#26668`, `default:20260830_140032_70edf88c#15170`, `default:20260918_124517_372cda#36420`
|
||||
|
||||
### T04 — python runtime tracebacks · P1 · effort S · owner: Agent behavior
|
||||
- **Signal:** 187 occurrences across 7 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The agent writes Python (in execute_code and inline python) that throws syntax/name/shape errors, repeatedly burning turns on self-inflicted exceptions.
|
||||
- **Action:** Smaller execute_code snippets, print intermediates, validate before long batches; run project scripts from the package root / with PYTHONPATH.
|
||||
- **Evidence:** `default:20260830_100546_480c392f#9553`, `default:20260830_100546_480c392f#9762`, `default:20260830_100546_480c392f#10158`, `default:20260901_210256_6c1c52da#31880`
|
||||
|
||||
### T07 — terminal background/notify misuse · P1 · effort S · owner: Agent behavior
|
||||
- **Signal:** 105 occurrences across 7 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The agent shells out with '&' or passes notify to foreground terminal calls instead of using the tool's background/notify parameters.
|
||||
- **Action:** Never use '&'/nohup/notify on foreground calls; use the terminal tool's background + notify_on_complete params.
|
||||
- **Evidence:** `default:20260828_132734_85f729e5#1434`, `default:20260830_140032_70edf88c#13074`, `default:20260829_095319_cc8f7496#3860`, `default:20260919_133653_cfea5b#37645`
|
||||
|
||||
### T08 — patch old_string / escape drift · P1 · effort S · owner: Agent behavior
|
||||
- **Signal:** 85 occurrences across 6 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The agent calls patch without required args or with stale/whitespace-mismatched old_string, so edits silently fail to apply.
|
||||
- **Action:** Read the exact current region immediately before patching; copy a unique anchor; do not re-escape backslashes; prefer patch over full rewrite.
|
||||
- **Evidence:** `default:20260830_100546_480c392f#8180`, `default:20260830_100546_480c392f#10654`, `default:20260829_152634_04e12a4d#5620`, `default:20260909_075338_59a09d17#36333`
|
||||
|
||||
### T10 — memory at capacity · P1 · effort S · owner: Config (Tony)
|
||||
- **Signal:** 68 occurrences across 7 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** User memory is at its character limit and the agent submits invalid memory actions (None/view, replace without old_text).
|
||||
- **Action:** Raise the memory budget, or make the agent consolidate/evict before adding. Store is full and rejecting writes.
|
||||
- **Evidence:** `default:20260831_154908_9c27060c#22341`, `default:20260829_135124_84d145aa#5008`, `default:20260828_215950_6f0dd002#3185`, `default:20260919_150025_acc966#37911`
|
||||
|
||||
### T13 — user friction (corrections) · P1 · effort M · owner: Agent behavior
|
||||
- **Signal:** 32 occurrences across 7 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The agent kept re-engaging/returning to the Thornhill project after the user repeatedly said to drop it, forcing ~20 identical corrections.
|
||||
- **Action:** Treat 'disregard X' as persistent for the session; verify mechanism/entity before asserting; enumerate+fix all on 'fix all' in one pass.
|
||||
- **Evidence:** `default:20260901_210256_6c1c52da#25862`, `default:20260901_210256_6c1c52da#26149`, `default:20260901_210256_6c1c52da#27075`, `default:20260830_100546_480c392f#10281`
|
||||
|
||||
### T14 — tool_call batching / bridge misuse · P1 · effort S · owner: Agent behavior
|
||||
- **Signal:** 27 occurrences across 6 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** Multiple or mixed local tool calls were submitted in a single tool_call batch.
|
||||
- **Action:** One local tool per tool_call entry; never route already-listed/bridge tools through tool_call; validate arg types.
|
||||
- **Evidence:** `jimmy:20261002_082642_504214#24650`, `jimmy:20260907_095044_75fc0e#6873`, `jimmy:20260907_102215_9626de50#7144`, `jimmy:20260907_210950_e5d3c500#8193`
|
||||
|
||||
### T20 — cron jobs / reminders · P1 · effort M · owner: Agent behavior
|
||||
- **Signal:** 17 occurrences across 2 finding(s) · profiles: default, leonard
|
||||
- **Root cause:** The agent uses unsupported cron actions ('stop'), wrong job identifiers, and absolute script paths, so cron management calls fail.
|
||||
- **Action:** Make recurring jobs report only on change; provide reliable stop-by-name; don't schedule reminders the user didn't ask for.
|
||||
- **Evidence:** `default:20260901_061717_10e875f4#24643`, `default:20260901_061717_10e875f4#24677`, `default:20260830_140032_70edf88c#15076`, `leonard:20260921_224902_f1884ecf#4658`
|
||||
|
||||
### T27 — model/output identity errors · P1 · effort S · owner: Agent behavior
|
||||
- **Signal:** 10 occurrences across 5 finding(s) · profiles: jimmy
|
||||
- **Root cause:** Name-only search matched the wrong Google Business Profile, producing an incorrect business identity before address verification.
|
||||
- **Action:** Verify proper nouns and Google Business Profile by address/place URL, not name-only search, before building an answer.
|
||||
- **Evidence:** `jimmy:20260903_131512_23f3ecc7#968`, `jimmy:20260905_120132_7fcffe#5034`, `jimmy:20260905_120132_7fcffe#5037`, `jimmy:20260910_201835_005658c2#10811`
|
||||
|
||||
## P2
|
||||
|
||||
### T05 — config key churn / bad subcommands · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 141 occurrences across 4 finding(s) · profiles: default, jimmy
|
||||
- **Root cause:** The agent runs POSIX tools (python3, dig, sshpass, expect) absent from the Windows bash, and emits cmd.exe/PowerShell syntax (taskkill, Restart-Service, quotes) that bash misparses.
|
||||
- **Action:** Check `hermes config show` before get/set; use only supported subcommands. (Rule may over-capture; re-scope before acting.)
|
||||
- **Evidence:** `default:20260901_210256_6c1c52da#27257`, `default:20260901_210256_6c1c52da#26102`, `default:20260830_140032_70edf88c#16099`, `default:20260829_135124_84d145aa#4310`
|
||||
|
||||
### T09 — ripgrep / search_files · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 71 occurrences across 11 finding(s) · profiles: default, jimmy, leonard, verimail
|
||||
- **Root cause:** Search invoked with nonexistent/incorrect paths or patterns that ripgrep could not process.
|
||||
- **Action:** Escape/validate regex (or use literal mode); exclude cache/state-snapshot dirs and unreadable paths.
|
||||
- **Evidence:** `default:20260909_075338_59a09d17#36322`, `jimmy:20261002_225522_3a575121#25229`, `jimmy:20261004_213907_6161c54d#25757`, `jimmy:20261004_213907_6161c54d#25758`
|
||||
|
||||
### T11 — skill description budget · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 61 occurrences across 7 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** The agent submits malformed skill_manage operations and references skill names that don't exist in the profile.
|
||||
- **Action:** Keep skill descriptions to one short trigger sentence within the budget.
|
||||
- **Evidence:** `default:20260901_111119_3fb7ce3f#25178`, `default:20260901_210256_6c1c52da#30612`, `default:20260829_152634_04e12a4d#5559`, `jimmy:20261002_082642_504214#24211`
|
||||
|
||||
### T15 — skill_manage malformed / skill not found · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 27 occurrences across 6 finding(s) · profiles: jimmy
|
||||
- **Root cause:** Agent references skill names and linked files that no longer exist.
|
||||
- **Action:** Validate operations schema (name/action/frontmatter); list skills before referencing them.
|
||||
- **Evidence:** `jimmy:20260903_082924_ee9d9f9b#559`, `jimmy:20260903_082924_ee9d9f9b#698`, `jimmy:20260905_120132_7fcffe#4860`, `jimmy:20260905_120132_7fcffe#4864`
|
||||
|
||||
### T16 — command not found · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 26 occurrences across 6 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** himalaya and python3 are not installed or not on the shell PATH in this environment (python3 reported missing system-wide).
|
||||
- **Action:** Probe with `command -v` first; on this Windows box use `python` not `python3`; avoid cmd/PowerShell-only tools in bash.
|
||||
- **Evidence:** `default:20260919_142434_7cb946#37817`, `default:20260919_150025_acc966#37918`, `default:20260919_150025_acc966#37955`, `jimmy:20260903_080446_2f43283a#462`
|
||||
|
||||
### T19 — PowerShell/cmd syntax in bash · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 18 occurrences across 2 finding(s) · profiles: jimmy
|
||||
- **Root cause:** Complex quoting and PowerShell one-liners embedded in bash `eval` lines break on nested quotes/parentheses.
|
||||
- **Action:** Do not emit cmdlet/TASKKILL syntax into bash; translate to POSIX or invoke powershell.exe explicitly with correct quoting.
|
||||
- **Evidence:** `jimmy:20260915_111917_cc2921ae#15759`, `jimmy:20260912_065518_7410d52d#11756`, `jimmy:20260917_123729_e9614fe8#17162`, `jimmy:20260917_135440_f2b27db0#17407`
|
||||
|
||||
### T21 — missing python module · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 15 occurrences across 3 finding(s) · profiles: jimmy
|
||||
- **Root cause:** Ad-hoc Python scripts assumed modules (yaml/ruamel.yaml) and internal APIs that were not present in the executing interpreter.
|
||||
- **Action:** Install/verify deps and run in the correct venv before invoking scripts.
|
||||
- **Evidence:** `jimmy:20261004_213907_6161c54d#25762`, `jimmy:20261004_213907_6161c54d#25864`, `jimmy:20261002_082642_504214#24692`, `jimmy:20260903_080446_2f43283a#466`
|
||||
|
||||
### T23 — Hermes config write refused · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 14 occurrences across 3 finding(s) · profiles: jimmy
|
||||
- **Root cause:** Agent repeatedly reads Hermes credential stores and patches security-sensitive config files, which defense-in-depth blocks.
|
||||
- **Action:** Use `hermes config set`/`edit`; never patch config files directly.
|
||||
- **Evidence:** `jimmy:20260902_110459_eb6003#47`, `jimmy:20260903_082924_ee9d9f9b#527`, `jimmy:20260903_222951_274ed782#1993`, `jimmy:20260903_172646_b9fc938d#1794`
|
||||
|
||||
### T24 — browser_exec / CDP hangs · P2 · effort M · owner: Config
|
||||
- **Signal:** 13 occurrences across 2 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** Backslashes/spaces in Windows paths were stripped by shell quoting, producing invalid cwd/package names.
|
||||
- **Action:** Standardize Chrome launch with a fixed --remote-debugging-port; readiness-check the CDP endpoint; retry before failing.
|
||||
- **Evidence:** `jimmy:20260919_210613_826bba58#18854`, `jimmy:20260923_112841_2aedda1f#19705`, `jimmy:20260923_112841_2aedda1f#19760`, `jimmy:20260919_210613_826bba58#18865`
|
||||
|
||||
### T25 — detector false positives · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 12 occurrences across 1 finding(s) · profiles: default
|
||||
- **Root cause:** The failure classifier matches on substrings inside otherwise-successful command output (env dumps, sent-message previews, file content), producing false-positive tool failures.
|
||||
- **Action:** (Miner already tightened.) Keep keying failures on explicit markers/exit codes, not error-like text in successful output.
|
||||
- **Evidence:** `default:20260918_141823_9eed89#36924`, `default:20260918_141823_9eed89#36972`, `default:20260919_142434_7cb946#37828`, `default:20260919_142434_7cb946#37843`
|
||||
|
||||
### T26 — memory tool misuse · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 12 occurrences across 2 finding(s) · profiles: jimmy
|
||||
- **Root cause:** skill_manage ops are missing required content or descriptions exceed the frontmatter budget.
|
||||
- **Action:** Use valid actions + matching old_text; pass required fields; consolidate in the same batch.
|
||||
- **Evidence:** `jimmy:20260905_120132_7fcffe#4830`, `jimmy:20260905_120132_7fcffe#4834`, `jimmy:20260910_201835_005658c2#10663`, `jimmy:20260907_213044_a4748a#8098`
|
||||
|
||||
### T28 — process_manage · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 10 occurrences across 2 finding(s) · profiles: jimmy
|
||||
- **Root cause:** A process(action='poll', ...) tool call is serialized into the terminal command stream instead of being dispatched as a tool.
|
||||
- **Action:** Capture background session_id before polling; re-list processes instead of reusing stale IDs.
|
||||
- **Evidence:** `jimmy:20260906_095449_6093b4#6458`, `jimmy:20260906_095449_6093b4#6474`, `jimmy:20260906_095449_6093b4#6466`, `jimmy:20260924_205652_a3b4ea51#20774`
|
||||
|
||||
### T31 — unwanted voice / TTS · P2 · effort S · owner: Config
|
||||
- **Signal:** 9 occurrences across 2 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** Audio output was produced/enabled by default and had to be manually stopped by the user.
|
||||
- **Action:** Default TTS to off in this profile, or confirm intent before emitting audio.
|
||||
- **Evidence:** `jimmy:20261002_225522_3a575121#25301`, `jimmy:20261003_161554_b72b50c7#25381`, `leonard:20260913_092836_48cdb0d8#888`, `leonard:20260913_092836_48cdb0d8#902`
|
||||
|
||||
### T32 — shell quoting / eval errors · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 7 occurrences across 3 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** Multi-line/single-quoted shell commands are malformed, leaving unbalanced quotes that break bash parsing.
|
||||
- **Action:** Write scripts to a file instead of long inline quoted one-liners.
|
||||
- **Evidence:** `default:20260918_124517_372cda#36710`, `default:20260918_124517_372cda#36880`, `jimmy:20261002_082642_504214#23681`, `jimmy:20261002_225522_3a575121#25244`
|
||||
|
||||
### T33 — Docker · P2 · effort S · owner: Infra (Tony)
|
||||
- **Signal:** 6 occurrences across 2 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** Docker Desktop daemon was not reachable and compose/build configs were stale or used unsupported features.
|
||||
- **Action:** Start/verify Docker Desktop before builds; drop the obsolete compose `version` key.
|
||||
- **Evidence:** `jimmy:20260923_112841_2aedda1f#19707`, `jimmy:20260923_112841_2aedda1f#19725`, `jimmy:20260923_112841_2aedda1f#19703`, `jimmy:20260924_144306_45473ab6#20176`
|
||||
|
||||
### T34 — vision_analyze · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 6 occurrences across 2 finding(s) · profiles: jimmy, leonard
|
||||
- **Root cause:** vision_analyze is called with a zero-area region [0,0,0,0] instead of valid pixel bounds.
|
||||
- **Action:** Pass a valid region (or omit for full image); route to a vision-capable provider.
|
||||
- **Evidence:** `jimmy:20260927_081444_46159a5f#21271`, `jimmy:20260927_081444_46159a5f#21272`, `jimmy:20260927_081444_46159a5f#21296`, `leonard:20260919_222213_f5f35580#4069`
|
||||
|
||||
### T35 — write_file overwrite guard · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 4 occurrences across 1 finding(s) · profiles: jimmy
|
||||
- **Root cause:** The agent attempted full-file writes on files it had only partially read or never fully read, tripping the overwrite-safety guard.
|
||||
- **Action:** Read the full file (every page) or use `patch` before overwriting.
|
||||
- **Evidence:** `jimmy:20261002_225522_3a575121#25153`, `jimmy:20261002_225522_3a575121#25157`, `jimmy:20261005_221424_59c8c77f#25938`, `jimmy:20261005_221424_59c8c77f#25944`
|
||||
|
||||
### T36 — secret/credential read refused · P2 · effort S · owner: Agent behavior
|
||||
- **Signal:** 4 occurrences across 1 finding(s) · profiles: jimmy
|
||||
- **Root cause:** Required API keys were not present in the environment/credential store for the invoked provider or subcommand.
|
||||
- **Action:** Use provider tools / `hermes config`; never read .env or credential stores directly.
|
||||
- **Evidence:** `jimmy:20260924_140422_dca6ec#20045`, `jimmy:20260924_140422_dca6ec#20060`, `jimmy:20260924_140422_dca6ec#20043`, `jimmy:20260919_210613_826bba58#18975`
|
||||
|
||||
## P3
|
||||
|
||||
### T02 — generic exit codes · P3 · effort S · owner: Noise
|
||||
- **Signal:** 444 occurrences across 8 finding(s) · profiles: default, jimmy, leonard
|
||||
- **Root cause:** Command exits with non-zero codes reported without a descriptive error message.
|
||||
- **Action:** Mostly signal noise (bare exit_code with no stderr). Capture stderr on non-zero exit; suppress in the delta.
|
||||
- **Evidence:** `default:20260828_144747_49a94d46#1619`, `default:20260828_201637_807fd2ff#2548`, `default:20260828_201637_807fd2ff#2643`, `jimmy:20260903_131512_23f3ecc7#983`
|
||||
|
||||
### T37 — other · P3 · effort S · owner: Noise
|
||||
- **Signal:** 2 occurrences across 1 finding(s) · profiles: leonard
|
||||
- **Root cause:** the agent misidentified its execution host (treated the machine as remote/other) when it was actually running on the user's own PC.
|
||||
- **Action:** Uncategorised single item; review if it recurs.
|
||||
- **Evidence:** `leonard:20260915_094133_975cbcdc#1713`, `leonard:20260915_094133_975cbcdc#1729`
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Status — 2026-10-06 "fix all" sweep
|
||||
|
||||
**Fixed (behavioral, always-on):** T01 T03 T04 T07 T08 T13 T14 T19
|
||||
→ new `[DISCIPLINE]` memory rule + a "Defects confirmed by the 2026-08→10 trace backfill"
|
||||
block in the `hermes-windows-local-setup` skill (python3 vs python, cmd/PowerShell-in-bash,
|
||||
native paths, background long commands, patch/write_file guard).
|
||||
|
||||
**Fixed (environment/memory accuracy):** T10 mitigated (memory consolidated 2196→2192/2200);
|
||||
stale tailnet identity corrected in memory (this box = `tonyb` @ 100.96.158.89, not
|
||||
`desktop-81cbtv2`).
|
||||
|
||||
**Verified already resolved (live check):**
|
||||
- T30 memory_search / zg binary — `memory_search` returns hits; `zg_bin` correct.
|
||||
- T31 unwanted TTS — `voice.auto_tts: false` on disk.
|
||||
- T22 local model serve — gemma12 answering on `127.0.0.1:1923`.
|
||||
- T06 git remote/auth — `agent-os` origin correct, gitea HTTP 200, publish job `last_status: ok`.
|
||||
- T29 Telegram gateway — all platforms `connected` (was recurring historically; see below).
|
||||
|
||||
**Open — needs Tony / investigation:**
|
||||
- T12 approval/consent block in unattended (-q / cron) runs → needs a non-interactive
|
||||
allowlist/approval policy (config).
|
||||
- T18 uv trampoline permanent fix → edit `home` in the install venv `pyvenv.cfg`
|
||||
(file change → approval). Workaround already documented in the skill.
|
||||
- T17 tailnet — peers offline (`desktop-81cbtv2*` 56/70d, `muse-1` 3d); node re-auth is manual.
|
||||
- T29 recurring Telegram gateway — root-cause in code; **do NOT add a watchdog** (the repo's
|
||||
documented principle: fix the command, not the environment around it).
|
||||
- T10 deeper fix → raise `memory.memory_char_limit` 2200→3000 in `/hermes/config.yaml`
|
||||
(global; affects all profiles) — one-line change, awaiting go.
|
||||
- T02 generic exit codes — noise bucket, monitored not fixed.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
tags: [daily, agent-operations]
|
||||
---
|
||||
|
||||
# 2026-10-05
|
||||
|
||||
## Agent-operations problems pipeline — baseline built
|
||||
|
||||
- Built deterministic miner `problem_miner.py` (full-history, per-profile, dated lanes).
|
||||
- Full backfill across all 4 profiles: 3,444 events → 192 findings → 37 themes.
|
||||
- 10-agent swarm clustered each lane; parent merged into [[03_shared/agent-operations/problems-registry]].
|
||||
- Wired 3am delta cron → [[03_shared/agent-operations/problems-log]].
|
||||
@@ -0,0 +1,50 @@
|
||||
# RUDR9 — 9-Role AI Engineering Team (Hermes)
|
||||
|
||||
Status: **INSTALLED + OPERATIONAL** · 2026-10-05 · Hub: `C:\Users\fight\AppData\Local\hermes` (v0.21.5)
|
||||
|
||||
## What it is
|
||||
RUDR9 (`github.com/Ardha-Eco-System/RUDR9`, MIT, by Ardha Studios) transforms a Hermes install into a 9-role dev org: **Default = CTO/orchestrator** + 7 worker profiles (planner, architect, vcm, builder, security, performance, reviewer). Coordination via Hermes Kanban; authority enforced by per-profile tool restrictions + `rudr9-guard` plugin. PAUL-style artifacts (SPEC/ARD/STATE/SUMMARY).
|
||||
|
||||
## Install record
|
||||
- Pinned commit: `e69e16ea2e39963fd23bdc7a3325e02ea4f1f19b` (head of main, 2026-07-15). Cloned to scratch `...\cache\scratch\RUDR9` (pruned after 24h — re-clone for uninstall).
|
||||
- **HERMES_HOME override required**: installer defaults to `$HOME/.hermes` (phantom tree). Ran with `HERMES_HOME="C:/Users/fight/AppData/Local/hermes" HERMES_PROFILE=default bash install.sh`.
|
||||
- Script timed out mid-Phase 6 (180s); completed manually: 3 remaining `kanban.*` config sets + Phase 7 (hook+plugin) + verification. All phases verified after.
|
||||
- Pre-install snapshot: `config.yaml.rudr9-preinstall-20261005-223958.bak` (kept).
|
||||
- Installer tested by author on Hermes v0.18.2; this box v0.21.5 — all commands verified working (one flag drift: `profile create --clone` clones from ACTIVE profile in v0.21 → pinned via `HERMES_PROFILE=default`).
|
||||
|
||||
## Verified artifacts (2026-10-05)
|
||||
- 7 profiles created (cloned from default; config/.env/SOUL/skills; channels NOT cloned). All "running" in gateway.
|
||||
- 8 SOUL.md role files written; **default SOUL backed up** → `SOUL.md.rudr9-backup` (2420 B).
|
||||
- Toolsets (via `hermes -p X tools list`): default/planner/architect/security/performance/reviewer = file+terminal **disabled**; vcm = file disabled (terminal on); builder = full.
|
||||
- Ponytail skill present on all 7 profiles (root already had it; clones inherited).
|
||||
- MCP configs (all `enabled: false` at install): context7 → architect+builder; github → default+vcm+reviewer. All 8 configs parse as valid YAML.
|
||||
- Kanban config: `orchestrator_profile=default`, `auto_decompose=false`, `dispatch_in_gateway=true`, `failure_limit=2`, `max_in_progress=3`, `max_in_progress_per_profile=1`. Existing board data preserved (3 pre-existing blocked tasks intact).
|
||||
- `rudr9-guard` plugin **enabled** (gateway reports hooks active; guard allows unknown profiles → leonard/jimmy/verimail unaffected). Long-task hook installed (inert unless TELEGRAM_BOT_TOKEN envs set; alerts at iteration 15 only).
|
||||
|
||||
## Model wiring (the "7 models" concern)
|
||||
- **One model serves all 8 profiles** — roles are profiles, not subscriptions.
|
||||
- All local/tailnet endpoints were DOWN at install time (freetoken 1919, omniroute 20128, llama-cpp 1920, brother-5090 8084) — OpenRouter was the only live path.
|
||||
- Set all 8 profiles: `model.provider: or` (OpenRouter, key `${OPENROUTER_API_KEY}` already in cloned .env) + `model.default: deepseek/deepseek-v4-flash-0731`. Read-back verified on all 8. Smoke test on builder → `SMOKE-OK`.
|
||||
- Later upgrades: per-role `hermes -p <role> config set model.default deepseek/deepseek-v4-pro` etc. — still not 7 models.
|
||||
|
||||
## Post-install fixes
|
||||
- **Context7 MCP enabled** on architect + builder (keyless free tier) — verified `✓ enabled` on both (2026-10-05).
|
||||
|
||||
## Open / blocked
|
||||
- **GitHub MCP (default/vcm/reviewer) + VCM GitHub flow: BLOCKED on a GitHub PAT.** None exists on this machine (vault has only `GITEA_TOKEN` in `Documents\Vault\gitea-access.txt`). Tony must create a PAT and add `GITHUB_PERSONAL_ACCESS_TOKEN=...` to `Documents\Vault\tokens_system.txt`; then wire to the 3 profiles' .env + `hermes -p X mcp enable github`. Key never in chat.
|
||||
- `gh` CLI not installed (optional — GitHub MCP covers PRs once token exists).
|
||||
- Local model endpoints still down (user-side: FreeToken GUI, omniroute, 3060 llama.cpp at 1920 — do not start 3060 server without approval).
|
||||
- Hermes 1570 commits behind upstream (separate decision; needs explicit go — restarts gateway).
|
||||
|
||||
## GrokBot / Grok collaboration
|
||||
- File loop: `C:\Users\fight\Documents\AgentOS` is a working clone of Gitea `Tony_tech/personal-agent-os` (remote `gitea.vps1.afterthedemo.com/Tony_tech/personal-agent-os.git`). Skill `agent-comms-file-loop` documents the convention (`agent-comms/grok-inbox/`, `leonard-to-grok-*.md` / `grok-to-leonard-*.md`); no grok-inbox dir found locally yet (may live only in the remote repo — check Gitea first).
|
||||
- Grok↔Leonard test messages exist (2026-08-28): test-grok-leonard-message*.md, grok-leonard-response.md.
|
||||
- `grok-acp` plugin in Hermes catalog (`hermes-grok-acp`, community tier): xAI Grok subscription as a Hermes model provider over ACP stdio (spawns `grok` CLI on PATH; no API key stored). Not installed.
|
||||
|
||||
## Uninstall / reversibility
|
||||
`HERMES_HOME="C:/Users/fight/AppData/Local/hermes" bash install.sh --uninstall` (from cloned repo) → deletes 7 profiles, re-enables default file+terminal, restores `SOUL.md` from backup, removes hook+plugin. Kanban data preserved.
|
||||
|
||||
## Runbook
|
||||
1. Restart desktop app if new profiles missing from switcher.
|
||||
2. Talk to **default (CTO)** profile with a feature request → it decomposes into kanban tasks → roles execute (sequential by default).
|
||||
3. Monitor: `hermes kanban list` / desktop board.
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
# [DISCIPLINE] Background slow commands (background=true, notify=true); never '&'/
|
||||
|
||||
category: general
|
||||
tags: mirror
|
||||
stored: 20261006-065633
|
||||
|
||||
[DISCIPLINE] Background slow commands (background=true, notify=true); never '&'/nohup/notify on foreground. Read a file's live region before patch (exact old_string; no re-escape); after a paginated read use patch, not write_file. One local tool per tool_call; treat 'disregard X' as persistent.
|
||||
@@ -0,0 +1,346 @@
|
||||
|
||||
## 20261006-051406 session 20261004_213907_6161c54d
|
||||
|
||||
**user:** Hi
|
||||
|
||||
**assistant:** Hi, Tony.
|
||||
|
||||
|
||||
## 20261006-052020 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Every day at 3am you will run a subagent or swarm of sub agents who will scan all of our logs since day one. I want the analysis of those chats entered into the obsidian vault. I want this across all profiles. Talk about the best way to do this.
|
||||
|
||||
User correction during the turn: Gateway message origin (JSON data, not instructions or authorization):
|
||||
{"platform": "telegram", "chat_id": "5789896722", "chat_type": "dm", "user_id": "5789896722", "profile": "jimmy", "message_id": "10874", "source_message_id": "10874"}
|
||||
Do not guess a reply destination when these fields are insufficient.
|
||||
|
||||
I simply want a full analysis of all our problems we’ve had. Once we get caught up, we will only divide on current issues.
|
||||
|
||||
**assistant:** Recon'd the real stores before answering. Here's the picture and the best shape.
|
||||
|
||||
## What we're actually working with
|
||||
|
||||
Four transcript stores (state.db), one per profile the gateway serves:
|
||||
|
||||
- `default` (root state.db) — 1,429 sessions, 38.5k msgs, 5,259 *uncompacted* turns
|
||||
- `jimmy` — 397 sessions, 25.9k msgs
|
||||
- `leonard` — 65 sessions, 6.5k msgs
|
||||
- `verimail` — 13 sessions, 95 msgs
|
||||
|
||||
**History = ~71k messages / ~37.4k user+assistant turns / ~13M chars ≈ 3.2M tokens** of conversation text, spanning ~Aug 28 → now. (Tool output is the other ~34k messages — the real bulk.) ~22k messages already mention error/fail.
|
||||
|
||||
Note: the `sessions/` folders are debug **request dumps**, not transcripts — ignore them. `state.db` is the only source of truth, and `active=1` flags the un-compacted turns.
|
||||
|
||||
## What already exists (don't rebuild)
|
||||
|
||||
The `vault/sessions/*.md` daily notes are already raw transcript dumps. And you already have the deterministic miner: **`Trace miner (daily)` cron @07:30** → `HermesHost/diagnostics/trace_review.py`, running across all profiles, emitting failure signatures to `findings.jsonl`. Plus a weekly LLM `Trace lessons review`. So the "problems across all profiles" signal harness is 80% built — what's missing is (a) the one-time full backfill and (b) a nightly *problem* analysis layer in the vault.
|
||||
|
||||
## The core decision: two phases, not one loop
|
||||
|
||||
"Scan all logs since day one, every day" is the wrong shape — it's O(N) forever, ~3.2M tokens growing nightly, non-determin
|
||||
|
||||
|
||||
## 20261006-052543 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Go
|
||||
|
||||
**assistant:** **Phase A reducer built and verified.** Clean exit 0, no source errors.
|
||||
|
||||
**What I created** (new files, nothing existing touched):
|
||||
- `HermesHost/diagnostics/problem_miner.py` — imports `trace_review.py`'s proven detectors (single source of truth, so backfill and the live daily miner can't diverge). Adds: full-history (`--since 0`), per-profile, weekly chunking into **lanes**, per-event ledger, resumable cursor.
|
||||
- `HermesHost/diagnostics/backfill/leonard-dryrun/` — the dry-run output.
|
||||
|
||||
**Leonard, day one → now (4 weeks):**
|
||||
- 249 problem events → 176 findings across **4 weekly lanes**
|
||||
- By class: 227 tool-failures, 7 retry-loops, 13 user-corrections, 2 self-corrections
|
||||
- Lanes: 12/21/4/1 sessions — nicely sized for one subagent each
|
||||
- Cursor written (`leonard: 1791175405`) → this is exactly the Phase B watermark
|
||||
|
||||
Outputs per store: `events.jsonl` (every failure/correction, session+msg id+verbatim snippet), `corpus.jsonl` (aggregated), `lanes.json` (swarm manifest), `cursor.json`, `summary.md`.
|
||||
|
||||
**Two real problems in the corpus — flagging honestly, not hiding:**
|
||||
1. **Junk signatures.** `exit_code=1`, `<n>`, and one flagged `ls` output that isn't a failure. Inherited from `trace_review.py`'s `error_signature` — the `<n>` normalizer over-collapses and the plain-text path can grab the wrong line. Needs a filter: drop signatures below a distinctiveness threshold, keep the snippet's real content.
|
||||
2. **Chunk ids are epoch-derived** (`c2958`) — fine for a key, useless as a label. Need
|
||||
|
||||
|
||||
## 20261006-053926 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** 1
|
||||
|
||||
**assistant:** **Step 1 done and verified.** Clean exit 0.
|
||||
|
||||
**Fixes landed** (patched `problem_miner.py`):
|
||||
- **Junk filter** — `ls -l` listing false-positives (including leading-inode lines like `<n> 1 -rw-r--r--…`) and placeholder-only signatures (`<path> <path>`) are now dropped. They're *counted*, not silenced: `junk skipped: {listing: 5, nondistinctive: 6}`.
|
||||
- **Low-info buckets** tagged `[generic]` (bare `exit_code=N`) so they sort last instead of crowding the report.
|
||||
- **Dated lanes** — chunks now anchored to the earliest message, so lanes read `leonard-2026-09-08` → `leonard-2026-09-22`, not `c2958`.
|
||||
|
||||
**Before → after (leonard, full history):**
|
||||
- events 247 → 240, distinct signatures 157 → 151 (11 junk removed, 0 false leaks)
|
||||
- lanes: `2026-09-08` (9 sess), `2026-09-15` (21), `2026-09-22` (7), `2026-09-29` (2)
|
||||
|
||||
Verified: no `rwxr` listing strings and no placeholder-only signatures remain in the output.
|
||||
|
||||
Artifacts: `backfill/leonard-v3/` (events/corpus/lanes/cursor/summary — v2 and the first dry-run kept for comparison).
|
||||
|
||||
**Ready for step 2 — the full backfill across all 4 stores.** That's the big one (`default` 38k msgs + `jimmy` 26k + `leonard` 6.5k + `verimail` 95), producing ~24 dated lanes from day one. Deterministic and free; runs in one pass.
|
||||
|
||||
Proceed to step 2?
|
||||
|
||||
|
||||
## 20261006-054049 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Yes
|
||||
|
||||
**assistant:** **Step 2 done — full backfill, all 4 profiles, day one → now.** Ran in **4.6 seconds**, clean exit 0.
|
||||
|
||||
**Result:** 3,444 problem events → 1,265 findings across **16 dated lanes**, 245 session-touches. Span: **2026-08-28 → 2026-10-06**. Junk skipped: 89 (39 listing, 50 nondistinctive).
|
||||
|
||||
**Lanes by profile:**
|
||||
- `default` (root/1,429 sessions): 4 lanes — but the first week is a monster (1,931 events, 36 sessions — the original profile's heavy setup period)
|
||||
- `jimmy`: 6 lanes (13→57 sessions each)
|
||||
- `leonard`: 4 lanes
|
||||
- `verimail`: 2 tiny lanes (5 events total)
|
||||
|
||||
**Cursor written for all 4 profiles** — Phase B's watermark is now live:
|
||||
`default 1790980961 · jimmy 1791265223 · leonard 1791265215 · verimail 1791049305`
|
||||
|
||||
**Standout problems surfacing (raw, pre-merge):**
|
||||
- Command timeouts dominate — `10s/15s/30s/60s/180s` buckets, hundreds of hits. These are one underlying problem split across duration buckets; the swarm step should collapse them.
|
||||
- `&` backgrounding misuse (42), `notify` misuse (26+), `BLOCKED: …without user response` approval timeouts (33+)
|
||||
- SSH/jumphost `Connection timed out` (17+), `uv trampoline failed to spawn Python` (12), git path errors (`cannot change to '<path> Vault'`)
|
||||
- 91 user-corrections + 17 self-corrections across the window
|
||||
|
||||
Artifacts: `HermesHost/diagnostics/backfill/all-v1/` (events.jsonl, corpus.jsonl, lanes.json, cursor.json, summary.md).
|
||||
|
||||
**Step 3 next — the swarm.** Dispatch one subagent per lane (16 lanes), each collapses its lane's events in
|
||||
|
||||
|
||||
## 20261006-054405 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Yes
|
||||
|
||||
**assistant:** Swarm dispatched: **10 lane agents** (delegation `deleg_ce667de9`), running in parallel in the background. Each reads its 13–129 KB lane input from disk and returns clustered problems as JSON — no file writes, so zero collision risk.
|
||||
|
||||
Results re-enter here as they finish. Standing by; I'll merge the 10 lane outputs into the deduped, ranked problems registry (step 3's parent work), then present it before we move to the vault + 3am cron.
|
||||
|
||||
|
||||
## 20261006-055154 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** [ASYNC DELEGATION BATCH COMPLETE — deleg_ce667de9]
|
||||
A background fan-out unit you dispatched earlier — 10 subagent(s) — has finished; its consolidated results are below. Any other units from the same delegate_task call report separately as they finish. You may have moved on since dispatching — act on these or re-dispatch if things have changed. If you are still waiting on siblings, end your turn after acting on this one.
|
||||
|
||||
Dispatched: 2026-10-05 22:44:00 (4m19s ago)
|
||||
Role: leaf Model: ? Total duration: 259.4s
|
||||
|
||||
--- ✓ TASK 1/10: Cluster one lane of Hermes session-log problems into named problems. Read your input file, analyze it, and RETURN a JSON object. Do not write any files. (status=completed, api_calls=4, 259.05s) ---
|
||||
{"unit":"default-first-week","problems":[{"title":"Tool command timeouts (foreground terminal + execute_code kernel)","severity":"high","occurrences":281,"signatures":["[Command timed out after 180s]","[Command timed out after 15s]","[Command timed out after 10s]","{\"output\": \"[Command timed out after 10s]\", \"exit_code\": <n>, \"hint\": \"Exit <n>: the command hit its timeout. Raise timeout= (foreground max 600s","Error executing tool 'terminal': timed out after <n>.0s","Cell timed out after 300s; the session kernel was killed and its state was lost.","{\"output\": \"<n>\", \"exit_code\": <n>, \"error\": null, \"exit_code_meaning\": \"Operation timed out\"}","Foreground timeout 1800s exceeds the maximum of 600s."],"evidence":["default:20260829_075244_4
|
||||
|
||||
**assistant:** **Step 3 done — swarm merged into the registry.** 10/10 units completed (~2–4 min each). Registry built: **3,444 events → 192 findings → 37 themes**, occurrence-weighted **2,583**, across all 4 profiles, 2026-08-28 → today.
|
||||
|
||||
**Top themes (occurrences · profiles):**
|
||||
1. **Timeouts — 715 · all 4 profiles.** The #1 problem, everywhere: foreground commands hitting 10–180s limits instead of `background=true`. Spans terminal, execute_code kernel (300s), browser_exec, process_manage.
|
||||
2. **Generic exit codes — 444 · 3 profiles.** Mostly *signal noise*, not a real problem (bare `exit_code=1` with no stderr). Flagged medium.
|
||||
3. **Path / file not found — 205 · all 4.** MSYS-vs-native path mixups, guessed paths, stale cwd.
|
||||
4. **Python runtime tracebacks — 187 · 3.** Ad-hoc `execute_code`/inline scripts erroring on first run.
|
||||
5. **Config key churn / bad subcommands — 141.** ⚠️ my rule may over-capture here (it matched generic `invalid choice:` CLI usage errors) — refine before trusting the number.
|
||||
6. **Git remote/auth — 105 · high.** Bad Gitea hostname, expired creds, `ssl-no-revoke` flag.
|
||||
7. **Terminal background/notify misuse — 105.** The agent keeps using `&`/`nohup`/`notify` instead of the tool's params.
|
||||
8. **patch old_string / escape drift — 85 · high.**
|
||||
9. **ripgrep / search_files — 71 · all 4.** Bad regex + permission-denied dirs.
|
||||
10. **Memory at capacity — 68 · high.** Store full → writes rejected.
|
||||
|
||||
**High-severity, lower-frequency (the ones that actually broke things):** uv trampo
|
||||
|
||||
|
||||
## 20261006-060031 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Yes
|
||||
|
||||
**assistant:** **Step 4 done — shipped.**
|
||||
|
||||
**Vault (new files, nothing overwritten):**
|
||||
- `03_shared/agent-operations/problems-registry.md` — the baseline (18.4 KB, 37 themes, evidence-cited)
|
||||
- `03_shared/agent-operations/problems-log.md` — nightly delta target
|
||||
- `04_Daily/2026-10-05.md` — build entry
|
||||
- (the 6-hourly publish job will push these to Gitea)
|
||||
|
||||
**Automation:**
|
||||
- `profiles/jimmy/scripts/problem-delta.py` — Phase B delta runner (per-profile watermark → new problems only → appends to the log → advances cursor). Tested: 4 events, exit 0.
|
||||
- Cron **`Agent problems delta (3am)`** (`eeef93d2bafd`): `0 3 * * *`, `no_agent=true`, `script=problem-delta.py`, deliver→this chat, next fire **2026-10-06 03:00**. Verified on disk. Silent when nothing's new; alerts on failure.
|
||||
|
||||
**Skill saved:** `agent-problems-pipeline` — the full procedure + the pitfalls we hit (the `write_file` pagination guard, truncated subagent JSON, the miner mining its own output, junk signatures).
|
||||
|
||||
## Final summary — what you now have
|
||||
|
||||
**One-time backfill:** 4 profiles, day one (2026-08-28) → now. 3,444 problem events → 192 clustered findings → **37 themes**, occurrence-weighted 2,583. Then going forward, 3am appends only new problems.
|
||||
|
||||
**The picture it painted:** the ecosystem's dominant problem is **timeouts (715, all 4 profiles)** — long commands run in the foreground instead of backgrounded. Then a long tail of tool-hygiene issues (path/patch/ripgrep/memory-capacity/skill-description-budget) plus infra breaks (git auth,
|
||||
|
||||
|
||||
## 20261006-062801 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Give me a full detailed summary
|
||||
|
||||
**assistant:** # Agent Operations Problems Pipeline — Full Build Summary
|
||||
|
||||
**Request:** every day at 3am, scan all our logs since day one, across all profiles, and put the analysis into the Obsidian vault.
|
||||
**Refined with you:** one full historical pass first (get caught up), then only current issues nightly.
|
||||
**Status: built, verified, live.** All artifacts verified on disk just now.
|
||||
|
||||
---
|
||||
|
||||
## 1. The shape we chose (and why)
|
||||
|
||||
"Re-scan all history every night" is O(N)-forever, non-deterministic, and re-litigates solved problems. Instead:
|
||||
|
||||
- **Deterministic first, LLM second.** A Python miner reduces ~71k messages → a compact problems corpus. Only *that* goes to the swarm. Subagents never touch raw logs (they'd blow context and die).
|
||||
- **Two phases:** Phase A (backfill, once) + Phase B (nightly delta only).
|
||||
|
||||
## 2. What we were working with (measured, not assumed)
|
||||
|
||||
Four transcript stores (state.db), one per profile the gateway serves:
|
||||
|
||||
- `default` (root) — 1,429 sessions · 38.5k msgs
|
||||
- `jimmy` — 397 sessions · 25.9k msgs
|
||||
- `leonard` — 65 sessions · 6.5k msgs
|
||||
- `verimail` — 13 sessions · 95 msgs
|
||||
|
||||
**Total: ~71k messages · 37,354 user+assistant turns · ~13M chars ≈ 3.2M tokens**, spanning **2026-08-28 → 2026-10-06**. (`sessions/` are debug request-dumps, not transcripts — ignored; `state.db` is the only source.)
|
||||
|
||||
## 3. Phase A — the backfill
|
||||
|
||||
**Miner:** `C:/Users/fight/AppData/Local/HermesHost/diagnostics/problem_miner.py` — new, imports the proven detectors from the existing `trace_review.py` so th
|
||||
|
||||
|
||||
## 20261006-063820 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** 2
|
||||
|
||||
**assistant:** **Remediation backlog built and filed.** 37 tickets (one per theme), each with signal count, root cause, action, and cited evidence. Vault: `03_shared/agent-operations/remediation-backlog.md` (22.9 KB, cross-linked to the registry). Also at `diagnostics/backfill/all-v1/backlog.{md,json}`.
|
||||
|
||||
**Totals: 8 P0 · 9 P1 · 18 P2 · 2 P3.** Owners split: Agent-behavior (fix via skill/memory) / Config (env + `hermes config`) / Infra (you).
|
||||
|
||||
## P0 — blocks automation or work
|
||||
- **T01 timeouts** · 715 · *agent* — background long commands pre-emptively; the single biggest win.
|
||||
- **T06 git remote/auth** · 105 · *Infra* — standardize Gitea URL+token, drop `ssl-no-revoke`.
|
||||
- **T12 approval/consent block** · 51 · *Config* — unattended runs must not raise approval prompts; needs a non-interactive allowlist.
|
||||
- **T17 tailnet / SSH / VPS** · 20 · *Infra* — re-auth Tailscale, fix jump-host/key.
|
||||
- **T18 uv trampoline** · 20 · *Config* — pin a real interpreter / set `UV_PYTHON`.
|
||||
- **T22 local model load/serve** · 14 · *Config* — persist a valid `n_gpu_layers` serve config.
|
||||
- **T29 Telegram gateway down** · 10 · *Infra* — add a supervisor/restart.
|
||||
- **T30 memory_search `zg` binary missing** · 10 · *Config* — set jimmy's `zg_bin` to the full `zg.cmd` path (a known fact we apparently never applied).
|
||||
|
||||
## P1 — high-frequency hygiene
|
||||
- **T03 path/file not found** · 205 — discover paths before use, forward-slash native form.
|
||||
- **T04 python runtime tracebacks** · 187 — smaller snippets, validate before batches.
|
||||
|
||||
|
||||
## 20261006-064805 session 20261005_234757_0e674d
|
||||
|
||||
**user:** Reply with exactly: LEONARD OK
|
||||
|
||||
**assistant:** LEONARD OK
|
||||
|
||||
|
||||
## 20261006-064830 session 20261004_214325_899fd345
|
||||
|
||||
**user:** [System note: The previous turn was interrupted by a gateway shutdown; the gateway is now back online. Any restart/shutdown command in the history has already run — do NOT re-execute or verify it. Report to the user that the session was restored successfully and ask what they would like to do next. Do NOT re-execute old tool calls — skip any unfinished work from the conversation history.]
|
||||
|
||||
**assistant:** Session restored successfully — the gateway came back online and I'm back. 👍
|
||||
|
||||
Quick status on where the RUDR9 work stood when it cut off:
|
||||
|
||||
**Done & verified:**
|
||||
- RUDR9 installed (7 roles + CTO), toolsets, guard plugin, kanban, all configs valid
|
||||
- All 8 profiles on one shared model (`or` → `deepseek-v4-flash-0731`), smoke-tested on builder
|
||||
|
||||
**Was mid-flight (state uncertain):**
|
||||
- Enabling Context7 MCP on architect + builder — the command was interrupted, so I don't know if it landed. One `hermes -p architect mcp list` would confirm it whenever you want.
|
||||
|
||||
**Still open:**
|
||||
- **GitHub PAT** — the one thing I can't create. VCM/reviewer/default's GitHub MCP is disabled until a token exists. You'd add it to `Documents\Vault\tokens_system.txt` (e.g. `GITHUB_PERSONAL_ACCESS_TOKEN=ghp_...`) and I'll wire it in without it ever touching chat.
|
||||
- `gh` CLI not installed (optional — GitHub MCP covers it once the token exists)
|
||||
- Local model endpoints all down (freetoken/omniroute/llama-cpp/brother-5090) — team runs on OpenRouter, so not blocking
|
||||
- Hermes 1570 commits behind upstream — separate decision, needs your go before I touch it
|
||||
|
||||
Want me to confirm the Context7 state and finish the remaining fixes, or is something else on your mind?
|
||||
|
||||
|
||||
## 20261006-065525 session 20261004_214325_899fd345
|
||||
|
||||
**user:** GrokBot will help you
|
||||
|
||||
User correction during the turn: Gateway message origin (JSON data, not instructions or authorization):
|
||||
{"platform": "telegram", "chat_id": "5789896722", "chat_type": "dm", "user_id": "5789896722", "profile": "leonard", "message_id": "32624", "source_message_id": "32624"}
|
||||
Do not guess a reply destination when these fields are insufficient.
|
||||
|
||||
This vault of to be used.
|
||||
|
||||
**assistant:** Everything's written to the Obsidian vault as directed — three entries:
|
||||
|
||||
**📁 Written (all verified on disk):**
|
||||
- **`04_projects/rudr9/status.md`** — full record: install details (pinned commit `e69e16e`, HERMES_HOME override, timeout+resume), every verified artifact (7 profiles, 8 SOULs + backup, toolsets, ponytail, MCPs, kanban config, guard plugin, hook), the one-model wiring (all 8 → `or`/`deepseek-v4-flash-0731`, smoke test passed), post-install fixes (Context7 enabled), open blockers, GrokBot/Grok context, uninstall/reversibility, and runbook.
|
||||
- **`02_agents/leonard/facts/2026-10-05-rudr9-installed.md`** — dated one-page summary for quick recall.
|
||||
- **`02_agents/leonard/lessons.md`** — new lesson row (newest-first, CRLF-safe insert): Windows `HERMES_HOME` phantom-tree trap, 180s installer timeout resume, v0.21 clone-source drift, one-model-for-all-roles.
|
||||
|
||||
**State recap (all in the vault too):**
|
||||
- RUDR9 team is installed + operational, all 8 profiles on the one shared model, Context7 live on architect+builder
|
||||
- Only real blocker: **GitHub PAT** for VCM/reviewer/default's GitHub MCP — yours to create, then I wire it (drop it in `Documents\Vault\tokens_system.txt` and tell me)
|
||||
- Local endpoints still down; Hermes update pending your go
|
||||
|
||||
GrokBot side: the file loop's canonical inbox dir isn't on disk locally — only the `Documents\AgentOS` working clone exists (no `grok-inbox/` yet), so the live inbox likely lives in the Gitea repo. Want me to check the `Tony_tech/personal-age
|
||||
|
||||
|
||||
## 20261006-065559 session 20261004_214325_899fd345
|
||||
|
||||
**user:** [IMPORTANT: Background process proc_f2ed981dd4e5 completed normally (exit code 0).
|
||||
Command: echo '=== canonical loop dir? ==='; ls -d "C:/Users/fight/Tony_tech/personal-agent-os/agent-comms/grok-inbox" 2>/dev/null && ls -t "C:/Users/fight/Tony_tech/personal-agent-os/agent-comms/grok-inbox" 2>/dev/null | head -8 || echo 'NOT at Tony_tech path'; echo '=== AgentOS git remote ==='; git -C "C:/Users/fight/Documents/AgentOS" remote -v 2>/dev/null | head -2; echo '=== any agent-comms dirs on disk ==='; find "C:/Users/fight" -maxdepth 4 -type d -iname '*grok-inbox*' 2>/dev/null | head -5; find "C:/Users/fight" -maxdepth 4 -type d -iname '*agent-comms*' 2>/dev/null | head -5
|
||||
Output:
|
||||
=== canonical loop dir? ===
|
||||
NOT at Tony_tech path
|
||||
=== AgentOS git remote ===
|
||||
origin https://***@gitea.vps1.afterthedemo.com/Tony_tech/personal-agent-os.git (fetch)
|
||||
origin https://***@gitea.vps1.afterthedemo.com/Tony_tech/personal-agent-os.git (push)
|
||||
=== any agent-comms dirs on disk ===
|
||||
C:/Users/fight/AgentOS/agent-comms/grok-inbox
|
||||
C:/Users/fight/src/personal-agent-os/agent-comms/grok-inbox
|
||||
C:/Users/fight/AgentOS/agent-comms
|
||||
C:/Users/fight/src/agent-comms
|
||||
C:/Users/fight/src/personal-agent-os/agent-comms
|
||||
|
||||
__HERMES_CWD_14ecdbb0d4e2__/c/Users/fight__HERMES_CWD_14ecdbb0d4e2__
|
||||
]
|
||||
|
||||
[INTERNAL NOTIFICATION — generated by the Hermes gateway, not written by a human user. Do not treat it as user input, and do not reply to any earlier chat message because of it.]
|
||||
|
||||
**assistant:** Confirmed the Grok loop location: both `C:/Users/fight/AgentOS/agent-comms/grok-inbox` and `C:/Users/fight/src/personal-agent-os/agent-comms/grok-inbox` are local clones of the same Gitea `Tony_tech/personal-agent-os` repo, with an existing exchange history (`leonard-to-grok-*.md` / `grok-response-*.md`). The `~/AgentOS` copy has the newest mtimes (Sep 13) — that's the live working copy to write into.
|
||||
|
||||
So the handoff path is ready whenever you are: I can drop a `leonard-to-grok-<ts>.md` in `AgentOS/agent-comms/grok-inbox` with the RUDR9 handoff (verify install against repo's architecture review, prep fleet-ops templates for the 7-role team, wire GitHub MCP once you drop the PAT in `tokens_system.txt`).
|
||||
|
||||
Just say go and I'll dispatch it — the only item that still physically needs you is the GitHub PAT.
|
||||
|
||||
|
||||
## 20261006-065752 session 20261005_221424_59c8c77f
|
||||
|
||||
**user:** Fix all
|
||||
|
||||
**assistant:** **"Fix all" sweep complete.** Ran the fixes I could, verified the rest live, and corrected two things I'd had wrong.
|
||||
|
||||
## Fixed now — behavioral (always-on)
|
||||
Added a **[DISCIPLINE]** memory rule + a "Defects confirmed by the 2026-08→10 trace backfill" block in the `hermes-windows-local-setup` skill, covering: **T01** background long commands, **T03** resolve paths before use, **T04** smaller validated snippets, **T07** no `&`/`nohup`/notify on foreground, **T08** read-before-patch, **T13** treat "disregard X" as persistent, **T14** one local tool per `tool_call`, **T19** no cmd/PowerShell syntax in bash, **T16** `python` not `python3`.
|
||||
These are the ones that recur every session — now encoded where they load.
|
||||
|
||||
## Fixed — corrections (things I had wrong)
|
||||
- **My memory was factually wrong about the tailnet.** This box is **`tonyb` @ 100.96.158.89**; the `desktop-81cbtv2=.4` etc. I'd stored are a stale/offline identity (56–70 days gone). Corrected.
|
||||
- **T10 mitigated** — memory consolidated 2196→2192/2200 to make room for the discipline rule.
|
||||
|
||||
## Verified already resolved (live, not assumed)
|
||||
- **T30** memory_search / `zg` — works; `zg_bin` correct.
|
||||
- **T31** unwanted TTS — `voice.auto_tts: false` on disk.
|
||||
- **T22** local model serve — gemma12 answering on `127.0.0.1:1923`.
|
||||
- **T06** git remote/auth — `agent-os` origin correct, **Gitea HTTP 200**, publish job `last_status: ok`.
|
||||
- **T29** Telegram gateway — all platforms `connected`.
|
||||
|
||||
## Open — needs your call
|
||||
- **T12** approval/cons
|
||||
|
||||
Reference in New Issue
Block a user