# Leonard — Lessons Learned

## Reusable lessons for the Leonard agent. One entry per lesson.
> Newest first.

| Date | Lesson | Context |
|------|--------|---------|
| 2026-09-25 | OMH (oh-my-hermes) install on Windows: the PowerShell remote-installer pattern (fetch install.ps1, run iex) is flagged by the command-safety gate and just times out on Telegram (no approval UI). Use the documented agent path: pin the SHA with `git ls-remote https://github.com/rlaope/oh-my-hermes.git refs/heads/main` (never `main`), then `uv tool install "git+https://github.com/rlaope/oh-my-hermes@$SHA"` (local build, no remote-exec, does not trip the gate), then `omh --hermes-home "C:\Users\fight\AppData\Local\hermes\profiles\leonard" setup --no-interactive --yes --no-omh-tui`, then `omh doctor` (target 54/54 passing, 0 blocking). `omh` is a native binary: pass Windows `C:\...` paths, never MSYS `/c/...`. Non-interactive setup does not touch model aliases. | OMH install (2026-09-25) |
| 2026-09-25 | Native-tool path discipline + pre-delete verification (near-miss during OMH setup): I passed an MSYS path `/c/Users/...` to the **native** `omh.exe`, which read it literally as a phantom `C:\c\Users\...`; that phantom root pre-existed and also held a `models/` subtree (git research repo, caches, GGUF weights) that I did not recognize when I `rm -rf`'d the phantom. Real data survived only because the authoritative `C:\models` (git clean at remote HEAD; all weights present; public remote sudoingX/bonsai2-small-gpu) was a separate tree. Rules: (a) pass `C:\Users\...` to native tools, never MSYS `/c/...`; (b) before deleting a dir, list its contents and confirm no pre-existing real files; (c) after any write, verify the intended target actually gained the expected files; (d) if a phantom root (`C:\c`) appears, check whether it shadows a real `models/` clone first. | OMH install side-effect (2026-09-25) |
| 2026-09-19 | Bonsai 2 27B fast build (sudoingX fork: kernel PR #218 + Hadamard #217, MTP head graft) runs ~2x stock: fresh tg128 39.3 tok/s (vs 26.3), streaming ~49 tok/s (code 53.4), greedy output BYTE-IDENTICAL to spec-off (lossless). Windows-native CUDA build BLOCKED on this PC: MSVC 14.44/14.51 crashes CUDA 12.4 cudafe++, no admin for CUDA 13/old MSVC, pip nvcc wheels ship no nvcc.exe on Windows. Build+serve in WSL2 (gcc-13 + CUDA 12.4 apt; GPU passthrough native) at 127.0.0.1:8899; model in WSL ext4 /opt/models, bind --host 0.0.0.0 (127.0.0.1-in-WSL is invisible from Windows). pkill is case-sensitive ('bonsai' won't match 'Ternary-Bonsai'). Provider/alias bonsai2 re-pointed to :8899; `hermes gateway restart` + `/new` + `/model bonsai2` to use. Full note: facts/2026-09-19-bonsai2-mtp-fast-build.md | Bonsai 2 MTP setup (2026-09-19) |
| 2026-09-17 | MiniCPM5-2B (FreeToken 127.0.0.1:1919) returns EMPTY/thinking-only completions under Hermes agent turns (44× `Empty response (no content or reasoning)` in errors.log, Sep 12–17), yet answers cleanly to direct curl (small/20K/22K prompts, tools, SSE, 4-way concurrency, multi-turn). Failure is model-vs-agent-load, not server/KV. Same box also logged `prompt is too long: 17105 > 8585 max (prompt + generation)` (Sep 15, Gemma) proving FreeToken real KV budget << advertised 131K. Fix: route agent turns to routeapi/deepseek-v4-flash-0731; keep 2B models for light direct use only. | Profile audit + MiniCPM5 diagnosis (2026-09-17) |
| 2026-09-17 | `providers.freetoken.max_tokens: 2000` is silently IGNORED (`unknown config keys ignored: max_tokens`) — key placement invalid for this schema, so the generation cap never applies. Also `model_aliases.minicpm5` had a duplicate `model:` key (Gemma dict + MiniCPM scalar) so the alias didn't point where labeled. | Profile audit (2026-09-17) |
| 2026-09-17 | `hermes cron create` grammar: the prompt positional MUST come immediately after the schedule and BEFORE any `--flags` (`hermes cron create <schedule> "<prompt>" --name ...`). Prompt after flags → `unrecognized arguments`. `--model`/`--provider` pin per-job to a working provider — restored jobs referencing `openrouter:deepseek` (dead provider) would silently fail. | Cron rebuild (2026-09-17) |
| 2026-09-17 | ROUTE_API_KEY is injected into the gateway env, NOT in `~/.hermes/profiles/leonard/.env` (grep `.env` comes up empty). The live session resolves routeapi fine. When auditing providers, check auth.json + gateway env, not just .env. | Profile audit (2026-09-17) |
| 2026-09-17 | git-bash grep does NOT support `\d` in ERE (`grep -E "^\d+"` silently matches nothing) — use `[0-9]`. Tailscale status on Windows prints peer rows as `IP name tag state`; capture `$1 $2 $3 $5` so the cron watch script sees online/offline transitions. | tailnet-watch.sh debug (2026-09-17) |
| 2026-09-17 | Tailscale health: this box (100.64.0.1) itself reports `offline` and health check warns it can't reach the coordination server — HTTPS front door `headscale.vps1.afterthedemo.com` is intercepted by a NETGEAR router login cert (CN=www.routerlogin.net). Matches known rule: control plane is HTTP-only at http://74.208.111.99:8443, never HTTPS. Watchdog will flag when state flips. | Profile audit, tailnet check (2026-09-17) |
| 2026-09-17 | Profile hygiene issue: leonard `cron/jobs.json` had been wiped to `{"jobs":[]}` (Sep 3) while `skills/.curator_backups/2026-08-27T18-02-27Z/cron-jobs.json` still held the 5 real jobs. Restore pattern: backup → parse JSON → re-create via `hermes cron create` with corrected provider pins + profile-scripts (curator-run.sh, havoc-nightly.sh, tailnet-watch.sh). Always verify pins in jobs.json after create. | Cron rebuild (2026-09-17) |
| 2026-09-01 | VPS SSH (74.208.111.99:22) is rate-limited/firewalled: >3 failed connect attempts within a period blocks the IP for a while. Don't loop SSH connect attempts against the VPS — ask Anthony to add a key/open the port. Gitea (:80/443) and headscale (:8443) still reachable, so HTTPS/Gitea endpoints stay up. | Athena Oracle deploy blocked |
| 2026-08-31 | Vault RAG chunk/file counts recorded independently in STATUS.md, HOME.md, and 01_user/current-context.md drift after every reindex (160/162/167/219 all appeared for the same index). Update all three together (or point at one canonical count) when reindexing to avoid stale numbers. | AgentOS merge reindex |
| 2026-08-30 | LFM2.5-2.6B is both agentic AND a pure reasoning model. It uses `think` tags in the chat template, putting reasoning in `reasoning_content` and visible output in `content`. Requires sufficient `max_tokens` (≥256) to complete reasoning before emitting visible answer. | LFM2.5 setup and testing |
| 2026-08-30 | `notify` parameter on `terminal` tool requires `background=true` — foreground commands always return exit code, and `notify` is rejected. Use background mode for daemons/services. | LFM2.5 launch debugging |
| 2026-08-30 | Llama.cpp `--n-gpu-layers 999` is required for full CUDA offload on Windows. Without it, the model runs on CPU only. | LFM2.5 setup |
| 2026-08-30 | LFM2.5-2.6B (Q4_K, ~2.1GB) + FreeToken gpt-oss-20b (~12.8GB) cannot coexist on RTX 3060 12GB. Only one can run at a time. | GPU VRAM budget |
| 2026-08-30 | Hermes foreground timeout (120s) kills long-running processes. Use `background=true` for daemons, then verify with separate curl. | Process management |
| 2026-08-29 | Put auth/authorization gates at the single central callback-dispatch point, not in each per-handler function. For callback-heavy integrations (Telegram inline buttons), a shared gate at dispatch covers every existing + future callback prefix at once and is regression-proof; per-handler gates silently miss new handlers (the model picker was the ungated one). | Telegram picker auth audit |
|------|--------|---------|
| 2026-08-29 | In a cron job config, provider prefix must NOT go in the `model` field — use bare `deepseek/deepseek-v4-flash-0731` + separate `provider: openrouter`. Sending `openrouter/deepseek/...` as the model ID → "not a valid model ID" → falls to a rate-limited free model then freetoken's tiny 8,448-token budget → job dies. | Agent OS cron consolidation setup |
| 2026-08-29 | Cron jobs inherit the full default-toolset system prompt (~16,931 tokens context). Huge context overflows tiny fallback models with small KV budgets. Give cron jobs a minimal toolset (e.g. `read, files`) like `check-agent-discussion` (~2,648 tokens). | Agent OS cron consolidation setup |
| 2026-08-29 | Files in a session that came back as base64 text on disk (memory_policy.md, lessons.md) were real corruption — the decoder (`read_file`) hid it. Always verify a file's on-disk content is plain markdown, not base64. | Vault Agent OS setup — base64 corruption bug |
| 2026-08-29 | `/model <name>` in Hermes is session-scoped; use `--global` or `hermes config set model.default` to persist. Config/status showing a different model than the live session = split-brain from session-level switches. | Model switching permanence |
| 2026-08-29 | `hermes fallback add|remove` are interactive pickers that reject bare model strings; set the whole `fallback_providers` list via `hermes config set` instead. | Model switching permanence |
| 2026-09-24 | Bonsai 2 27B + MTP verified end-to-end on the 3060: decode 48.6-50.8 tok/s, cold prefill 241 tok/s, VRAM 11,166 MiB used / 950 free at -c 131072 - matches the advertised ~50 tok/s. Fixed an intermittent EMPTY-content failure by adding `--reasoning-effort medium` to /opt/serve-bonsai-mtp.sh (template default xhigh let reasoning eat the whole max_tokens budget: 2,524ch reasoning, finish_reason=length, empty content). Do NOT misdiagnose that as the /new context trap or the post-abort scheduler wedge - /health is 200 and decode speed is normal; compare reasoning_content length vs usage.completion_tokens. Also: the 'official PrismML installer' (Bonsai-demo/PQ2_0/:8080) advice is SUPERSEDED on this box by the sudoingX PTQ1_0+MTP build at :8899, and a Windows-native CUDA build of the fork remains impossible (MSVC 14.44 crashes CUDA 12.4 cudafe++). Full note: facts/2026-09-24-bonsai2-mtp-verified.md | Bonsai 2 verification + reasoning-effort fix |