jimmy
9.9 KiB
9.9 KiB
Leonard — Lessons Learned
Reusable lessons for the Leonard agent. One entry per lesson.
Newest first.
| Date | Lesson | Context |
|---|---|---|
| 2026-09-25 | OMH (oh-my-hermes) install on Windows: the PowerShell remote-installer pattern (fetch install.ps1, run iex) is flagged by the command-safety gate and just times out on Telegram (no approval UI). Use the documented agent path: pin the SHA with git ls-remote https://github.com/rlaope/oh-my-hermes.git refs/heads/main (never main), then uv tool install "git+https://github.com/rlaope/oh-my-hermes@$SHA" (local build, no remote-exec, does not trip the gate), then omh --hermes-home "C:\Users\fight\AppData\Local\hermes\profiles\leonard" setup --no-interactive --yes --no-omh-tui, then omh doctor (target 54/54 passing, 0 blocking). omh is a native binary: pass Windows C:\... paths, never MSYS /c/.... Non-interactive setup does not touch model aliases. |
OMH install (2026-09-25) |
| 2026-09-25 | Native-tool path discipline + pre-delete verification (near-miss during OMH setup): I passed an MSYS path /c/Users/... to the native omh.exe, which read it literally as a phantom C:\c\Users\...; that phantom root pre-existed and also held a models/ subtree (git research repo, caches, GGUF weights) that I did not recognize when I rm -rf'd the phantom. Real data survived only because the authoritative C:\models (git clean at remote HEAD; all weights present; public remote sudoingX/bonsai2-small-gpu) was a separate tree. Rules: (a) pass C:\Users\... to native tools, never MSYS /c/...; (b) before deleting a dir, list its contents and confirm no pre-existing real files; (c) after any write, verify the intended target actually gained the expected files; (d) if a phantom root (C:\c) appears, check whether it shadows a real models/ clone first. |
OMH install side-effect (2026-09-25) |
| 2026-09-19 | Bonsai 2 27B fast build (sudoingX fork: kernel PR #218 + Hadamard #217, MTP head graft) runs ~2x stock: fresh tg128 39.3 tok/s (vs 26.3), streaming ~49 tok/s (code 53.4), greedy output BYTE-IDENTICAL to spec-off (lossless). Windows-native CUDA build BLOCKED on this PC: MSVC 14.44/14.51 crashes CUDA 12.4 cudafe++, no admin for CUDA 13/old MSVC, pip nvcc wheels ship no nvcc.exe on Windows. Build+serve in WSL2 (gcc-13 + CUDA 12.4 apt; GPU passthrough native) at 127.0.0.1:8899; model in WSL ext4 /opt/models, bind --host 0.0.0.0 (127.0.0.1-in-WSL is invisible from Windows). pkill is case-sensitive ('bonsai' won't match 'Ternary-Bonsai'). Provider/alias bonsai2 re-pointed to :8899; hermes gateway restart + /new + /model bonsai2 to use. Full note: facts/2026-09-19-bonsai2-mtp-fast-build.md |
Bonsai 2 MTP setup (2026-09-19) |
| 2026-09-17 | MiniCPM5-2B (FreeToken 127.0.0.1:1919) returns EMPTY/thinking-only completions under Hermes agent turns (44× Empty response (no content or reasoning) in errors.log, Sep 12–17), yet answers cleanly to direct curl (small/20K/22K prompts, tools, SSE, 4-way concurrency, multi-turn). Failure is model-vs-agent-load, not server/KV. Same box also logged prompt is too long: 17105 > 8585 max (prompt + generation) (Sep 15, Gemma) proving FreeToken real KV budget << advertised 131K. Fix: route agent turns to routeapi/deepseek-v4-flash-0731; keep 2B models for light direct use only. |
Profile audit + MiniCPM5 diagnosis (2026-09-17) |
| 2026-09-17 | providers.freetoken.max_tokens: 2000 is silently IGNORED (unknown config keys ignored: max_tokens) — key placement invalid for this schema, so the generation cap never applies. Also model_aliases.minicpm5 had a duplicate model: key (Gemma dict + MiniCPM scalar) so the alias didn't point where labeled. |
Profile audit (2026-09-17) |
| 2026-09-17 | hermes cron create grammar: the prompt positional MUST come immediately after the schedule and BEFORE any --flags (hermes cron create <schedule> "<prompt>" --name ...). Prompt after flags → unrecognized arguments. --model/--provider pin per-job to a working provider — restored jobs referencing openrouter:deepseek (dead provider) would silently fail. |
Cron rebuild (2026-09-17) |
| 2026-09-17 | ROUTE_API_KEY is injected into the gateway env, NOT in ~/.hermes/profiles/leonard/.env (grep .env comes up empty). The live session resolves routeapi fine. When auditing providers, check auth.json + gateway env, not just .env. |
Profile audit (2026-09-17) |
| 2026-09-17 | git-bash grep does NOT support \d in ERE (grep -E "^\d+" silently matches nothing) — use [0-9]. Tailscale status on Windows prints peer rows as IP name tag state; capture $1 $2 $3 $5 so the cron watch script sees online/offline transitions. |
tailnet-watch.sh debug (2026-09-17) |
| 2026-09-17 | Tailscale health: this box (100.64.0.1) itself reports offline and health check warns it can't reach the coordination server — HTTPS front door headscale.vps1.afterthedemo.com is intercepted by a NETGEAR router login cert (CN=www.routerlogin.net). Matches known rule: control plane is HTTP-only at http://74.208.111.99:8443, never HTTPS. Watchdog will flag when state flips. |
Profile audit, tailnet check (2026-09-17) |
| 2026-09-17 | Profile hygiene issue: leonard cron/jobs.json had been wiped to {"jobs":[]} (Sep 3) while skills/.curator_backups/2026-08-27T18-02-27Z/cron-jobs.json still held the 5 real jobs. Restore pattern: backup → parse JSON → re-create via hermes cron create with corrected provider pins + profile-scripts (curator-run.sh, havoc-nightly.sh, tailnet-watch.sh). Always verify pins in jobs.json after create. |
Cron rebuild (2026-09-17) |
| 2026-09-01 | VPS SSH (74.208.111.99:22) is rate-limited/firewalled: >3 failed connect attempts within a period blocks the IP for a while. Don't loop SSH connect attempts against the VPS — ask Anthony to add a key/open the port. Gitea (:80/443) and headscale (:8443) still reachable, so HTTPS/Gitea endpoints stay up. | Athena Oracle deploy blocked |
| 2026-08-31 | Vault RAG chunk/file counts recorded independently in STATUS.md, HOME.md, and 01_user/current-context.md drift after every reindex (160/162/167/219 all appeared for the same index). Update all three together (or point at one canonical count) when reindexing to avoid stale numbers. | AgentOS merge reindex |
| 2026-08-30 | LFM2.5-2.6B is both agentic AND a pure reasoning model. It uses think tags in the chat template, putting reasoning in reasoning_content and visible output in content. Requires sufficient max_tokens (≥256) to complete reasoning before emitting visible answer. |
LFM2.5 setup and testing |
| 2026-08-30 | notify parameter on terminal tool requires background=true — foreground commands always return exit code, and notify is rejected. Use background mode for daemons/services. |
LFM2.5 launch debugging |
| 2026-08-30 | Llama.cpp --n-gpu-layers 999 is required for full CUDA offload on Windows. Without it, the model runs on CPU only. |
LFM2.5 setup |
| 2026-08-30 | LFM2.5-2.6B (Q4_K, ~2.1GB) + FreeToken gpt-oss-20b (~12.8GB) cannot coexist on RTX 3060 12GB. Only one can run at a time. | GPU VRAM budget |
| 2026-08-30 | Hermes foreground timeout (120s) kills long-running processes. Use background=true for daemons, then verify with separate curl. |
Process management |
| 2026-08-29 | Put auth/authorization gates at the single central callback-dispatch point, not in each per-handler function. For callback-heavy integrations (Telegram inline buttons), a shared gate at dispatch covers every existing + future callback prefix at once and is regression-proof; per-handler gates silently miss new handlers (the model picker was the ungated one). | Telegram picker auth audit |
| ------ | -------- | --------- |
| 2026-08-29 | In a cron job config, provider prefix must NOT go in the model field — use bare deepseek/deepseek-v4-flash-0731 + separate provider: openrouter. Sending openrouter/deepseek/... as the model ID → "not a valid model ID" → falls to a rate-limited free model then freetoken's tiny 8,448-token budget → job dies. |
Agent OS cron consolidation setup |
| 2026-08-29 | Cron jobs inherit the full default-toolset system prompt (~16,931 tokens context). Huge context overflows tiny fallback models with small KV budgets. Give cron jobs a minimal toolset (e.g. read, files) like check-agent-discussion (~2,648 tokens). |
Agent OS cron consolidation setup |
| 2026-08-29 | Files in a session that came back as base64 text on disk (memory_policy.md, lessons.md) were real corruption — the decoder (read_file) hid it. Always verify a file's on-disk content is plain markdown, not base64. |
Vault Agent OS setup — base64 corruption bug |
| 2026-08-29 | /model <name> in Hermes is session-scoped; use --global or hermes config set model.default to persist. Config/status showing a different model than the live session = split-brain from session-level switches. |
Model switching permanence |
| 2026-08-29 | `hermes fallback add | removeare interactive pickers that reject bare model strings; set the wholefallback_providerslist viahermes config set` instead. |
| 2026-09-24 | Bonsai 2 27B + MTP verified end-to-end on the 3060: decode 48.6-50.8 tok/s, cold prefill 241 tok/s, VRAM 11,166 MiB used / 950 free at -c 131072 - matches the advertised ~50 tok/s. Fixed an intermittent EMPTY-content failure by adding --reasoning-effort medium to /opt/serve-bonsai-mtp.sh (template default xhigh let reasoning eat the whole max_tokens budget: 2,524ch reasoning, finish_reason=length, empty content). Do NOT misdiagnose that as the /new context trap or the post-abort scheduler wedge - /health is 200 and decode speed is normal; compare reasoning_content length vs usage.completion_tokens. Also: the 'official PrismML installer' (Bonsai-demo/PQ2_0/:8080) advice is SUPERSEDED on this box by the sudoingX PTQ1_0+MTP build at :8899, and a Windows-native CUDA build of the fork remains impossible (MSVC 14.44 crashes CUDA 12.4 cudafe++). Full note: facts/2026-09-24-bonsai2-mtp-verified.md |
Bonsai 2 verification + reasoning-effort fix |