# Leonard — Lessons Learned

## Reusable lessons for the Leonard agent. One entry per lesson.
> Newest first.

| Date | Lesson | Context |
|------|--------|---------|
| 2026-09-17 | MiniCPM5-2B (FreeToken 127.0.0.1:1919) returns EMPTY/thinking-only completions under Hermes agent turns (44× `Empty response (no content or reasoning)` in errors.log, Sep 12–17), yet answers cleanly to direct curl (small/20K/22K prompts, tools, SSE, 4-way concurrency, multi-turn). Failure is model-vs-agent-load, not server/KV. Same box also logged `prompt is too long: 17105 > 8585 max (prompt + generation)` (Sep 15, Gemma) proving FreeToken real KV budget << advertised 131K. Fix: route agent turns to routeapi/deepseek-v4-flash-0731; keep 2B models for light direct use only. | Profile audit + MiniCPM5 diagnosis (2026-09-17) |
| 2026-09-17 | `providers.freetoken.max_tokens: 2000` is silently IGNORED (`unknown config keys ignored: max_tokens`) — key placement invalid for this schema, so the generation cap never applies. Also `model_aliases.minicpm5` had a duplicate `model:` key (Gemma dict + MiniCPM scalar) so the alias didn't point where labeled. | Profile audit (2026-09-17) |
| 2026-09-17 | `hermes cron create` grammar: the prompt positional MUST come immediately after the schedule and BEFORE any `--flags` (`hermes cron create <schedule> "<prompt>" --name ...`). Prompt after flags → `unrecognized arguments`. `--model`/`--provider` pin per-job to a working provider — restored jobs referencing `openrouter:deepseek` (dead provider) would silently fail. | Cron rebuild (2026-09-17) |
| 2026-09-17 | ROUTE_API_KEY is injected into the gateway env, NOT in `~/.hermes/profiles/leonard/.env` (grep `.env` comes up empty). The live session resolves routeapi fine. When auditing providers, check auth.json + gateway env, not just .env. | Profile audit (2026-09-17) |
| 2026-09-17 | git-bash grep does NOT support `\d` in ERE (`grep -E "^\d+"` silently matches nothing) — use `[0-9]`. Tailscale status on Windows prints peer rows as `IP name tag state`; capture `$1 $2 $3 $5` so the cron watch script sees online/offline transitions. | tailnet-watch.sh debug (2026-09-17) |
| 2026-09-17 | Tailscale health: this box (100.64.0.1) itself reports `offline` and health check warns it can't reach the coordination server — HTTPS front door `headscale.vps1.afterthedemo.com` is intercepted by a NETGEAR router login cert (CN=www.routerlogin.net). Matches known rule: control plane is HTTP-only at http://74.208.111.99:8443, never HTTPS. Watchdog will flag when state flips. | Profile audit, tailnet check (2026-09-17) |
| 2026-09-17 | Profile hygiene issue: leonard `cron/jobs.json` had been wiped to `{"jobs":[]}` (Sep 3) while `skills/.curator_backups/2026-08-27T18-02-27Z/cron-jobs.json` still held the 5 real jobs. Restore pattern: backup → parse JSON → re-create via `hermes cron create` with corrected provider pins + profile-scripts (curator-run.sh, havoc-nightly.sh, tailnet-watch.sh). Always verify pins in jobs.json after create. | Cron rebuild (2026-09-17) |
| 2026-09-01 | VPS SSH (74.208.111.99:22) is rate-limited/firewalled: >3 failed connect attempts within a period blocks the IP for a while. Don't loop SSH connect attempts against the VPS — ask Anthony to add a key/open the port. Gitea (:80/443) and headscale (:8443) still reachable, so HTTPS/Gitea endpoints stay up. | Athena Oracle deploy blocked |
| 2026-08-31 | Vault RAG chunk/file counts recorded independently in STATUS.md, HOME.md, and 01_user/current-context.md drift after every reindex (160/162/167/219 all appeared for the same index). Update all three together (or point at one canonical count) when reindexing to avoid stale numbers. | AgentOS merge reindex |
| 2026-08-30 | LFM2.5-2.6B is both agentic AND a pure reasoning model. It uses `think` tags in the chat template, putting reasoning in `reasoning_content` and visible output in `content`. Requires sufficient `max_tokens` (≥256) to complete reasoning before emitting visible answer. | LFM2.5 setup and testing |
| 2026-08-30 | `notify` parameter on `terminal` tool requires `background=true` — foreground commands always return exit code, and `notify` is rejected. Use background mode for daemons/services. | LFM2.5 launch debugging |
| 2026-08-30 | Llama.cpp `--n-gpu-layers 999` is required for full CUDA offload on Windows. Without it, the model runs on CPU only. | LFM2.5 setup |
| 2026-08-30 | LFM2.5-2.6B (Q4_K, ~2.1GB) + FreeToken gpt-oss-20b (~12.8GB) cannot coexist on RTX 3060 12GB. Only one can run at a time. | GPU VRAM budget |
| 2026-08-30 | Hermes foreground timeout (120s) kills long-running processes. Use `background=true` for daemons, then verify with separate curl. | Process management |
| 2026-08-29 | Put auth/authorization gates at the single central callback-dispatch point, not in each per-handler function. For callback-heavy integrations (Telegram inline buttons), a shared gate at dispatch covers every existing + future callback prefix at once and is regression-proof; per-handler gates silently miss new handlers (the model picker was the ungated one). | Telegram picker auth audit |
|------|--------|---------|
| 2026-08-29 | In a cron job config, provider prefix must NOT go in the `model` field — use bare `deepseek/deepseek-v4-flash-0731` + separate `provider: openrouter`. Sending `openrouter/deepseek/...` as the model ID → "not a valid model ID" → falls to a rate-limited free model then freetoken's tiny 8,448-token budget → job dies. | Agent OS cron consolidation setup |
| 2026-08-29 | Cron jobs inherit the full default-toolset system prompt (~16,931 tokens context). Huge context overflows tiny fallback models with small KV budgets. Give cron jobs a minimal toolset (e.g. `read, files`) like `check-agent-discussion` (~2,648 tokens). | Agent OS cron consolidation setup |
| 2026-08-29 | Files in a session that came back as base64 text on disk (memory_policy.md, lessons.md) were real corruption — the decoder (`read_file`) hid it. Always verify a file's on-disk content is plain markdown, not base64. | Vault Agent OS setup — base64 corruption bug |
| 2026-08-29 | `/model <name>` in Hermes is session-scoped; use `--global` or `hermes config set model.default` to persist. Config/status showing a different model than the live session = split-brain from session-level switches. | Model switching permanence |
| 2026-08-29 | `hermes fallback add|remove` are interactive pickers that reject bare model strings; set the whole `fallback_providers` list via `hermes config set` instead. | Model switching permanence |