vault-sync: 2026-09-15 12:00 (jimmy)

memory-systems/agent-os/04_projects/ashbot/status.md

@@ -2,7 +2,7 @@
 
 > Personal assistant agent for **Ashley** (Phoenix Salon + Spa — Veripath's active client),
 > delivered over **iMessage via Photon**. Hermes profile `ashbot`.
-> Last worked: 2026-09-12.
+> Last worked: 2026-09-15.
 
 ## Identity
 
@@ -13,7 +13,7 @@
 | Channel | Photon → iMessage. **Own Photon project**, separate from the coach line |
 | Reachable by | Ashley (home channel + allowed user); Tony tests from his own line |
 | Sidecar | Photon sidecar on `127.0.0.1:8790` |
-| Gateway | PID 17968, platform `photon: connected`, Hermes 0.21.1 |
+| Gateway | PID 10384, platform `photon: connected`, Hermes 0.21.2 |
 
 ## Model stack (changed 2026-09-12)
 
@@ -45,6 +45,25 @@ the failover chain. See [[01_Wiki/omniroute-routing-failover]] for the failure m
   `config.yaml.bak.20260912_133720`) **and** `blockedProviders = ["auggie","opencode"]` set in
   OmniRoute (`~/.omniroute/storage.sqlite`). Catalog went 476 → 453 models.
 
+### 2026-09-15 — fleet-wide gateway outage after Hermes 0.21.2 auto-update (fixed)
+
+- 07:27–07:31 PT. The 0.21.2 auto-update requested graceful restarts of every profile's
+  gateway (ashbot 07:31:27, leonard 07:31:35, jimmy 07:31:45); the post-restart relaunch
+  never fired, so all gateways stayed down. The machine had rebooted ~07:27 — login items
+  started gateways, the update then stopped them, nothing restarted them.
+- Ashbot additionally lost its whole model provider: the OmniRoute server itself was dead
+  (no process, nothing on :20128 — it has no service/login item and does not survive a
+  reboot). Restarted via `omniroute serve --no-open --daemon` (NOT `node
+  dist/server-ws.mjs`, which serves only Dashboard/WS and never the 20128 API).
+- Root-caused the combo's per-turn lag: `theoldllm` targets 403'd on every request and
+  OpenRouter connection was `credits_exhausted`, so `auto/best-free` burned ~6-20 s/turn
+  on dead hops before landing on the local freetoken Gemma engine (which answers every
+  turn today). Combo verdict logs `499 "all targets exhausted"` even when the last
+  successful step's body was delivered.
+- Fixed: `blockedProviders` += `theoldllm` (catalog 453 → 427 models); `fallback_providers`
+  chain verified still registered (`hermes -p ashbot fallback list`). Turn latency back to
+  ~10 s. Live check-ins continue to route `omniroute` → local Gemma-4-26B.
+
 ## Active automation
 
 - **Cron job `e4df5902e96f` — "Tony daily check-in"**, `0 9 * * *`, provider pinned
@@ -59,6 +78,11 @@ the failover chain. See [[01_Wiki/omniroute-routing-failover]] for the failure m
   exceed it — cap her output or chunk it.
 - **Primary is still the free roulette.** A failure now costs the retry window (~1 min
   typical, up to ~4.5 min) before the fallback takes over, instead of killing the turn.
+- **OpenRouter connection is `credits_exhausted`** (2026-09-15). Every `auto/*` combo hop
+  that tries OpenRouter dies instantly and the chain leans on the local Gemma engine; this
+  is also why the coach profile 402s. Decision needed: top up, or accept local-first.
+  Combo still tries ~3-4 fast-failing no-auth targets per turn (duckduckgo-web 418,
+  chipotle 502, mimocode 499) before the local hop — sub-second each, cosmetic.
 - **Browser mid-scan hiccup** — `browser_exec` dropped during the marketplace scan; the retry
   path is untested since the model fix.
 
@@ -67,7 +91,7 @@ the failover chain. See [[01_Wiki/omniroute-routing-failover]] for the failure m
 | Path | Change |
 |---|---|
 | `profiles/ashbot/config.yaml` | `+ fallback_providers` (2026-09-12); backup alongside |
-| `~/.omniroute/storage.sqlite` | `settings.blockedProviders` = `["auggie","opencode"]` |
+| `~/.omniroute/storage.sqlite` | `settings.blockedProviders` = `["auggie","opencode","theoldllm"]` (2026-09-15) |
 | Hermes skill | `omniroute-routing-failover` (jimmy profile) |
 
 See also: [[04_projects/phoenix-salon/status]] · [[04_projects/veripath/status]] ·