vault-sync: 2026-09-13 12:07 (jimmy)

2026-09-13 12:07:23 -07:00
parent e918f23707
commit aa52eecb5c
50 changed files with 4101 additions and 103 deletions
@@ -0,0 +1,124 @@
+# OmniRoute auto-combo failover — the 200-with-empty-body trap
+
+> Failure mode + fix. Diagnosed 2026-09-12 while repairing the AshBot profile
+> (see [[04_projects/ashbot/status]]). Applies to **any** Hermes profile whose primary
+> model is an OmniRoute `auto/*` combo.
+
+## Symptom
+
+A profile on an `auto/*` combo (e.g. `auto/best-free`) runs for minutes, then the turn ends
+with a stub or an error reply — often mid-task, with tool calls already made.
+
+```
+agent.stream_diag: Stream exhausted on attempt 3/3 — retrying
+  error_type=EmptyStreamError http_status=200 bytes=0 chunks=0
+agent.conversation_loop: API call failed after 3 retries
+  provider=custom model=auto/best-free
+```
+
+## Root cause
+
+The target pool for the built-in `auto/*` combos is **dynamic** — there is no combo row to
+edit. OmniRoute scores a candidate pool from every configured provider each request.
+
+1. The combo's primary is the OpenRouter free pool. It rate-limits constantly
+   (`429 Provider returned error`), and the connection is marked
+   `unavailable / rate_limited` in `provider_connections`.
+2. The chain then falls through to targets that are **broken but look healthy**:
+   - `opencode/minimax-m3-free` → `401 Model minimax-m3-free is not supported`
+   - `auggie/claude-sonnet-4.6`, `auggie/gemini-3.1-pro` → **HTTP 200 with a 0-byte body**,
+     ~1.0 s, repeatedly
+3. OmniRoute counts a 200 as success. The provider circuit breaker only trips on
+   `408/500/502/503/504`, so **a 200-with-empty-body never trips it, never cools down, and
+   never fails over**. The combo returns the empty stream to the client.
+4. Hermes sees an empty SSE → `EmptyStreamError` → retries → turn dies.
+
+**Never assume "200 = healthy" when reading OmniRoute's `call_logs`.**
+
+## Quirk: `model.fallback:` is not the failover chain
+
+A profile can look protected and have zero failover. In `config.yaml`:
+
+- `model.fallback: custom:omniroute/auto/best-free` — ignored as a chain
+- top-level `fallback_providers:` — **the real chain**
+
+```bash
+hermes -p <name> fallback list     # "No fallback providers configured" = no safety net
+```
+
+Hermes does have explicit empty-response and retry-exhausted fallback paths
+(`agent/turn_empty_response.py`, `agent/turn_api_error.py`) — they just never fire without
+a chain.
+
+## Fix A — give the profile its own failover chain
+
+Back up `config.yaml`, then append:
+
+```yaml
+fallback_providers:
+- provider: custom:omniroute
+  model: openrouter/deepseek/deepseek-v4.1-flash
+- provider: custom:omniroute
+  model: freetoken-gemma26b/Gemma-4-26B-A4B-NVFP4   # local, free, no rate limits
+```
+
+- `custom:<name>` must match a `providers:` entry in that profile's config.
+- **No gateway restart needed** — the gateway re-reads `fallback_providers` from disk for the
+  next agent create/reuse.
+- **Prove it fires, don't assume.** Run one-shot with a deliberately bogus primary, then check
+  which model actually answered:
+
+```bash
+hermes -p <name> -m "openrouter/does-not-exist" -z "Reply with exactly: FB-OK"
+# then: state.db → session_model_usage → the row names the model that really served it
+```
+
+## Fix B — disable the broken no-auth providers
+
+The mechanism is the `blockedProviders` setting (matches provider **id or alias**). For true
+no-auth providers (`auggie`, `opencode`, `theoldllm`, …) this is the documented disable switch.
+
+The management API is **not reachable on this Windows install**: `GET /api/settings` returns
+401 even from loopback with the CLI machine token, because the server's `node-machine-id`
+require fails inside the standalone bundle — its expected token is empty, and `doctor`
+reports the health endpoint as "likely requires MANAGEMENT_TOKEN". Write the setting directly
+instead:
+
+```python
+import sqlite3, json, os
+con = sqlite3.connect(os.path.expanduser("~/.omniroute/storage.sqlite"), timeout=20)
+con.execute("INSERT OR REPLACE INTO key_value (namespace,key,value) VALUES ('settings','blockedProviders',?)",
+            (json.dumps(["auggie", "opencode"]),))
+con.commit()
+```
+
+Settings cache TTL is 5 s, so it applies with no restart. Verify on the unauthenticated
+catalog endpoint — the provider's models disappear and the total count drops:
+
+```bash
+curl -s localhost:20128/v1/models | tr ',' '\n' | grep -c '"id":"'
+```
+
+## Where OmniRoute actually lives
+
+| Thing | Path |
+|---|---|
+| Running server | global npm package `~/AppData/Roaming/npm/node_modules/omniroute` (`node dist/server-ws.mjs`), v3.8.48 |
+| Data dir | `~/.omniroute` — `storage.sqlite` (`call_logs`, `key_value`, `provider_connections`, `combos`), `call_logs/<date>/` |
+| Repo checkout | `~/AppData/Local/hermes/OmniRoute` — **not** the running server; source of truth for behavior/config keys |
+
+## Verification of this fix (2026-09-12)
+
+- `fallback list` shows both entries.
+- Forced primary failure → reply returned, and `session_model_usage` records
+  `openrouter/deepseek/deepseek-v4.1-flash` via `custom:omniroute`.
+- Catalog after the block: **476 → 453** models; no `aug/*` or `oc/*` entries remain.
+- No other profile referenced those two providers.
+
+## Open issues
+
+- OmniRoute's loopback CLI-token management auth (401) — blocks API-driven config on Windows.
+- The primary is still a free-tier roulette: a failure now costs the retry window
+  (~1 min typical, up to ~4.5 min observed) before recovering instead of dying.
+
+See also: [[01_Wiki/hermes-gitea-ecosystem-manual]]