jimmy
125 lines
5.3 KiB
Markdown
125 lines
5.3 KiB
Markdown
# OmniRoute auto-combo failover — the 200-with-empty-body trap
|
|
|
|
> Failure mode + fix. Diagnosed 2026-09-12 while repairing the AshBot profile
|
|
> (see [[04_projects/ashbot/status]]). Applies to **any** Hermes profile whose primary
|
|
> model is an OmniRoute `auto/*` combo.
|
|
|
|
## Symptom
|
|
|
|
A profile on an `auto/*` combo (e.g. `auto/best-free`) runs for minutes, then the turn ends
|
|
with a stub or an error reply — often mid-task, with tool calls already made.
|
|
|
|
```
|
|
agent.stream_diag: Stream exhausted on attempt 3/3 — retrying
|
|
error_type=EmptyStreamError http_status=200 bytes=0 chunks=0
|
|
agent.conversation_loop: API call failed after 3 retries
|
|
provider=custom model=auto/best-free
|
|
```
|
|
|
|
## Root cause
|
|
|
|
The target pool for the built-in `auto/*` combos is **dynamic** — there is no combo row to
|
|
edit. OmniRoute scores a candidate pool from every configured provider each request.
|
|
|
|
1. The combo's primary is the OpenRouter free pool. It rate-limits constantly
|
|
(`429 Provider returned error`), and the connection is marked
|
|
`unavailable / rate_limited` in `provider_connections`.
|
|
2. The chain then falls through to targets that are **broken but look healthy**:
|
|
- `opencode/minimax-m3-free` → `401 Model minimax-m3-free is not supported`
|
|
- `auggie/claude-sonnet-4.6`, `auggie/gemini-3.1-pro` → **HTTP 200 with a 0-byte body**,
|
|
~1.0 s, repeatedly
|
|
3. OmniRoute counts a 200 as success. The provider circuit breaker only trips on
|
|
`408/500/502/503/504`, so **a 200-with-empty-body never trips it, never cools down, and
|
|
never fails over**. The combo returns the empty stream to the client.
|
|
4. Hermes sees an empty SSE → `EmptyStreamError` → retries → turn dies.
|
|
|
|
**Never assume "200 = healthy" when reading OmniRoute's `call_logs`.**
|
|
|
|
## Quirk: `model.fallback:` is not the failover chain
|
|
|
|
A profile can look protected and have zero failover. In `config.yaml`:
|
|
|
|
- `model.fallback: custom:omniroute/auto/best-free` — ignored as a chain
|
|
- top-level `fallback_providers:` — **the real chain**
|
|
|
|
```bash
|
|
hermes -p <name> fallback list # "No fallback providers configured" = no safety net
|
|
```
|
|
|
|
Hermes does have explicit empty-response and retry-exhausted fallback paths
|
|
(`agent/turn_empty_response.py`, `agent/turn_api_error.py`) — they just never fire without
|
|
a chain.
|
|
|
|
## Fix A — give the profile its own failover chain
|
|
|
|
Back up `config.yaml`, then append:
|
|
|
|
```yaml
|
|
fallback_providers:
|
|
- provider: custom:omniroute
|
|
model: openrouter/deepseek/deepseek-v4.1-flash
|
|
- provider: custom:omniroute
|
|
model: freetoken-gemma26b/Gemma-4-26B-A4B-NVFP4 # local, free, no rate limits
|
|
```
|
|
|
|
- `custom:<name>` must match a `providers:` entry in that profile's config.
|
|
- **No gateway restart needed** — the gateway re-reads `fallback_providers` from disk for the
|
|
next agent create/reuse.
|
|
- **Prove it fires, don't assume.** Run one-shot with a deliberately bogus primary, then check
|
|
which model actually answered:
|
|
|
|
```bash
|
|
hermes -p <name> -m "openrouter/does-not-exist" -z "Reply with exactly: FB-OK"
|
|
# then: state.db → session_model_usage → the row names the model that really served it
|
|
```
|
|
|
|
## Fix B — disable the broken no-auth providers
|
|
|
|
The mechanism is the `blockedProviders` setting (matches provider **id or alias**). For true
|
|
no-auth providers (`auggie`, `opencode`, `theoldllm`, …) this is the documented disable switch.
|
|
|
|
The management API is **not reachable on this Windows install**: `GET /api/settings` returns
|
|
401 even from loopback with the CLI machine token, because the server's `node-machine-id`
|
|
require fails inside the standalone bundle — its expected token is empty, and `doctor`
|
|
reports the health endpoint as "likely requires MANAGEMENT_TOKEN". Write the setting directly
|
|
instead:
|
|
|
|
```python
|
|
import sqlite3, json, os
|
|
con = sqlite3.connect(os.path.expanduser("~/.omniroute/storage.sqlite"), timeout=20)
|
|
con.execute("INSERT OR REPLACE INTO key_value (namespace,key,value) VALUES ('settings','blockedProviders',?)",
|
|
(json.dumps(["auggie", "opencode"]),))
|
|
con.commit()
|
|
```
|
|
|
|
Settings cache TTL is 5 s, so it applies with no restart. Verify on the unauthenticated
|
|
catalog endpoint — the provider's models disappear and the total count drops:
|
|
|
|
```bash
|
|
curl -s localhost:20128/v1/models | tr ',' '\n' | grep -c '"id":"'
|
|
```
|
|
|
|
## Where OmniRoute actually lives
|
|
|
|
| Thing | Path |
|
|
|---|---|
|
|
| Running server | global npm package `~/AppData/Roaming/npm/node_modules/omniroute` (`node dist/server-ws.mjs`), v3.8.48 |
|
|
| Data dir | `~/.omniroute` — `storage.sqlite` (`call_logs`, `key_value`, `provider_connections`, `combos`), `call_logs/<date>/` |
|
|
| Repo checkout | `~/AppData/Local/hermes/OmniRoute` — **not** the running server; source of truth for behavior/config keys |
|
|
|
|
## Verification of this fix (2026-09-12)
|
|
|
|
- `fallback list` shows both entries.
|
|
- Forced primary failure → reply returned, and `session_model_usage` records
|
|
`openrouter/deepseek/deepseek-v4.1-flash` via `custom:omniroute`.
|
|
- Catalog after the block: **476 → 453** models; no `aug/*` or `oc/*` entries remain.
|
|
- No other profile referenced those two providers.
|
|
|
|
## Open issues
|
|
|
|
- OmniRoute's loopback CLI-token management auth (401) — blocks API-driven config on Windows.
|
|
- The primary is still a free-tier roulette: a failure now costs the retry window
|
|
(~1 min typical, up to ~4.5 min observed) before recovering instead of dying.
|
|
|
|
See also: [[01_Wiki/hermes-gitea-ecosystem-manual]]
|