OmniRoute auto-combo failover — the 200-with-empty-body trap

Failure mode + fix. Diagnosed 2026-09-12 while repairing the AshBot profile (see 04_projects/ashbot/status). Applies to any Hermes profile whose primary model is an OmniRoute auto/* combo.

Symptom

A profile on an auto/* combo (e.g. auto/best-free) runs for minutes, then the turn ends with a stub or an error reply — often mid-task, with tool calls already made.

agent.stream_diag: Stream exhausted on attempt 3/3 — retrying
  error_type=EmptyStreamError http_status=200 bytes=0 chunks=0
agent.conversation_loop: API call failed after 3 retries
  provider=custom model=auto/best-free

Root cause

The target pool for the built-in auto/* combos is dynamic — there is no combo row to edit. OmniRoute scores a candidate pool from every configured provider each request.

The combo's primary is the OpenRouter free pool. It rate-limits constantly (429 Provider returned error), and the connection is marked unavailable / rate_limited in provider_connections.
The chain then falls through to targets that are broken but look healthy:
- opencode/minimax-m3-free → 401 Model minimax-m3-free is not supported
- auggie/claude-sonnet-4.6, auggie/gemini-3.1-pro → HTTP 200 with a 0-byte body, ~1.0 s, repeatedly
OmniRoute counts a 200 as success. The provider circuit breaker only trips on 408/500/502/503/504, so a 200-with-empty-body never trips it, never cools down, and never fails over. The combo returns the empty stream to the client.
Hermes sees an empty SSE → EmptyStreamError → retries → turn dies.

Never assume "200 = healthy" when reading OmniRoute's call_logs.

Quirk: `model.fallback:` is not the failover chain

A profile can look protected and have zero failover. In config.yaml:

model.fallback: custom:omniroute/auto/best-free — ignored as a chain
top-level fallback_providers: — the real chain

hermes -p <name> fallback list     # "No fallback providers configured" = no safety net

Hermes does have explicit empty-response and retry-exhausted fallback paths (agent/turn_empty_response.py, agent/turn_api_error.py) — they just never fire without a chain.

Fix A — give the profile its own failover chain

Back up config.yaml, then append:

fallback_providers:
- provider: custom:omniroute
  model: openrouter/deepseek/deepseek-v4.1-flash
- provider: custom:omniroute
  model: freetoken-gemma26b/Gemma-4-26B-A4B-NVFP4   # local, free, no rate limits

custom:<name> must match a providers: entry in that profile's config.
No gateway restart needed — the gateway re-reads fallback_providers from disk for the next agent create/reuse.
Prove it fires, don't assume. Run one-shot with a deliberately bogus primary, then check which model actually answered:

hermes -p <name> -m "openrouter/does-not-exist" -z "Reply with exactly: FB-OK"
# then: state.db → session_model_usage → the row names the model that really served it

Fix B — disable the broken no-auth providers

The mechanism is the blockedProviders setting (matches provider id or alias). For true no-auth providers (auggie, opencode, theoldllm, …) this is the documented disable switch.

The management API is not reachable on this Windows install: GET /api/settings returns 401 even from loopback with the CLI machine token, because the server's node-machine-id require fails inside the standalone bundle — its expected token is empty, and doctor reports the health endpoint as "likely requires MANAGEMENT_TOKEN". Write the setting directly instead:

import sqlite3, json, os
con = sqlite3.connect(os.path.expanduser("~/.omniroute/storage.sqlite"), timeout=20)
con.execute("INSERT OR REPLACE INTO key_value (namespace,key,value) VALUES ('settings','blockedProviders',?)",
            (json.dumps(["auggie", "opencode"]),))
con.commit()

Settings cache TTL is 5 s, so it applies with no restart. Verify on the unauthenticated catalog endpoint — the provider's models disappear and the total count drops:

curl -s localhost:20128/v1/models | tr ',' '\n' | grep -c '"id":"'

Where OmniRoute actually lives

Thing	Path
Running server	global npm package `~/AppData/Roaming/npm/node_modules/omniroute` (`node dist/server-ws.mjs`), v3.8.48
Data dir	`~/.omniroute` — `storage.sqlite` (`call_logs`, `key_value`, `provider_connections`, `combos`), `call_logs/<date>/`
Repo checkout	`~/AppData/Local/hermes/OmniRoute` — not the running server; source of truth for behavior/config keys

Verification of this fix (2026-09-12)

fallback list shows both entries.
Forced primary failure → reply returned, and session_model_usage records openrouter/deepseek/deepseek-v4.1-flash via custom:omniroute.
Catalog after the block: 476 → 453 models; no aug/* or oc/* entries remain.
No other profile referenced those two providers.

Open issues

OmniRoute's loopback CLI-token management auth (401) — blocks API-driven config on Windows.
The primary is still a free-tier roulette: a failure now costs the retry window (~1 min typical, up to ~4.5 min observed) before recovering instead of dying.

5.3 KiB Raw Blame History