# OmniRoute auto-combo failover — the 200-with-empty-body trap

> Failure mode + fix. Diagnosed 2026-09-12 while repairing the AshBot profile
> (see [[04_projects/ashbot/status]]). Applies to **any** Hermes profile whose primary
> model is an OmniRoute `auto/*` combo.

## Symptom

A profile on an `auto/*` combo (e.g. `auto/best-free`) runs for minutes, then the turn ends
with a stub or an error reply — often mid-task, with tool calls already made.

```
agent.stream_diag: Stream exhausted on attempt 3/3 — retrying
  error_type=EmptyStreamError http_status=200 bytes=0 chunks=0
agent.conversation_loop: API call failed after 3 retries
  provider=custom model=auto/best-free
```

## Root cause

The target pool for the built-in `auto/*` combos is **dynamic** — there is no combo row to
edit. OmniRoute scores a candidate pool from every configured provider each request.

1. The combo's primary is the OpenRouter free pool. It rate-limits constantly
   (`429 Provider returned error`), and the connection is marked
   `unavailable / rate_limited` in `provider_connections`.
2. The chain then falls through to targets that are **broken but look healthy**:
   - `opencode/minimax-m3-free` → `401 Model minimax-m3-free is not supported`
   - `auggie/claude-sonnet-4.6`, `auggie/gemini-3.1-pro` → **HTTP 200 with a 0-byte body**,
     ~1.0 s, repeatedly
3. OmniRoute counts a 200 as success. The provider circuit breaker only trips on
   `408/500/502/503/504`, so **a 200-with-empty-body never trips it, never cools down, and
   never fails over**. The combo returns the empty stream to the client.
4. Hermes sees an empty SSE → `EmptyStreamError` → retries → turn dies.

**Never assume "200 = healthy" when reading OmniRoute's `call_logs`.**

## Quirk: `model.fallback:` is not the failover chain

A profile can look protected and have zero failover. In `config.yaml`:

- `model.fallback: custom:omniroute/auto/best-free` — ignored as a chain
- top-level `fallback_providers:` — **the real chain**

```bash
hermes -p <name> fallback list     # "No fallback providers configured" = no safety net
```

Hermes does have explicit empty-response and retry-exhausted fallback paths
(`agent/turn_empty_response.py`, `agent/turn_api_error.py`) — they just never fire without
a chain.

## Fix A — give the profile its own failover chain

Back up `config.yaml`, then append:

```yaml
fallback_providers:
- provider: custom:omniroute
  model: openrouter/deepseek/deepseek-v4.1-flash
- provider: custom:omniroute
  model: freetoken-gemma26b/Gemma-4-26B-A4B-NVFP4   # local, free, no rate limits
```

- `custom:<name>` must match a `providers:` entry in that profile's config.
- **No gateway restart needed** — the gateway re-reads `fallback_providers` from disk for the
  next agent create/reuse.
- **Prove it fires, don't assume.** Run one-shot with a deliberately bogus primary, then check
  which model actually answered:

```bash
hermes -p <name> -m "openrouter/does-not-exist" -z "Reply with exactly: FB-OK"
# then: state.db → session_model_usage → the row names the model that really served it
```

## Fix B — disable the broken no-auth providers

The mechanism is the `blockedProviders` setting (matches provider **id or alias**). For true
no-auth providers (`auggie`, `opencode`, `theoldllm`, …) this is the documented disable switch.

The management API is **not reachable on this Windows install**: `GET /api/settings` returns
401 even from loopback with the CLI machine token, because the server's `node-machine-id`
require fails inside the standalone bundle — its expected token is empty, and `doctor`
reports the health endpoint as "likely requires MANAGEMENT_TOKEN". Write the setting directly
instead:

```python
import sqlite3, json, os
con = sqlite3.connect(os.path.expanduser("~/.omniroute/storage.sqlite"), timeout=20)
con.execute("INSERT OR REPLACE INTO key_value (namespace,key,value) VALUES ('settings','blockedProviders',?)",
            (json.dumps(["auggie", "opencode"]),))
con.commit()
```

Settings cache TTL is 5 s, so it applies with no restart. Verify on the unauthenticated
catalog endpoint — the provider's models disappear and the total count drops:

```bash
curl -s localhost:20128/v1/models | tr ',' '\n' | grep -c '"id":"'
```

## Where OmniRoute actually lives

| Thing | Path |
|---|---|
| Running server | global npm package `~/AppData/Roaming/npm/node_modules/omniroute` (`node dist/server-ws.mjs`), v3.8.48 |
| Data dir | `~/.omniroute` — `storage.sqlite` (`call_logs`, `key_value`, `provider_connections`, `combos`), `call_logs/<date>/` |
| Repo checkout | `~/AppData/Local/hermes/OmniRoute` — **not** the running server; source of truth for behavior/config keys |

## Verification of this fix (2026-09-12)

- `fallback list` shows both entries.
- Forced primary failure → reply returned, and `session_model_usage` records
  `openrouter/deepseek/deepseek-v4.1-flash` via `custom:omniroute`.
- Catalog after the block: **476 → 453** models; no `aug/*` or `oc/*` entries remain.
- No other profile referenced those two providers.

## Open issues

- OmniRoute's loopback CLI-token management auth (401) — blocks API-driven config on Windows.
- The primary is still a free-tier roulette: a failure now costs the retry window
  (~1 min typical, up to ~4.5 min observed) before recovering instead of dying.

See also: [[01_Wiki/hermes-gitea-ecosystem-manual]]