jimmy
1.7 KiB
1.7 KiB
Local Inference Failure Ledger — Bonsai 2 / 3060
Purpose: track every attempted local-inference request, success/failure, latency, root cause. Rule: every tested model run gets one row. Update after each test, not after each crisis.
| # | Date (PST) | Model | Request size | Result | Latency | Root cause / notes |
|---|---|---|---|---|---|---|
| 1 | 2026-09-18 ~07:00 | Bonsai-2-27B Vulkan build | 62 tok | SUCCESS | 145s | PQ2_0 via generic Vulkan kernels — no Vulkan kernels in fork |
| 2 | 2026-09-18 ~07:11 | Bonsai-2-27B CUDA build | 62 tok | SUCCESS | 4.6s | After CUDA build swap — 32x faster |
| 3 | 2026-09-18 ~07:20 | Bonsai-2-27B CUDA (3 models loaded) | 2048 tok | FAIL | ~56s then cancel | VRAM 9.3/12GB used, model spilled ~5GB to RAM; prefill 6% after 27s; Hermes 60s client timeout cancelled |
| 4 | 2026-09-18 ~07:24 | Bonsai-2-27B CUDA (card dedicated) | 3971 tok | SUCCESS | 84.9s | Full model in VRAM 11969/12288 MiB; slow but works |
| 5 | 2026-09-18 ~07:24 | Bonsai-2-27B CUDA (dedicated) | 61 tok | SUCCESS | 6.8s | content correct |
| 6 | 2026-09-18 ~07:31 | Bonsai-2-27B via Hermes picker | ~72,700 tok | FAIL | ~7s → abort | Context overflow: session 72.7k tokens > Bonsai ctx 65,536. Hermes pre-call compression failed 2x (failure_class=lock_contended), then request aborted WinError 10053. Not a model fault. Fix: /new before bonsai2, or reload with -c 131072 |
Baseline so far: 4/5 success; the one failure was resource contention (my error), not the model. Open: track picker-switch requests through real agent traffic, not just hand tests.
Status: TRACKING SINCE 2026-09-18. No telemetry existed before this file.