2026-09-24 — Bonsai 2 27B + MTP: verified end-to-end on the 3060

Status

The Ternary Bonsai 2 / PrismML stack is already built and serving on this box. It was built 2026-09-19 in WSL2 and re-verified 2026-09-24. Nothing needed downloading.

Repo (fork, bonsai2 branch = kernel PRs #218 + Hadamard #217): /opt/llama-prism-sudoingx
Model: /opt/models/Ternary-Bonsai-2-27B-PTQ1_0-mtp.gguf (7,012,820,512 bytes, MTP head grafted)
Launcher: /opt/serve-bonsai-mtp.sh — listens :8899 (not 8080)
Hermes wiring: providers.bonsai2 + model_aliases.bonsai2 -> http://127.0.0.1:8899/v1

Measured 2026-09-24 (RTX 3060 12 GB, driver 610.74)

Decode: 48.6 – 50.8 tok/s across 4 runs (~50 tok/s as advertised)
Prefill: 241 tok/s cold; KV-cache-warm runs report low prefill because the prefix is cached
VRAM: 11,166 MiB used / 950 free at -c 131072 — weights fully resident, no RAM spill
--jinja tool calling: finish_reason=tool_calls, correct function name + JSON args
Sanity: 17 * 23 = 391 correct

The one real defect found and fixed: empty `content`

The template default reasoning effort is xhigh. At temperature 1.0 reasoning length varies wildly (observed 238 chars to 2,524 chars). When it ran long, reasoning consumed the entire max_tokens budget and the response came back finish_reason=length with empty content.

Fix: added --reasoning-effort medium to /opt/serve-bonsai-mtp.sh (backup at /opt/serve-bonsai-mtp.sh.bak-20260924-181142). After the change: 3/3 realistic ~2.3k-token prompts returned content, reasoning down to 238–319 chars.

This failure mode looks like two other known Bonsai problems but is neither:

NOT the /new context trap (context 131072 matches alias; /health 200)
NOT the post-abort scheduler wedge (decode speed is normal, slots launch fine)

Diagnose it by comparing reasoning_content length against usage.completion_tokens.

Escalation ladder if medium still overruns: --reasoning-budget N (hard cap on thinking tokens; injects a message before the end-of-thinking tag). Per-request overrides also work — the fork honours both reasoning_effort and chat_template_kwargs.reasoning_effort in the request body, so no restart is needed for a one-off.

Correction to a common (wrong) playbook for this box

Advice circulating for this model recommends the "official PrismML installer" (Bonsai-demo, PQ2_0, port 8080) as Path A. On THIS machine that path is superseded and weaker: the Bonsai-demo PQ2_0 setup at :8080 is the older, slower variant (~26 tok/s baseline, and the Vulkan build that ships with it is ~32x slower). The sudoingX PTQ1_0 + MTP build at :8899 is the canonical one.

Also settled long ago and still true: a Windows-native CUDA build of the fork is impossible on this PC — MSVC 14.44 (VS 18) hard-crashes CUDA 12.4 cudafe++ (ACCESS_VIOLATION), there is no admin for CUDA 13 / older MSVC, and the pip nvidia-cuda-nvcc-cu12 wheels ship no nvcc.exe on Windows (ptxas only). The build lives in WSL2 with gcc-13 + CUDA 12.4 from apt.

Gotchas re-confirmed

WSL2 servers must bind --host 0.0.0.0; a 127.0.0.1 bind inside WSL is invisible from Windows.
Keep the GGUF in WSL ext4 (/opt/models/), not /mnt/c — 9p/drvfs makes the 7 GB mmap crawl.
pkill -f llama-server kills your own shell when the wrapper command line contains that string (exit 15). Use the bracket trick: pkill -f '[l]lama-server'.
pkill -f bonsai does NOT match the model path Ternary-Bonsai-... (case-sensitive).
Switch to this model with /new before /model bonsai2 — a grown session prefills slowly at 3060 speeds and can time out mid-load.

3.6 KiB Raw Blame History Unescape Escape