2026-09-19 — Bonsai 2 fast build + MTP head on the 3060 (setup complete, verified)

What was built

The sudoingX bonsai2-small-gpu recipe, end to end, on DESKTOP-81CBTV2:

Fork: bonsai2 branch of sudoingX/llama.cpp (= PrismML fork + kernel PR #218 + Hadamard fix #217, stacked). Built in WSL2 (Ubuntu 26.04) with gcc-13 + CUDA 12.4, sm_86. Source /opt/llama-prism-sudoingx, binaries build/bin/.
Model: Ternary-Bonsai-2-27B-PTQ1_0.gguf (5.95 GB, prism-ml HF repo) + Qwen 3.8 27B MTP head grafted on via the repo's byte-exact GGUF tools → Ternary-Bonsai-2-27B-PTQ1_0-mtp.gguf (7.0 GB) in /opt/models/ (WSL ext4). Graft round-trip verified byte-exact (strip = original sha256).
Server: WSL2 llama-server on 127.0.0.1:8899, reachable from Windows via WSL2 localhost forwarding (bound 0.0.0.0). MTP spec decode on, n-max 1, GGML_CUDA_BATCH_INVARIANT=1. Launchers /opt/serve-bonsai-mtp.sh / /opt/serve-bonsai-off.sh.

Verified numbers (this machine, RTX 3060 12GB)

metric	result	author's claim
llama-bench tg128 (fresh)	39.34 tok/s	40.47
llama-bench pp512 (prefill)	272.7 tok/s	~268
streaming, code	53.4 tok/s	53.2
streaming, prose	41.9	41.8
streaming, bash	48.8	50.1
greedy identity (off vs on)	byte-identical, all 3 prompts	identical
test-backend-ops ptq1_0 MUL_MAT	all pass vs CPU ref	47/47

Stock fork baseline was 26.3 tok/s → ~49 streaming overall. Lossless (outputs unchanged), not just fast.

Why WSL2

Windows-native build is blocked on this PC: MSVC 14.44/14.51 (VS 18 Build Tools) hard-crashes CUDA 12.4's cudafe++ (ACCESS_VIOLATION — too-new compiler), no admin rights to install CUDA 13.x or an older MSVC, no clang, and the pip nvidia-cuda-nvcc-cu12 wheels ship only ptxas.exe on Windows (no nvcc). WSL2 has native GPU passthrough (nvidia-smi works) and CUDA 12.4 works with gcc-13. This matches the author's own tested toolchain.

Config

providers.bonsai2 + model_aliases.bonsai2 → http://127.0.0.1:8899/v1, model /opt/models/Ternary-Bonsai-2-27B-PTQ1_0-mtp.gguf, ctx 131072. Backup config.yaml.good-20260919221502. Gateway restart required (hermes --profile leonard gateway restart from a separate shell) then /new + /model bonsai2.

Pitfalls learned

pkill -f is case-sensitive: "bonsai" doesn't match Ternary-Bonsai in the model path. Kill by PID.
WSL2 servers must bind --host 0.0.0.0; 127.0.0.1 inside WSL is invisible from Windows.
Keep the model in WSL ext4, not /mnt/c (9p slowness).
Hermes venv python refuses --user installs and nesting venvs — use uv venv.
One model at a time on the 3060 still applies (FreeToken ~1 GB co-resident is fine; the MTP server needs ~10.6 GB).

Disk footprint

22.4 GB downloads (PTQ1_0 + Qwen donor), +7 GB merged model, +5.9 GB strip-check (cleanup candidate: C:\models\bonsai2-mtp\strip-check.gguf). Source/donor can stay for re-grafting.

Cross-ref

Skills: bonsai2-local-inference (updated 2026-09-19 with canonical setup), call-3060-inference.

3.0 KiB Raw Blame History