vault-sync: commit pending changes from Leonard session

2026-09-13 11:46:28 -07:00
parent 8b26efd3b7
commit 1255434c1a
25 changed files with 409 additions and 57 deletions
@@ -0,0 +1,55 @@
+# Inference Performance Benchmarks — Local Hardware
+
+**Machine:** Windows 11, RTX 3060 12GB  
+**Date:** 2026-09-13
+
+## FreeToken Benchmarks
+
+| Model | Engine | Tokens/sec | Notes |
+|-------|--------|------------|-------|
+| Gemma-4-26B-A4B-NVFP4 | FreeToken | ? | 262K context, loaded as default |
+| gpt-oss-20b | FreeToken | **50+** | Amazingly fast for 20B on 12GB card |
+
+## Why gpt-oss-20b Hits 50 tok/s on a 3060
+
+From the [AI Bottleneck Thesis](ai-bottleneck-thesis-tony-simons.md):
+
+> *"Inference is memory-bound, not compute-bound. The ceiling is `BW / (P × b)`"*
+
+### The FreeToken Advantage: Heterogeneous MoE Offload
+
+FreeToken uses **Mixture of Experts (MoE) technology** that doesn't just run on GPU. It creates a **heterogeneous memory hierarchy**:
+
+| Tier | Hardware | Role |
+|------|----------|------|
+| **Tier 1 (fastest)** | GPU VRAM (12GB) | Active expert weights, KV cache, hot path |
+| **Tier 2** | System RAM (~32GB) | Inactive expert weights, overflow |
+| **Tier 3** | CPU | Orchestration, expert routing, offload management |
+
+This means:
+- **Not all 20B parameters need to be in VRAM simultaneously**
+- Only the **active experts** for a given token need GPU memory
+- The rest sits in system RAM and gets paged in on demand
+- **Result:** A 20B model runs on a 12GB card because at any given moment, only ~6-8B parameters are "hot"
+
+### The Math (Revised)
+
+- **RTX 3060:** 12GB VRAM, ~360 GB/s bandwidth
+- **Active experts per token:** ~6-8B parameters (estimated)
+- **Theoretical ceiling:** 360 / (8B × 0.5 bytes) = ~90 tok/s
+- **Achieved:** 50+ tok/s → overhead from CPU-GPU data transfer, but still excellent
+- **Without MoE offload:** 20B × 0.5 bytes = 10GB → barely fits, no room for KV cache
+
+## Key Insight
+
+The 3060's **12GB VRAM capacity** is the reason this works. Smaller cards (3060 Ti with 8GB) couldn't fit the model. The memory bandwidth (360 GB/s) is the ceiling — and we're hitting above theoretical because FreeToken's CUDA kernels are well-optimized.
+
+## Constraints
+
+- **Cannot coexist with LFM2.5:** Combined ~15GB > 12GB VRAM
+- **Context size limited:** ~128K context at Q4 before running out of VRAM
+- **Model swap required:** Need to stop one to start the other
+
+## Decision Signal
+
+For local agent work requiring **high throughput on a single question**, gpt-oss-20b at 50 tok/s is a strong choice. For **long-running daemon tasks**, LFM2.5 on llama-cpp (port 1920) is more appropriate.