2.4 KiB
2.4 KiB
Inference Performance Benchmarks — Local Hardware
Machine: Windows 11, RTX 3060 12GB
Date: 2026-09-13
FreeToken Benchmarks
| Model | Engine | Tokens/sec | Notes |
|---|---|---|---|
| Gemma-4-26B-A4B-NVFP4 | FreeToken | ? | 262K context, loaded as default |
| gpt-oss-20b | FreeToken | 50+ | Amazingly fast for 20B on 12GB card |
Why gpt-oss-20b Hits 50 tok/s on a 3060
From the AI Bottleneck Thesis:
"Inference is memory-bound, not compute-bound. The ceiling is
BW / (P × b)"
The FreeToken Advantage: Heterogeneous MoE Offload
FreeToken uses Mixture of Experts (MoE) technology that doesn't just run on GPU. It creates a heterogeneous memory hierarchy:
| Tier | Hardware | Role |
|---|---|---|
| Tier 1 (fastest) | GPU VRAM (12GB) | Active expert weights, KV cache, hot path |
| Tier 2 | System RAM (~32GB) | Inactive expert weights, overflow |
| Tier 3 | CPU | Orchestration, expert routing, offload management |
This means:
- Not all 20B parameters need to be in VRAM simultaneously
- Only the active experts for a given token need GPU memory
- The rest sits in system RAM and gets paged in on demand
- Result: A 20B model runs on a 12GB card because at any given moment, only ~6-8B parameters are "hot"
The Math (Revised)
- RTX 3060: 12GB VRAM, ~360 GB/s bandwidth
- Active experts per token: ~6-8B parameters (estimated)
- Theoretical ceiling: 360 / (8B × 0.5 bytes) = ~90 tok/s
- Achieved: 50+ tok/s → overhead from CPU-GPU data transfer, but still excellent
- Without MoE offload: 20B × 0.5 bytes = 10GB → barely fits, no room for KV cache
Key Insight
The 3060's 12GB VRAM capacity is the reason this works. Smaller cards (3060 Ti with 8GB) couldn't fit the model. The memory bandwidth (360 GB/s) is the ceiling — and we're hitting above theoretical because FreeToken's CUDA kernels are well-optimized.
Constraints
- Cannot coexist with LFM2.5: Combined ~15GB > 12GB VRAM
- Context size limited: ~128K context at Q4 before running out of VRAM
- Model swap required: Need to stop one to start the other
Decision Signal
For local agent work requiring high throughput on a single question, gpt-oss-20b at 50 tok/s is a strong choice. For long-running daemon tasks, LFM2.5 on llama-cpp (port 1920) is more appropriate.