56 lines
2.4 KiB
Markdown
56 lines
2.4 KiB
Markdown
# Inference Performance Benchmarks — Local Hardware
|
||
|
||
**Machine:** Windows 11, RTX 3060 12GB
|
||
**Date:** 2026-09-13
|
||
|
||
## FreeToken Benchmarks
|
||
|
||
| Model | Engine | Tokens/sec | Notes |
|
||
|-------|--------|------------|-------|
|
||
| Gemma-4-26B-A4B-NVFP4 | FreeToken | ? | 262K context, loaded as default |
|
||
| gpt-oss-20b | FreeToken | **50+** | Amazingly fast for 20B on 12GB card |
|
||
|
||
## Why gpt-oss-20b Hits 50 tok/s on a 3060
|
||
|
||
From the [AI Bottleneck Thesis](ai-bottleneck-thesis-tony-simons.md):
|
||
|
||
> *"Inference is memory-bound, not compute-bound. The ceiling is `BW / (P × b)`"*
|
||
|
||
### The FreeToken Advantage: Heterogeneous MoE Offload
|
||
|
||
FreeToken uses **Mixture of Experts (MoE) technology** that doesn't just run on GPU. It creates a **heterogeneous memory hierarchy**:
|
||
|
||
| Tier | Hardware | Role |
|
||
|------|----------|------|
|
||
| **Tier 1 (fastest)** | GPU VRAM (12GB) | Active expert weights, KV cache, hot path |
|
||
| **Tier 2** | System RAM (~32GB) | Inactive expert weights, overflow |
|
||
| **Tier 3** | CPU | Orchestration, expert routing, offload management |
|
||
|
||
This means:
|
||
- **Not all 20B parameters need to be in VRAM simultaneously**
|
||
- Only the **active experts** for a given token need GPU memory
|
||
- The rest sits in system RAM and gets paged in on demand
|
||
- **Result:** A 20B model runs on a 12GB card because at any given moment, only ~6-8B parameters are "hot"
|
||
|
||
### The Math (Revised)
|
||
|
||
- **RTX 3060:** 12GB VRAM, ~360 GB/s bandwidth
|
||
- **Active experts per token:** ~6-8B parameters (estimated)
|
||
- **Theoretical ceiling:** 360 / (8B × 0.5 bytes) = ~90 tok/s
|
||
- **Achieved:** 50+ tok/s → overhead from CPU-GPU data transfer, but still excellent
|
||
- **Without MoE offload:** 20B × 0.5 bytes = 10GB → barely fits, no room for KV cache
|
||
|
||
## Key Insight
|
||
|
||
The 3060's **12GB VRAM capacity** is the reason this works. Smaller cards (3060 Ti with 8GB) couldn't fit the model. The memory bandwidth (360 GB/s) is the ceiling — and we're hitting above theoretical because FreeToken's CUDA kernels are well-optimized.
|
||
|
||
## Constraints
|
||
|
||
- **Cannot coexist with LFM2.5:** Combined ~15GB > 12GB VRAM
|
||
- **Context size limited:** ~128K context at Q4 before running out of VRAM
|
||
- **Model swap required:** Need to stop one to start the other
|
||
|
||
## Decision Signal
|
||
|
||
For local agent work requiring **high throughput on a single question**, gpt-oss-20b at 50 tok/s is a strong choice. For **long-running daemon tasks**, LFM2.5 on llama-cpp (port 1920) is more appropriate.
|