# Inference Performance Benchmarks — Local Hardware

**Machine:** Windows 11, RTX 3060 12GB  
**Date:** 2026-09-13

## FreeToken Benchmarks

| Model | Engine | Tokens/sec | Notes |
|-------|--------|------------|-------|
| Gemma-4-26B-A4B-NVFP4 | FreeToken | ? | 262K context, loaded as default |
| gpt-oss-20b | FreeToken | **50+** | Amazingly fast for 20B on 12GB card |

## Why gpt-oss-20b Hits 50 tok/s on a 3060

From the [AI Bottleneck Thesis](ai-bottleneck-thesis-tony-simons.md):

> *"Inference is memory-bound, not compute-bound. The ceiling is `BW / (P × b)`"*

### The FreeToken Advantage: Heterogeneous MoE Offload

FreeToken uses **Mixture of Experts (MoE) technology** that doesn't just run on GPU. It creates a **heterogeneous memory hierarchy**:

| Tier | Hardware | Role |
|------|----------|------|
| **Tier 1 (fastest)** | GPU VRAM (12GB) | Active expert weights, KV cache, hot path |
| **Tier 2** | System RAM (~32GB) | Inactive expert weights, overflow |
| **Tier 3** | CPU | Orchestration, expert routing, offload management |

This means:
- **Not all 20B parameters need to be in VRAM simultaneously**
- Only the **active experts** for a given token need GPU memory
- The rest sits in system RAM and gets paged in on demand
- **Result:** A 20B model runs on a 12GB card because at any given moment, only ~6-8B parameters are "hot"

### The Math (Revised)

- **RTX 3060:** 12GB VRAM, ~360 GB/s bandwidth
- **Active experts per token:** ~6-8B parameters (estimated)
- **Theoretical ceiling:** 360 / (8B × 0.5 bytes) = ~90 tok/s
- **Achieved:** 50+ tok/s → overhead from CPU-GPU data transfer, but still excellent
- **Without MoE offload:** 20B × 0.5 bytes = 10GB → barely fits, no room for KV cache

## Key Insight

The 3060's **12GB VRAM capacity** is the reason this works. Smaller cards (3060 Ti with 8GB) couldn't fit the model. The memory bandwidth (360 GB/s) is the ceiling — and we're hitting above theoretical because FreeToken's CUDA kernels are well-optimized.

## Constraints

- **Cannot coexist with LFM2.5:** Combined ~15GB > 12GB VRAM
- **Context size limited:** ~128K context at Q4 before running out of VRAM
- **Model swap required:** Need to stop one to start the other

## Decision Signal

For local agent work requiring **high throughput on a single question**, gpt-oss-20b at 50 tok/s is a strong choice. For **long-running daemon tasks**, LFM2.5 on llama-cpp (port 1920) is more appropriate.