# The Real Bottleneck Series — Tony Simons (Tomorrow Explained)

**Source:** [@tonysimons_](https://x.com/tonysimons_) — March 2026  
**Captured:** 2026-09-13  
**Topic:** AI infrastructure, inference economics, system architecture  
**Tags:** #ai #inference #memory-bandwidth #scaling-laws #agents #system-design

---

## Core Thesis (4-Part Series)

The AI industry keeps blaming the wrong bottleneck. The constraint has shifted — but the crowd hasn't noticed.

### Part 1: "The real bottleneck is not smarter models"
- Models fail in production not because they lack intelligence, but because they lack **state management and structural error recovery**
- Compounding probability: 95% reliability × 10 sequential steps = 60% final success rate
- Hallucinations compound because auto-regressive models can't backtrack — they commit to every token
- The fix isn't bigger models; it's **deterministic scaffolding** that catches failures before users see them
- Future: systems that make thousands of mistakes per second and catch all of them

### Part 2: "The real bottleneck is not model size"
- The era of the monolithic, know-it-all base model is over
- Progress has migrated from **training-time compute → inference-time compute**
- The compound system: models that think through hundreds of possibilities before showing a result
- Routing layer matters more than raw model intelligence — cheap models for simple tasks, expensive ones only when needed
- Memory layer (vector DBs, stateful architectures) is the real intelligence multiplier
- The skill shift: from prompt engineering → **systems engineering**

### Part 3: "Memory bandwidth is the real bottleneck"
- Inference is **memory-bound, not compute-bound** — most GPU cores sit idle waiting for data
- The ceiling: `BW / (P × b)` — memory bandwidth divided by parameters × bytes per parameter
- A 70B model at fp16 on 3 TB/s HBM ≈ 21 tokens/sec (matches real-world observation)
- Batching raises throughput but not single-user speed
- What actually helps: quantization, speculative decoding, sparse activation (MoE), shrinking KV cache
- More FLOPs don't help if you can't feed the beast fast enough

### Part 4: "The real bottleneck is not training compute"
- The industry is training-constrained in narrative, **inference-constrained in reality**
- Base models are "good enough" for most tasks and commoditizing fast
- The winners: those who can run multi-step reasoning loops at lowest cost/latency
- Hardware shift: from training clusters → inference-optimized silicon (fast memory access > raw compute)
- Market bifurcation: massive frontier models for complex/reasoning tasks, small optimized models for everything else
- The real question: "Can you run this agent loop a million times a day without going bankrupt?"

---

## Key Insights for Builders

| Insight | Implication |
|---------|-------------|
| Reliability is a systems problem, not a model problem | Build scaffolding, not bigger prompts |
| Inference cost dominates at target | Optimize for serving, not training |
| Memory bandwidth is the hard ceiling | Quantize, batch, use MoE, shrink context |
| Routing > raw intelligence | Use cheap models for 90% of tasks |
| State management is the missing piece | External memory + error recovery loops |

---

## Connections to Our Work

- **Hermes Agent OS** — this is exactly what we're building: deterministic scaffolding around probabilistic engines
- **Local LLMs on RTX 3060** — memory bandwidth is why 12GB is the ceiling; why LFM2.5 (2.6B) runs but larger models don't fit
- **FreeToken vs llama.cpp coexistence** — VRAM is the bottleneck; can't run both on 12GB
- **Vault search via search_files** — zero-infrastructure routing; no separate embedding server needed
- **VeriPath audits** — narrow, constrained workflows beat general-purpose agents (aligned with the thesis)

---

## Quote

> "We spent two years obsessed with the engine, convinced that building a bigger motor was the only way to move forward. Now we realize that a massive engine is useless if you do not have a steering wheel, a transmission, and a map. The era of blind scaling is over. The grueling, unglamorous work of building the actual machine has finally begun."