72 lines
4.2 KiB
Markdown
72 lines
4.2 KiB
Markdown
# The Real Bottleneck Series — Tony Simons (Tomorrow Explained)
|
||
|
||
**Source:** [@tonysimons_](https://x.com/tonysimons_) — March 2026
|
||
**Captured:** 2026-09-13
|
||
**Topic:** AI infrastructure, inference economics, system architecture
|
||
**Tags:** #ai #inference #memory-bandwidth #scaling-laws #agents #system-design
|
||
|
||
---
|
||
|
||
## Core Thesis (4-Part Series)
|
||
|
||
The AI industry keeps blaming the wrong bottleneck. The constraint has shifted — but the crowd hasn't noticed.
|
||
|
||
### Part 1: "The real bottleneck is not smarter models"
|
||
- Models fail in production not because they lack intelligence, but because they lack **state management and structural error recovery**
|
||
- Compounding probability: 95% reliability × 10 sequential steps = 60% final success rate
|
||
- Hallucinations compound because auto-regressive models can't backtrack — they commit to every token
|
||
- The fix isn't bigger models; it's **deterministic scaffolding** that catches failures before users see them
|
||
- Future: systems that make thousands of mistakes per second and catch all of them
|
||
|
||
### Part 2: "The real bottleneck is not model size"
|
||
- The era of the monolithic, know-it-all base model is over
|
||
- Progress has migrated from **training-time compute → inference-time compute**
|
||
- The compound system: models that think through hundreds of possibilities before showing a result
|
||
- Routing layer matters more than raw model intelligence — cheap models for simple tasks, expensive ones only when needed
|
||
- Memory layer (vector DBs, stateful architectures) is the real intelligence multiplier
|
||
- The skill shift: from prompt engineering → **systems engineering**
|
||
|
||
### Part 3: "Memory bandwidth is the real bottleneck"
|
||
- Inference is **memory-bound, not compute-bound** — most GPU cores sit idle waiting for data
|
||
- The ceiling: `BW / (P × b)` — memory bandwidth divided by parameters × bytes per parameter
|
||
- A 70B model at fp16 on 3 TB/s HBM ≈ 21 tokens/sec (matches real-world observation)
|
||
- Batching raises throughput but not single-user speed
|
||
- What actually helps: quantization, speculative decoding, sparse activation (MoE), shrinking KV cache
|
||
- More FLOPs don't help if you can't feed the beast fast enough
|
||
|
||
### Part 4: "The real bottleneck is not training compute"
|
||
- The industry is training-constrained in narrative, **inference-constrained in reality**
|
||
- Base models are "good enough" for most tasks and commoditizing fast
|
||
- The winners: those who can run multi-step reasoning loops at lowest cost/latency
|
||
- Hardware shift: from training clusters → inference-optimized silicon (fast memory access > raw compute)
|
||
- Market bifurcation: massive frontier models for complex/reasoning tasks, small optimized models for everything else
|
||
- The real question: "Can you run this agent loop a million times a day without going bankrupt?"
|
||
|
||
---
|
||
|
||
## Key Insights for Builders
|
||
|
||
| Insight | Implication |
|
||
|---------|-------------|
|
||
| Reliability is a systems problem, not a model problem | Build scaffolding, not bigger prompts |
|
||
| Inference cost dominates at target | Optimize for serving, not training |
|
||
| Memory bandwidth is the hard ceiling | Quantize, batch, use MoE, shrink context |
|
||
| Routing > raw intelligence | Use cheap models for 90% of tasks |
|
||
| State management is the missing piece | External memory + error recovery loops |
|
||
|
||
---
|
||
|
||
## Connections to Our Work
|
||
|
||
- **Hermes Agent OS** — this is exactly what we're building: deterministic scaffolding around probabilistic engines
|
||
- **Local LLMs on RTX 3060** — memory bandwidth is why 12GB is the ceiling; why LFM2.5 (2.6B) runs but larger models don't fit
|
||
- **FreeToken vs llama.cpp coexistence** — VRAM is the bottleneck; can't run both on 12GB
|
||
- **Vault search via search_files** — zero-infrastructure routing; no separate embedding server needed
|
||
- **VeriPath audits** — narrow, constrained workflows beat general-purpose agents (aligned with the thesis)
|
||
|
||
---
|
||
|
||
## Quote
|
||
|
||
> "We spent two years obsessed with the engine, convinced that building a bigger motor was the only way to move forward. Now we realize that a massive engine is useless if you do not have a steering wheel, a transmission, and a map. The era of blind scaling is over. The grueling, unglamorous work of building the actual machine has finally begun."
|