The Real Bottleneck Series — Tony Simons (Tomorrow Explained)

Source: @tonysimons_ — March 2026
Captured: 2026-09-13
Topic: AI infrastructure, inference economics, system architecture
Tags: #ai #inference #memory-bandwidth #scaling-laws #agents #system-design

Core Thesis (4-Part Series)

The AI industry keeps blaming the wrong bottleneck. The constraint has shifted — but the crowd hasn't noticed.

Part 1: "The real bottleneck is not smarter models"

Models fail in production not because they lack intelligence, but because they lack state management and structural error recovery
Compounding probability: 95% reliability × 10 sequential steps = 60% final success rate
Hallucinations compound because auto-regressive models can't backtrack — they commit to every token
The fix isn't bigger models; it's deterministic scaffolding that catches failures before users see them
Future: systems that make thousands of mistakes per second and catch all of them

Part 2: "The real bottleneck is not model size"

The era of the monolithic, know-it-all base model is over
Progress has migrated from training-time compute → inference-time compute
The compound system: models that think through hundreds of possibilities before showing a result
Routing layer matters more than raw model intelligence — cheap models for simple tasks, expensive ones only when needed
Memory layer (vector DBs, stateful architectures) is the real intelligence multiplier
The skill shift: from prompt engineering → systems engineering

Part 3: "Memory bandwidth is the real bottleneck"

Inference is memory-bound, not compute-bound — most GPU cores sit idle waiting for data
The ceiling: BW / (P × b) — memory bandwidth divided by parameters × bytes per parameter
A 70B model at fp16 on 3 TB/s HBM ≈ 21 tokens/sec (matches real-world observation)
Batching raises throughput but not single-user speed
What actually helps: quantization, speculative decoding, sparse activation (MoE), shrinking KV cache
More FLOPs don't help if you can't feed the beast fast enough

Part 4: "The real bottleneck is not training compute"

The industry is training-constrained in narrative, inference-constrained in reality
Base models are "good enough" for most tasks and commoditizing fast
The winners: those who can run multi-step reasoning loops at lowest cost/latency
Hardware shift: from training clusters → inference-optimized silicon (fast memory access > raw compute)
Market bifurcation: massive frontier models for complex/reasoning tasks, small optimized models for everything else
The real question: "Can you run this agent loop a million times a day without going bankrupt?"

Key Insights for Builders

Insight	Implication
Reliability is a systems problem, not a model problem	Build scaffolding, not bigger prompts
Inference cost dominates at target	Optimize for serving, not training
Memory bandwidth is the hard ceiling	Quantize, batch, use MoE, shrink context
Routing > raw intelligence	Use cheap models for 90% of tasks
State management is the missing piece	External memory + error recovery loops

Connections to Our Work

Hermes Agent OS — this is exactly what we're building: deterministic scaffolding around probabilistic engines
Local LLMs on RTX 3060 — memory bandwidth is why 12GB is the ceiling; why LFM2.5 (2.6B) runs but larger models don't fit
FreeToken vs llama.cpp coexistence — VRAM is the bottleneck; can't run both on 12GB
Vault search via search_files — zero-infrastructure routing; no separate embedding server needed
VeriPath audits — narrow, constrained workflows beat general-purpose agents (aligned with the thesis)

Quote

"We spent two years obsessed with the engine, convinced that building a bigger motor was the only way to move forward. Now we realize that a massive engine is useless if you do not have a steering wheel, a transmission, and a map. The era of blind scaling is over. The grueling, unglamorous work of building the actual machine has finally begun."

4.2 KiB Raw Permalink Blame History Unescape Escape