vault-sync: commit pending changes from Leonard session
This commit is contained in:
@@ -0,0 +1,71 @@
|
||||
# The Real Bottleneck Series — Tony Simons (Tomorrow Explained)
|
||||
|
||||
**Source:** [@tonysimons_](https://x.com/tonysimons_) — March 2026
|
||||
**Captured:** 2026-09-13
|
||||
**Topic:** AI infrastructure, inference economics, system architecture
|
||||
**Tags:** #ai #inference #memory-bandwidth #scaling-laws #agents #system-design
|
||||
|
||||
---
|
||||
|
||||
## Core Thesis (4-Part Series)
|
||||
|
||||
The AI industry keeps blaming the wrong bottleneck. The constraint has shifted — but the crowd hasn't noticed.
|
||||
|
||||
### Part 1: "The real bottleneck is not smarter models"
|
||||
- Models fail in production not because they lack intelligence, but because they lack **state management and structural error recovery**
|
||||
- Compounding probability: 95% reliability × 10 sequential steps = 60% final success rate
|
||||
- Hallucinations compound because auto-regressive models can't backtrack — they commit to every token
|
||||
- The fix isn't bigger models; it's **deterministic scaffolding** that catches failures before users see them
|
||||
- Future: systems that make thousands of mistakes per second and catch all of them
|
||||
|
||||
### Part 2: "The real bottleneck is not model size"
|
||||
- The era of the monolithic, know-it-all base model is over
|
||||
- Progress has migrated from **training-time compute → inference-time compute**
|
||||
- The compound system: models that think through hundreds of possibilities before showing a result
|
||||
- Routing layer matters more than raw model intelligence — cheap models for simple tasks, expensive ones only when needed
|
||||
- Memory layer (vector DBs, stateful architectures) is the real intelligence multiplier
|
||||
- The skill shift: from prompt engineering → **systems engineering**
|
||||
|
||||
### Part 3: "Memory bandwidth is the real bottleneck"
|
||||
- Inference is **memory-bound, not compute-bound** — most GPU cores sit idle waiting for data
|
||||
- The ceiling: `BW / (P × b)` — memory bandwidth divided by parameters × bytes per parameter
|
||||
- A 70B model at fp16 on 3 TB/s HBM ≈ 21 tokens/sec (matches real-world observation)
|
||||
- Batching raises throughput but not single-user speed
|
||||
- What actually helps: quantization, speculative decoding, sparse activation (MoE), shrinking KV cache
|
||||
- More FLOPs don't help if you can't feed the beast fast enough
|
||||
|
||||
### Part 4: "The real bottleneck is not training compute"
|
||||
- The industry is training-constrained in narrative, **inference-constrained in reality**
|
||||
- Base models are "good enough" for most tasks and commoditizing fast
|
||||
- The winners: those who can run multi-step reasoning loops at lowest cost/latency
|
||||
- Hardware shift: from training clusters → inference-optimized silicon (fast memory access > raw compute)
|
||||
- Market bifurcation: massive frontier models for complex/reasoning tasks, small optimized models for everything else
|
||||
- The real question: "Can you run this agent loop a million times a day without going bankrupt?"
|
||||
|
||||
---
|
||||
|
||||
## Key Insights for Builders
|
||||
|
||||
| Insight | Implication |
|
||||
|---------|-------------|
|
||||
| Reliability is a systems problem, not a model problem | Build scaffolding, not bigger prompts |
|
||||
| Inference cost dominates at target | Optimize for serving, not training |
|
||||
| Memory bandwidth is the hard ceiling | Quantize, batch, use MoE, shrink context |
|
||||
| Routing > raw intelligence | Use cheap models for 90% of tasks |
|
||||
| State management is the missing piece | External memory + error recovery loops |
|
||||
|
||||
---
|
||||
|
||||
## Connections to Our Work
|
||||
|
||||
- **Hermes Agent OS** — this is exactly what we're building: deterministic scaffolding around probabilistic engines
|
||||
- **Local LLMs on RTX 3060** — memory bandwidth is why 12GB is the ceiling; why LFM2.5 (2.6B) runs but larger models don't fit
|
||||
- **FreeToken vs llama.cpp coexistence** — VRAM is the bottleneck; can't run both on 12GB
|
||||
- **Vault search via search_files** — zero-infrastructure routing; no separate embedding server needed
|
||||
- **VeriPath audits** — narrow, constrained workflows beat general-purpose agents (aligned with the thesis)
|
||||
|
||||
---
|
||||
|
||||
## Quote
|
||||
|
||||
> "We spent two years obsessed with the engine, convinced that building a bigger motor was the only way to move forward. Now we realize that a massive engine is useless if you do not have a steering wheel, a transmission, and a map. The era of blind scaling is over. The grueling, unglamorous work of building the actual machine has finally begun."
|
||||
@@ -0,0 +1,109 @@
|
||||
# Hermes Agent Masterclass — Tonbi Studio
|
||||
|
||||
**Source:** [@tonbistudio tweet 2095877834409644064](https://x.com/tonbistudio/status/2095877834409644064)
|
||||
**Captured:** 2026-09-13
|
||||
**Topic:** Hermes configuration, workflow, optimization
|
||||
**Tags:** #hermes #masterclass #workflow #skills #agents #configuration
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Tonbi Studio's **Hermes Agent Masterclass** is a 10-part video course (5h 14m) taking Hermes from fresh install to hardened multi-agent team. It's the most comprehensive publicly available training for running Hermes in production.
|
||||
|
||||
- **hermesatlas.com/masterclass** — video + transcript notes
|
||||
- **hermesbible.com** — unofficial docs companion
|
||||
- **hermesatlas.com/dev/** — developer tutorial (10 modules)
|
||||
|
||||
---
|
||||
|
||||
## Module Map
|
||||
|
||||
| # | Module | Key Topics |
|
||||
|---|--------|------------|
|
||||
| 01 | Start Here | Installation, setup, basic commands |
|
||||
| 02 | Run It Anywhere | VPS, Docker, messaging |
|
||||
| 03 | Make Context Compound | Memory, Honcho, Obsidian |
|
||||
| 04 | Procedural Memory | Skills that grow with the agent |
|
||||
| 05 | Spend Intelligence Wisely | Providers, models, local inference |
|
||||
| 06 | Give the Model Hands | Tools & MCP servers |
|
||||
| 07 | Remove Yourself From the Loop | Cron & automation |
|
||||
| 08 | Parallelize the Right Work | Subagents & delegation |
|
||||
| 09 | Build a Durable Team | Profiles & Kanban |
|
||||
| 10 | Defense in Depth | Security layers |
|
||||
|
||||
---
|
||||
|
||||
## Key Insights (From Transcripts)
|
||||
|
||||
### The Agent Loop (Module 2)
|
||||
- **Think → call tool → see result → repeat → answer**
|
||||
- `finish_reason: "tool_calls"` → run tools, loop again
|
||||
- `finish_reason: "stop"` → answer, exit
|
||||
- `finish_reason: "length"` → compress history, retry
|
||||
- System prompt is built **once** per conversation — prompt caching is sacred
|
||||
- You extend Hermes at the **edges** (tools, skills), not by growing the core
|
||||
|
||||
### Memory Architecture (Module 3)
|
||||
- **Tiny built-in files** (memory.md, user.md) = tightly bounded prompt budget
|
||||
- Forces retention of durable preferences, not junk drawer
|
||||
- **Session search** only when needed — don't load everything
|
||||
- **Obsidian vault** for project-scale knowledge (this is our setup)
|
||||
- **Honcho** for richer recall (optional provider)
|
||||
|
||||
### Skills as Procedural Memory (Module 4)
|
||||
- When a complex tool sequence works → capture as a skill
|
||||
- Next run is faster and more consistent than rediscovering steps
|
||||
- **Agent-written skills** are the self-improvement loop
|
||||
- Curator grades and consolidates custom skills
|
||||
- Pin what must not drift
|
||||
|
||||
### Local Inference (Module 5)
|
||||
- Provider model picker: switch models in-session with `/model`
|
||||
- Compact long context with `/compress`
|
||||
- Background or queued prompts for non-blocking work
|
||||
|
||||
### MCP + Gateway + Cron (Module 6)
|
||||
- **MCP** = USB for AI tools — bolt on capabilities without writing Python
|
||||
- **Gateway** = one process, 20+ platforms (Telegram, Discord, Slack, etc.)
|
||||
- **Cron** = natural-language jobs, fresh agent per run
|
||||
- The three **compose**: cron job → gateway platform → MCP tools
|
||||
|
||||
### Security (Module 10)
|
||||
- **Seven security layers as dials**, not a checklist
|
||||
- Layer 1: who can talk to it (gateway allowlist)
|
||||
- Default: **DENY**
|
||||
- Tradeoff is structural: maximally useful ↔ maximally dangerous
|
||||
- Scope every dial per profile
|
||||
|
||||
---
|
||||
|
||||
## Relevant Commands
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `hermes` | Start chat |
|
||||
| `hermes gateway` | Start messaging |
|
||||
| `hermes doctor` | Check install |
|
||||
| `hermes model` | Reopen model config |
|
||||
| `/model` | Switch models in-session |
|
||||
| `/compress` | Compact long context |
|
||||
| `/tools` | See toolbox |
|
||||
| `/skills` | See procedural knowledge |
|
||||
| `/reload skills` | Pick up skill changes without restart |
|
||||
|
||||
---
|
||||
|
||||
## Connections to Our Setup
|
||||
|
||||
- **Module 3 (Memory)** — directly validates our vault architecture
|
||||
- **Module 4 (Skills)** — our skill system (hermes-agent, ponytail, etc.) is exactly this
|
||||
- **Module 5 (Local Inference)** — our FreeToken + llama-cpp setup is the local inference pattern
|
||||
- **Module 6 (MCP/Gateway/Cron)** — our vault-sync and vault-consolidation crons are this pattern
|
||||
- **Module 10 (Security)** — our standing constraints (Ty_Tech RO, no VPS SSH, explicit authorization) are security dials
|
||||
|
||||
---
|
||||
|
||||
## Quote
|
||||
|
||||
> "The core is a narrow waist; capability lives at the edges. Hermes is extended primarily through plugins and skills, not by growing the core."
|
||||
@@ -0,0 +1,55 @@
|
||||
# Inference Performance Benchmarks — Local Hardware
|
||||
|
||||
**Machine:** Windows 11, RTX 3060 12GB
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## FreeToken Benchmarks
|
||||
|
||||
| Model | Engine | Tokens/sec | Notes |
|
||||
|-------|--------|------------|-------|
|
||||
| Gemma-4-26B-A4B-NVFP4 | FreeToken | ? | 262K context, loaded as default |
|
||||
| gpt-oss-20b | FreeToken | **50+** | Amazingly fast for 20B on 12GB card |
|
||||
|
||||
## Why gpt-oss-20b Hits 50 tok/s on a 3060
|
||||
|
||||
From the [AI Bottleneck Thesis](ai-bottleneck-thesis-tony-simons.md):
|
||||
|
||||
> *"Inference is memory-bound, not compute-bound. The ceiling is `BW / (P × b)`"*
|
||||
|
||||
### The FreeToken Advantage: Heterogeneous MoE Offload
|
||||
|
||||
FreeToken uses **Mixture of Experts (MoE) technology** that doesn't just run on GPU. It creates a **heterogeneous memory hierarchy**:
|
||||
|
||||
| Tier | Hardware | Role |
|
||||
|------|----------|------|
|
||||
| **Tier 1 (fastest)** | GPU VRAM (12GB) | Active expert weights, KV cache, hot path |
|
||||
| **Tier 2** | System RAM (~32GB) | Inactive expert weights, overflow |
|
||||
| **Tier 3** | CPU | Orchestration, expert routing, offload management |
|
||||
|
||||
This means:
|
||||
- **Not all 20B parameters need to be in VRAM simultaneously**
|
||||
- Only the **active experts** for a given token need GPU memory
|
||||
- The rest sits in system RAM and gets paged in on demand
|
||||
- **Result:** A 20B model runs on a 12GB card because at any given moment, only ~6-8B parameters are "hot"
|
||||
|
||||
### The Math (Revised)
|
||||
|
||||
- **RTX 3060:** 12GB VRAM, ~360 GB/s bandwidth
|
||||
- **Active experts per token:** ~6-8B parameters (estimated)
|
||||
- **Theoretical ceiling:** 360 / (8B × 0.5 bytes) = ~90 tok/s
|
||||
- **Achieved:** 50+ tok/s → overhead from CPU-GPU data transfer, but still excellent
|
||||
- **Without MoE offload:** 20B × 0.5 bytes = 10GB → barely fits, no room for KV cache
|
||||
|
||||
## Key Insight
|
||||
|
||||
The 3060's **12GB VRAM capacity** is the reason this works. Smaller cards (3060 Ti with 8GB) couldn't fit the model. The memory bandwidth (360 GB/s) is the ceiling — and we're hitting above theoretical because FreeToken's CUDA kernels are well-optimized.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Cannot coexist with LFM2.5:** Combined ~15GB > 12GB VRAM
|
||||
- **Context size limited:** ~128K context at Q4 before running out of VRAM
|
||||
- **Model swap required:** Need to stop one to start the other
|
||||
|
||||
## Decision Signal
|
||||
|
||||
For local agent work requiring **high throughput on a single question**, gpt-oss-20b at 50 tok/s is a strong choice. For **long-running daemon tasks**, LFM2.5 on llama-cpp (port 1920) is more appropriate.
|
||||
Reference in New Issue
Block a user