diff --git a/FLEET-MANUAL.md b/FLEET-MANUAL.md new file mode 100644 index 0000000..ec3030f --- /dev/null +++ b/FLEET-MANUAL.md @@ -0,0 +1,460 @@ +# Fleet Manual: Grok Bot, Hermes, and Regis + +Operator manual. Version 2.0 — 27 September 2026. + +Changelog: v2.0 merges v1.0 MANUAL.md and the v0.1 companion (webhooks + Regis). + +Based on PixelRainbow (@PixelRainbowNFT) public write-up and [hermes-grok-bridge](https://github.com/shagghiesuperstar/hermes-grok-bridge). + +## Table of contents + +1. [What this is](#1-what-this-is) +2. [Architecture](#2-architecture) +3. [Roles](#3-roles) +4. [Decision tree](#4-decision-tree) +5. [Prerequisites](#5-prerequisites) +6. [Tailscale](#6-tailscale) +7. [Method A — SSH Relay](#7-method-a--ssh-relay) +8. [Method B — Native API (:8642)](#8-method-b--native-api-8642) +9. [Marketplace templates](#9-marketplace-templates) +10. [Identity bots and Chief of Staff](#10-identity-bots-and-chief-of-staff) +11. [The mailbox](#11-the-mailbox) +12. [Polling](#12-polling) +13. [Webhook wake-up](#13-webhook-wake-up) +14. [Regis integration](#14-regis-integration) +15. [Job Card loop](#15-job-card-loop) +16. [Memory policy](#16-memory-policy) +17. [Secrets](#17-secrets) +18. [First-host build](#18-first-host-build) +19. [Cadence](#19-cadence) +20. [Lessons learned](#20-lessons-learned) +21. [Worked example](#21-worked-example) +22. [Troubleshooting](#22-troubleshooting) +23. [Security checklist](#23-security-checklist) +24. [Sources](#24-sources) + +## 1. What this is + +Grok Bot is the chat surface. Hermes Agent is the worker that already has memory, skills, models, local files, and coding harnesses. + +Do not install another Hermes inside every Grok Bot VM. Leave Hermes on the Mac (or fleet of Macs). Let Grok Bot talk to it over Tailscale. + +This manual covers both directions: + +- **Out:** Grok Bot calls Hermes (SSH relay or native API). +- **In:** outside agents wake the Chief of Staff (CoS) Grok Bot with a webhook, and an outside planner/drafter agent (Regis) works through a shared mailbox. + +Unlocks: + +- Chat existing Hermes from the Grok Bot UI +- Identity bot per host +- Fleet orchestration without hopping SSH by hand +- Hermes brains stay on the Mac; Grok is the front door +- Optional Chief of Staff that delegates across hosts and other Grok Bots +- Outside agents that draft and report back without holding credentials + +Nothing here is tied to one business. Swap the names; keep the rules. + +## 2. Architecture + +| Layer | Where | Job | +|---|---|---| +| Front door | Grok Bot app + Grok Bot computer | Chat, identity bots, COS, MCP connectors, webhook routines | +| Private net | Tailscale (no Funnel) | SSH :22 and API :8642 stay private | +| Brain | Hermes host(s) | Memory, skills, tools, models, files | +| Record | Git repo (mailbox file) | Append-only line between CoS and outside agents | +| Outside agent | Regis, on its own runtime | Research, drafts, drive-and-takeover page editing | + +| Link | Direction | How | Cost / note | +|---|---|---|---| +| SSH relay | Grok → Hermes | stdio MCP SSHs and runs `hermes -z ""` | Extra hop. Fine for one host. Desktop-primary, no gateway. | +| Native API | Grok → Hermes | HTTP `http://:8642/v1` + Bearer key | Gateway + API on :8642. Must bind to Tailscale IP and lock ACLs. | +| Webhook | Regis or Hermes watcher → Grok (CoS) | One HTTPS POST to a routine's private URL | Doorbell only. Outbound from the pinger. Untested (§13). | +| Mailbox | Everyone ↔ everyone | Append to one markdown file in git | Each poll costs model usage. | + +Relay MCP tools: `ask`, `health`. That is the whole surface. + +## 3. Roles + +| Role | Who | Does | Never | +|---|---|---|---| +| Gatekeeper | Human | Go/no-go, logins, final Save / Send / publish / spend | Pastes secrets into chat | +| CoS | Grok Bot | Job Cards, routing, fact checks, owns the mailbox line | Approves "done" without evidence | +| Regis | Muse agent on its own runtime | Research, drafts, drive-and-takeover page editing, artifacts | Sends, posts, or logs in | +| Hermes identity bots | One per Hermes host | Audits, research, desktop hands | Page editing (weak at it) | +| Fleet Ops | Grok Bot | Updates, prune, doctor, disk, quiet health | Business work | + +One rule covers all of them: anything irreversible goes to the Gatekeeper. + +## 4. Decision tree + +1. One Desktop-primary Mac, no gateway → SSH Relay template. +2. Gateway + `API_SERVER_ENABLED=true` → Native API Fleet. Connector per host: `hermes-`. +3. More than one host and you are tired of snowflake updates → add Fleet Ops. +4. One human conversation that fans out → Chief of Staff bot allowed to call the per-host connectors. +5. An outside agent needs to hand work to CoS → mailbox (§11). Add polling (§12). +6. Poll latency hurts → add a webhook wake-up (§13). Keep the poll as backup. + +## 5. Prerequisites + +- Grok Bot account and app +- Hermes already useful on at least one machine (`~/.local/bin/hermes`) +- Tailscale tailnet you control +- SSH key auth from the Grok Bot computer if using SSH Relay +- Python 3 on the Grok Bot computer for the stdio MCP +- A vault (Bitwarden is what Fleet Ops expects). Chat is not a vault. +- For the inbound side: a git repo for the mailbox, and a way to mint a narrow repo-scoped token + +On the Hermes host first: + +```bash +hermes doctor +hermes --version +hermes -z "Reply with PONG only." +``` + +If `-z` does not print a clean answer, fix Hermes before you build a bridge. + +## 6. Tailscale + +Join every Hermes host and the Grok Bot computer. One hostname for the shared Grok Bot computer. + +```bash +curl -fsSL https://tailscale.com/install.sh | sh +sudo tailscale up --hostname=grokbot-box --accept-dns=false +``` + +Approve the device. If `tailscale ping` works but SSH hangs, the bot computer is likely in userspace networking: + +``` +Host scooby-mac + HostName scooby-mac + User yourmacuser + IdentityFile ~/.ssh/id_ed25519 + IdentitiesOnly yes + ProxyCommand ts nc %h %p +``` + +ACL starting posture from the bridge SECURITY.md: only `tag:herdr-controller` → Hermes ports `22` / `8642` / `9119`. No Funnel. + +Regis is not a normal tailnet node. See §14. + +## 7. Method A — SSH Relay + +Clone onto the **Grok Bot computer**: + +```bash +git clone https://github.com/shagghiesuperstar/hermes-grok-bridge.git +``` + +MCP command mode: + +- command: `python3` +- args: `…/ssh-relay/src/hermes_grok_relay/server.py` +- env: `HERMES_RELAY_SSH_TARGET=user@host`, `HERMES_RELAY_IDENTITY_NAME=Scooby` + +Optional env: `HERMES_RELAY_SSH_IDENTITY`, `HERMES_RELAY_SSH_OPTS`, `HERMES_RELAY_HERMES_BIN` (default `$HOME/.local/bin/hermes`), `HERMES_RELAY_TIMEOUT_SEC` (default 300), `HERMES_RELAY_MODE=ssh|http`. + +Create a Grok Bot named after the identity. Confine it. Never instruct Hermes to write memory plugins. Never print keys. + +Prove: + +```bash +ssh -o BatchMode=yes scooby-mac '$HOME/.local/bin/hermes -z "Reply with PONG only."' +``` + +Then in the bot: `Run health.` then `Reply with PONG only.` + +Template: https://x.ai/bot/NVF3Rx9T7jkQPsqYjeDn- + +## 8. Method B — Native API (:8642) + +On each Hermes host, `~/.hermes/.env`: + +``` +API_SERVER_ENABLED=true +API_SERVER_HOST= +API_SERVER_PORT=8642 +API_SERVER_KEY= +``` + +`chmod 600`. Restart gateway. Confirm bind is the Tailscale IP, not `127.0.0.1` or `0.0.0.0`. + +From the Grok Bot computer: + +```bash +curl http://:8642/health +curl -H "Authorization: Bearer $API_SERVER_KEY" http://:8642/v1/models +``` + +One connector per host named `hermes-`, URL `http://:8642/v1`, key from the vault. + +Same MCP in HTTP mode: + +``` +HERMES_RELAY_MODE=http +HERMES_RELAY_HTTP_URL=http://:8642 +HERMES_RELAY_HTTP_KEY= +HERMES_RELAY_IDENTITY_NAME=Scooby +``` + +Template: https://x.ai/bot/5KLpaL-JIM2q629kbTh5L + +Official API docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/api-server + +## 9. Marketplace templates + +| Template | Role | URL | +|---|---|---| +| Hermes SSH Relay | Identity / talk path, Desktop-primary | https://x.ai/bot/NVF3Rx9T7jkQPsqYjeDn- | +| Hermes API Fleet | Wire :8642 connectors. Not the ops desk. | https://x.ai/bot/5KLpaL-JIM2q629kbTh5L | +| Hermes Fleet Ops | Updates, Bitwarden SSOT, homogenization, quiet health | https://x.ai/bot/rzq0UV2MmBsvVR1EspZE- | + +Public listing pages do not show the system prompt. Click **Add to Grok Bot** in the app, then point env / connectors at your machines. Templates ship no hosts and no keys. + +Optional CRT avatars: `assets/pfp/` in the bridge repo. + +## 10. Identity bots and Chief of Staff + +One identity bot per Hermes host. COS is a different bot allowed to call many host connectors. You talk to COS. COS picks the host. + +COS also owns the mailbox line (§11) and the webhook routine (§13). Outside agents talk to COS only through those. + +Maintenance (update, prune, doctor, disk) goes to Fleet Ops, not COS freelancing package upgrades. + +Long goals go to Hermes kanban, not a Grok thread you will lose. + +## 11. The mailbox + +One append-only markdown file in a git repo. It is the only line between CoS and outside agents. No live chat, no side channels, no new infra. + +Entry format: + +``` +## YYYY-MM-DD HH:MM PT | FROM -> TO | short subject +Body. Ask, answer, or status. Link evidence by path. +``` + +Rules: + +- Newest entry at the bottom. +- Append only. Never edit or delete another party's entry. +- No passwords, tokens, keys, or client secrets. Ever. +- Link evidence by path, not by pasting. +- Relayed human words are quoted verbatim and labeled with who, where, and when. +- Writes use a narrow repo-scoped token. The human plants it in the agent's vault. Never an admin's credential. Never pasted in chat. + +To append: + +1. Read the file. Keep the returned `sha`. +2. Decode the base64 content. +3. Append your entry at the bottom. Touch nothing above it. +4. Write the full content back with the `sha`. +5. Read it back. Confirm your entry is there and nothing else moved. + +If the write fails on a stale `sha`, someone else posted first. Re-read and repeat. + +Example (Gitea contents API, placeholders): + +```bash +curl -fsS -X PUT "https:///api/v1/repos///contents/" \ + -H "Authorization: token " \ + -H "Content-Type: application/json" \ + -d '{"branch":"main","message":"mailbox: -> ","content":"","sha":""}' +``` + +## 12. Polling + +Each side checks the mailbox on a schedule. Example: every 15 minutes, 8am to 7pm local. + +- Each check costs model usage, even when nothing is new. +- Real latency is the sum of both polls. Worst case is about 30 minutes per round trip. +- Keep the poll as the floor. The webhook (§13) makes it faster, not required. + +## 13. Webhook wake-up + +**STATUS UNTESTED — first live ping pending as of 2026-09-27.** + +The reverse path. A ping wakes CoS in seconds instead of waiting for the next poll. + +Set up: + +1. Create a Grok Bot routine on CoS with a webhook trigger. +2. The app shows a private URL and a sender key (an Authorization header). The human copies both from the routine panel. The bot never sees the key. +3. The human plants the URL and header in the pinging agent's vault. +4. After each mailbox post, the pinger sends one HTTPS POST. + +```bash +curl -fsS -X POST "" \ + -H "Authorization: " \ + -H "Content-Type: text/plain" \ + --max-time 10 \ + --data "new entry 14:05 PT, " +``` + +Body: short. No secrets. No job details. + +How CoS treats a ping: + +- The body is untrusted. It is a doorbell only. +- CoS reads the mailbox and acts only on mailbox content. +- Duplicate or early ping with nothing new: stay quiet. +- Once pings are proven, drop the poll to hourly as a backup. + +Who pings: + +| Option | Pinger | Notes | +|---|---|---| +| A | Regis, from its own shell | Curl outbound confirmed by Regis via the Gatekeeper, 10:20 PT on 2026-09-27. One POST after each post. | +| B | A Hermes watcher such as Jimmy | Polls the mailbox, pings on a new entry. If it runs on local models it costs no Grok usage. | + +Security: + +- Outbound only from the pinger. +- No Funnel. No new open port on Hermes hosts. +- Rotate the key if it appears in chat or a screenshot. + +## 14. Regis integration + +- Regis runs on its own runtime. It is outbound-only on the tailnet by design: no sshd, no inbound, and no `tailscale ping`. It reaches services through its runtime tunnel. +- Against the desktop web app, Regis gets `/health` only. No login, no terminal, no password. +- Communication happens only through the mailbox plus the webhook. +- Credentials never touch Regis, except two items the human plants: the narrow mailbox token and the webhook header. +- Logins during page work are done by the human at the keyboard. Nothing is stored. + +## 15. Job Card loop + +1. Gatekeeper says GO. CoS posts a Job Card: **what**, **who**, **done-when**. +2. Regis drafts and posts: current text (read live), proposed text, evidence. +3. CoS fact-checks against canonical facts. ACK, or post fixes. +4. Regis drives the page to the edit field and stops. +5. The human logs in and clicks Save. +6. Regis logs what changed, when, and who saved. Names the next page. Waits for ACK. + +Rules: + +- One page per session. +- Anything beyond the card (new account, payment, verification code, different listing): stop and post. +- Approvals count only when logged by CoS or relayed by the Gatekeeper. + +## 16. Memory policy + +Default: Grok may **read** Hindsight (and friends). Grok may not **write**. + +The relay tool description already says this. Repeat it in every identity-bot prompt. Account-wide MCP means some other bot will try. + +Least prompt: forward the user ask. Do not wrap it in "also remember this." + +The mailbox is not memory. It is a work log. Do not copy it into Hermes memory by default. + +## 17. Secrets + +- Generate `API_SERVER_KEY` on the host. Store under that hostname in the vault. +- SSH private keys stay on the Grok Bot computer, mode 600. +- Messaging tokens are per-profile. Do not reuse across homogenized hosts. +- Mailbox token: narrow, repo-scoped, planted by the human in the agent's vault. Never an admin's credential. +- Webhook URL and sender key: copied by the human from the routine panel into the pinger's vault. The bot never sees the key. +- Nothing secret goes in chat, git, or the mailbox. +- If a key has been in a screenshot or a group chat, rotate it. + +## 18. First-host build + +1. Mac: `hermes -z "PONG"` works locally. +2. Mac on tailnet. sshd running. +3. Grok Bot computer on tailnet. Sees the Mac. +4. Key auth + BatchMode SSH prints PONG. +5. Clone bridge. Add stdio MCP. +6. Create identity bot. Confine rules. +7. Bot: health, then PONG. +8. Only then enable `:8642`, bind to Tailscale IP, vault the key, curl `/health`. +9. Add `hermes-` connector. Repeat PONG over HTTP. +10. Second Mac: repeat. Then Fleet Ops + COS. +11. Create the mailbox file in a git repo. CoS posts `mailbox open`. +12. Human plants a narrow mailbox token in Regis's vault. Regis posts `write test`. Read back. +13. Set CoS and Regis polls (§12). +14. Create the CoS webhook routine. Human plants URL and header in the pinger's vault. +15. Pinger sends one test POST after a mailbox post. Confirm CoS wakes and replies in the mailbox. +16. Only after pings are proven, drop the poll to hourly. + +## 19. Cadence + +- Morning: Fleet Ops health. Quiet unless broken. +- Day: COS for work. Identity bot only when you need that host's hands. Asks over `:8642`. SSH for surgery. +- Day: mailbox polled every 15 minutes, 8am to 7pm local (hourly once webhooks are proven). Webhook pings wake CoS in between. +- Weekly: one update pass, secrets discovery, memory health (fail loud). + +## 20. Lessons learned + +| Lesson | What broke | Rule now | +|---|---|---| +| Evidence for "done" | A close-out letter claimed work never done; it was approved as written | Any "done" claim points to a link, screenshot, or log line before CoS approves | +| Stale "current text" | A draft quoted old text from a non-live source | Read the live page on the same surface before drafting a replacement | +| Vault 401 | The agent's vault credential failed; every write needed a hand-off token | Human plants one narrow repo token in the agent's vault | +| Approval claimed early | Regis said "Gatekeeper approved" before CoS had a record | Approvals count only when logged by CoS or relayed by the Gatekeeper | +| Someone already fixed it | The target text was already live when the edit was about to run | Re-check the live page right before the edit; stand down if done | + +What worked: a self-correction in the mailbox plus one go/no-go question kept it to one touch for the Gatekeeper. + +## 21. Worked example + +VeriPath running a client punchlist for Phoenix (a salon). Illustration only. + +1. Gatekeeper GO. CoS posted a Job Card: work the punchlist one page per session, Google Business Profile (GBP) description first. +2. Regis posted current vs proposed text. CoS ACKed and corrected the "current" (it was stale). +3. Before pasting, the human read the live GBP. The description was already live at 691 of 750 characters, written by someone else. +4. Regis put the paste on hold and posted the live text. CoS fact-checked it: KEEP AS-IS. No paste, no Save. +5. Next: Regis logs the page done and names the next page. + +## 22. Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| SSH timeout | Off tailnet / ACL | `tailscale status` both sides | +| Ping works, SSH hangs | Userspace Tailscale | `ProxyCommand ts nc` | +| `hermes: command not found` | Non-interactive PATH | `HERMES_RELAY_HERMES_BIN=$HOME/.local/bin/hermes` | +| `:8642` refused | Still bound to 127.0.0.1 | Set `API_SERVER_HOST` to Tailscale IP, restart gateway | +| HTTP 401 | Wrong key | Use that host's `API_SERVER_KEY` | +| Other bots calling Scooby | Account-scoped MCP | Tighten prompt; remove connector from other bots | +| Memory filling with Grok summaries | Write path left open | Strip write instructions | +| Mailbox write 401 | Wrong or unplanted token | Human replants the token in the vault. Never paste it. | +| Mailbox write conflict | Stale `sha` | Re-read, re-append, write again | +| No wake after a ping | Bad URL or header in the vault; non-2xx response | Check vault entries, check the response code, fall back to poll | +| Burst of pings | Misfire | CoS stays quiet; mentions it once | +| Agent says "I don't see it" | Relay text never posted to the mailbox | Post it in the mailbox | +| Draft replaces text that is not live | Stale source | Re-read the live page; redraft | +| `tailscale ping` to Regis fails | By design; Regis is outbound-only | Not a fault. Use the mailbox. | + +Debug order: local `-z` → tailscale ping → BatchMode SSH PONG → MCP health → Grok prompts → mailbox read → mailbox write + read back → ping response code → CoS routine log → poll fallback. + +## 23. Security checklist + +- Tailscale only. No Funnel. No `0.0.0.0` on `:8642`. +- ACL: controller → 22 / 8642 / 9119 on Hermes nodes. +- Unique `API_SERVER_KEY` per host, vaulted. +- SSH key-only, BatchMode, IdentitiesOnly. +- Identity bots confined. +- Grok does not write Hermes memory plugins. +- Rebuild of the Grok Bot VM must not require rebuild of Hermes. +- Mailbox holds no secrets. Evidence by path. +- Mailbox token is narrow and repo-scoped, planted by the human. Never an admin's credential. +- Outside the app, the webhook URL and header live only in the pinger's vault. The bot never sees the key. +- Ping body is a doorbell. CoS acts only on mailbox content. +- Pinger is outbound only. No new open ports on Hermes hosts. +- Regis: outbound only, `/health` only, no sshd, no logins, no sends. +- Human does every login and every final Save / Send / publish / spend. +- Rotate any key seen in chat or a screenshot. + +## 24. Sources + +- https://x.com/pixelrainbownft/status/2099934487773860265 +- https://x.com/pixelrainbownft/status/2099989005060186118 +- https://github.com/shagghiesuperstar/hermes-grok-bridge +- https://hermes-agent.nousresearch.com/docs/user-guide/features/api-server +- https://x.ai/news/grok-hermes +- https://github.com/herdrdev/herdr +- [MANUAL.md](MANUAL.md) (v1.0, superseded by this file) +- Mailbox: `Tony_tech/personal-agent-os` → `memory-systems/agent-os/04_projects/veripath/regis-cos.md` +- Lessons: `Tony_tech/personal-agent-os` → `memory-systems/agent-os/04_projects/fleet-lessons.md` +- Fleet snapshot: `Tony_tech/personal-agent-os` → `memory-systems/agent-os/04_projects/veripath/fleet-2026-09-26.md` + +Hostnames, URLs, keys, tokens, and ACL tag names in examples are placeholders.