vault-sync: 2026-10-02 07:10 (jimmy)
This commit is contained in:
jimmy
+109
@@ -0,0 +1,109 @@
|
||||
# 2026-09-19 — Post-mortem: Ashbot Gate 3 / inbox day (CoS + Hermes)
|
||||
|
||||
**Scope:** Phoenix Salon Gate 3 pilot → Birdeye send → inbox replies → STATUS watcher.
|
||||
**Outcome:** Birdeye email SENT; inbox pass found no returns; several process failures burned time and trust.
|
||||
**Author:** CoS (Chief of Staff) for Gatekeeper Anthony.
|
||||
|
||||
---
|
||||
|
||||
## 1. Timeline (PT, 2026-09-19)
|
||||
|
||||
| Time (approx) | Event |
|
||||
|---|---|
|
||||
| ~1:20 | Anthony: "Nothing" — expected Ashbot iMessage after Gate 3 assign |
|
||||
| ~1:21 | First Gate 3 dispatch **failed in ~5s** (Tailscale SOCKS AssertionError). CoS had implied start; folder never created |
|
||||
| ~1:32–1:36 | Diagnose → redispatch. Detached `start /B` / Start-Process **did not run**. SSH-sync `_launch.py` finally worked (pid 38648) |
|
||||
| ~1:36–1:52 | Gate 3 pilot **done**: FindGlocal WONT-DO (canonical); hairsalonsinusa Cloudflare WONT-DO; Birdeye draft only (no send_mail in that session). iMessage blocked (Photon free-tier / sidecar) |
|
||||
| ~2:23–2:28 | Birdeye **SENT** via SMTP from `ashbotphoenix@agentmail.to` (after username fix from rejected `ashbot@`) |
|
||||
| ~2:40–2:42 | Park Phoenix domain mailbox; lock AshBot Phoenix = AgentMail identity; patch config username |
|
||||
| ~2:58 | Close-out list for Anthony (IG / YP / Gate 3 leftovers) |
|
||||
| ~2:59 | Standing rule: CC `tonyjbala@gmail.com` on listing reply threads |
|
||||
| ~3:00 | Inbox pass dispatched — **oneshot missing AGENTMAIL_API_KEY** |
|
||||
| ~3:00–3:14 | Ashbot credential-hunt spiral (himalaya missing, safety blocks, 143s source grep). STATUS stuck `working` |
|
||||
| ~3:08+ | STATUS watcher created (@every 5m) after Anthony demanded visibility |
|
||||
| ~3:12–3:14 | Trace analysis; Anthony: intervene as superior |
|
||||
| ~3:15 | CoS killed pid 36384; redispatched **v2** with `.env` inject (`key_present=True`, pid 38988) |
|
||||
| ~3:26 | Mid-job ping: curling AgentMail inbox |
|
||||
| ~3:32 | **Inbox v2 done**: 4 msgs scanned, **NONE_FOUND** (no Birdeye/listing returns). ACTIVE_JOB cleared |
|
||||
| ~3:39–4:16 | Desktop often unreachable; watcher correctly stayed silent (idle) but kept waking CoS every 5m |
|
||||
|
||||
---
|
||||
|
||||
## 2. What "went off the rails"
|
||||
|
||||
### A. False starts / silent failures
|
||||
- Claimed or implied Ashbot "started" before SOCKS/SSH/folder/pid were real.
|
||||
- Windows detached launch looked dispatched with empty logs.
|
||||
- Anthony waited with "Nothing" / "It's been a long time" because CoS did not fail-loud or done-ping on time.
|
||||
|
||||
### B. Ashbot oneshot environment gap (root technical failure)
|
||||
- Method A `_launch.py` did **not** load `profiles/ashbot/.env`.
|
||||
- Gateway/Desktop sessions had AgentMail; SSH oneshot did not.
|
||||
- Birdeye send luckily had key in that session; inbox pass did not → multi-minute dead-end hunt.
|
||||
- Ashbot did not flip STATUS to `blocked` after first KEY_MISSING; CoS was slow to intervene.
|
||||
|
||||
### C. Visibility debt
|
||||
- No reliable mid-job channel until STATUS watcher.
|
||||
- iMessage contract failed (Photon free-tier + sidecar down) — chat ping is the real guarantee.
|
||||
- Watcher later correct for idle, but **@every 5m forever** creates noise when no job (and when desktop offline).
|
||||
|
||||
### D. Scope / attention scatter
|
||||
- Retro Revolution paste briefly looked like wrong-thread pollution.
|
||||
- Desktop Local dropped after inbox done → watcher "offline" ticks (silent to Anthony, still wake tax).
|
||||
|
||||
### E. What actually worked
|
||||
- File-brief + SSH-sync launch (once SOCKS up).
|
||||
- Birdeye outbound SMTP with `ashbotphoenix@agentmail.to`.
|
||||
- Intervene rule applied: kill spiral → inject key → v2 completed cleanly.
|
||||
- Lessons filed in Obsidian; several standing rules locked in CoS memory.
|
||||
|
||||
---
|
||||
|
||||
## 3. Root causes (ordered)
|
||||
|
||||
1. **CoS process immature:** announce before prove; late intervene; late done-ping.
|
||||
2. **Launcher incomplete:** oneshot must always inject AgentMail (and profile) from ashbot `.env`.
|
||||
3. **Hermes worker defaults:** will source-dive instead of STATUS=blocked + stop.
|
||||
4. **Notify path fragile:** Photon not reliable; chat must be primary.
|
||||
5. **Watcher cadence too aggressive when idle.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Locked rules (do not regress)
|
||||
|
||||
1. Never say Hermes "started" until folder + STATUS pid + launch log LAUNCH line.
|
||||
2. SSH-sync `_launch.py` + `_cos_brief.txt`; no `start /B` until proven.
|
||||
3. Inject `AGENTMAIL_API_KEY` from ashbot `.env` before every mail oneshot.
|
||||
4. Key missing once → STATUS blocked + stop; CoS fixes; no source archaeology.
|
||||
5. Dead-end path → CoS intervenes immediately (superior).
|
||||
6. Done/blocked → CoS pings Anthony in chat immediately.
|
||||
7. Mid-job ~5m STATUS pings only while ACTIVE_JOB points at live work.
|
||||
8. Ashbot From = `ashbotphoenix@agentmail.to`; CC Anthony on listing reply threads.
|
||||
9. Phoenix domain mailbox parked; do not block on `assistant@phoenixcameronpark.com`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Corrective actions (next)
|
||||
|
||||
| Action | Owner | Status |
|
||||
|---|---|---|
|
||||
| Standardize hermes-relay / `_launch.py` with .env inject | CoS | Partially done (v2 pattern) — promote to default template |
|
||||
| Pause STATUS watcher when ACTIVE_JOB empty; resume on dispatch | CoS | Do now |
|
||||
| Optional: daily/idle inbox poll for Birdeye replies (with key inject) | CoS | Not started |
|
||||
| Anthony: IG + YP confirm (close letter blockers) | Gatekeeper | Open |
|
||||
| Mirror lessons into Gitea `docs/agents/` | CoS | Pending |
|
||||
|
||||
---
|
||||
|
||||
## 6. Honest scorecard
|
||||
|
||||
| Goal | Result |
|
||||
|---|---|
|
||||
| Gate 3 3-URL pilot | Done (2 WONT-DO + Birdeye draft→sent) |
|
||||
| Birdeye email out | **Success** |
|
||||
| Inbox replies + CC Anthony | N/A — no returns yet (pass succeeded) |
|
||||
| iMessage start/done | Failed (infrastructure) |
|
||||
| Gatekeeper experience | **Poor mid-day** — silence, false starts, late intervene |
|
||||
| Learning loop | **Strong** — traces → lessons → rules → intervene |
|
||||
|
||||
**Bottom line:** The company work mostly completed; the *operating system* for CoS↔Hermes nearly failed Anthony’s trust. Paying attention to traces was correct. Default path forward is thin CoS, injected env, intervene early, chat as source of truth.
|
||||
Reference in New Issue
Block a user