2026-09-19 — Post-mortem: Ashbot Gate 3 / inbox day (CoS + Hermes)

Scope: Phoenix Salon Gate 3 pilot → Birdeye send → inbox replies → STATUS watcher.
Outcome: Birdeye email SENT; inbox pass found no returns; several process failures burned time and trust.
Author: CoS (Chief of Staff) for Gatekeeper Anthony.

1. Timeline (PT, 2026-09-19)

Time (approx)	Event
~1:20	Anthony: "Nothing" — expected Ashbot iMessage after Gate 3 assign
~1:21	First Gate 3 dispatch failed in ~5s (Tailscale SOCKS AssertionError). CoS had implied start; folder never created
~1:32–1:36	Diagnose → redispatch. Detached `start /B` / Start-Process did not run. SSH-sync `_launch.py` finally worked (pid 38648)
~1:36–1:52	Gate 3 pilot done: FindGlocal WONT-DO (canonical); hairsalonsinusa Cloudflare WONT-DO; Birdeye draft only (no send_mail in that session). iMessage blocked (Photon free-tier / sidecar)
~2:23–2:28	Birdeye SENT via SMTP from `ashbotphoenix@agentmail.to` (after username fix from rejected `ashbot@`)
~2:40–2:42	Park Phoenix domain mailbox; lock AshBot Phoenix = AgentMail identity; patch config username
~2:58	Close-out list for Anthony (IG / YP / Gate 3 leftovers)
~2:59	Standing rule: CC `tonyjbala@gmail.com` on listing reply threads
~3:00	Inbox pass dispatched — oneshot missing AGENTMAIL_API_KEY
~3:00–3:14	Ashbot credential-hunt spiral (himalaya missing, safety blocks, 143s source grep). STATUS stuck `working`
~3:08+	STATUS watcher created (@every 5m) after Anthony demanded visibility
~3:12–3:14	Trace analysis; Anthony: intervene as superior
~3:15	CoS killed pid 36384; redispatched v2 with `.env` inject (`key_present=True`, pid 38988)
~3:26	Mid-job ping: curling AgentMail inbox
~3:32	Inbox v2 done: 4 msgs scanned, NONE_FOUND (no Birdeye/listing returns). ACTIVE_JOB cleared
~3:39–4:16	Desktop often unreachable; watcher correctly stayed silent (idle) but kept waking CoS every 5m

2. What "went off the rails"

A. False starts / silent failures

Claimed or implied Ashbot "started" before SOCKS/SSH/folder/pid were real.
Windows detached launch looked dispatched with empty logs.
Anthony waited with "Nothing" / "It's been a long time" because CoS did not fail-loud or done-ping on time.

B. Ashbot oneshot environment gap (root technical failure)

Method A _launch.py did not load profiles/ashbot/.env.
Gateway/Desktop sessions had AgentMail; SSH oneshot did not.
Birdeye send luckily had key in that session; inbox pass did not → multi-minute dead-end hunt.
Ashbot did not flip STATUS to blocked after first KEY_MISSING; CoS was slow to intervene.

C. Visibility debt

No reliable mid-job channel until STATUS watcher.
iMessage contract failed (Photon free-tier + sidecar down) — chat ping is the real guarantee.
Watcher later correct for idle, but @every 5m forever creates noise when no job (and when desktop offline).

D. Scope / attention scatter

Retro Revolution paste briefly looked like wrong-thread pollution.
Desktop Local dropped after inbox done → watcher "offline" ticks (silent to Anthony, still wake tax).

E. What actually worked

File-brief + SSH-sync launch (once SOCKS up).
Birdeye outbound SMTP with ashbotphoenix@agentmail.to.
Intervene rule applied: kill spiral → inject key → v2 completed cleanly.
Lessons filed in Obsidian; several standing rules locked in CoS memory.

3. Root causes (ordered)

CoS process immature: announce before prove; late intervene; late done-ping.
Launcher incomplete: oneshot must always inject AgentMail (and profile) from ashbot .env.
Hermes worker defaults: will source-dive instead of STATUS=blocked + stop.
Notify path fragile: Photon not reliable; chat must be primary.
Watcher cadence too aggressive when idle.

4. Locked rules (do not regress)

Never say Hermes "started" until folder + STATUS pid + launch log LAUNCH line.
SSH-sync _launch.py + _cos_brief.txt; no start /B until proven.
Inject AGENTMAIL_API_KEY from ashbot .env before every mail oneshot.
Key missing once → STATUS blocked + stop; CoS fixes; no source archaeology.
Dead-end path → CoS intervenes immediately (superior).
Done/blocked → CoS pings Anthony in chat immediately.
Mid-job ~5m STATUS pings only while ACTIVE_JOB points at live work.
Ashbot From = ashbotphoenix@agentmail.to; CC Anthony on listing reply threads.
Phoenix domain mailbox parked; do not block on assistant@phoenixcameronpark.com.

5. Corrective actions (next)

Action	Owner	Status
Standardize hermes-relay / `_launch.py` with .env inject	CoS	Partially done (v2 pattern) — promote to default template
Pause STATUS watcher when ACTIVE_JOB empty; resume on dispatch	CoS	Do now
Optional: daily/idle inbox poll for Birdeye replies (with key inject)	CoS	Not started
Anthony: IG + YP confirm (close letter blockers)	Gatekeeper	Open
Mirror lessons into Gitea `docs/agents/`	CoS	Pending

6. Honest scorecard

Goal	Result
Gate 3 3-URL pilot	Done (2 WONT-DO + Birdeye draft→sent)
Birdeye email out	Success
Inbox replies + CC Anthony	N/A — no returns yet (pass succeeded)
iMessage start/done	Failed (infrastructure)
Gatekeeper experience	Poor mid-day — silence, false starts, late intervene
Learning loop	Strong — traces → lessons → rules → intervene

Bottom line: The company work mostly completed; the operating system for CoS↔Hermes nearly failed Anthony’s trust. Paying attention to traces was correct. Default path forward is thin CoS, injected env, intervene early, chat as source of truth.

5.8 KiB Raw Blame History Unescape Escape