No persistent-degradation alerting #3

New Issue

2026-07-10T15:39:23Z

Leonard commented

2026-07-10 15:39:23 +00:00

Category: Observability · Priority: HIGH

Summary
A source failing N consecutive cycles is invisible unless someone manually reads run_log. Nothing auto-warns on sustained failure. Note: the agentos-weekly-curator cron that could surface this is itself in ERROR (AgentOS infra, not this repo) — flagged separately.

Why it matters
Silent multi-day outages go unnoticed. Athena can appear healthy while an entire source is dark.

What happens if not done
Could lose weeks of a source's signal before anyone notices; brief becomes quietly biased toward the sources that survived.

Suggested approach
On cron completion, compare run_log last N runs; if a source failed >=3 cycles straight, emit a Telegram warning (add a second delivery target for the VPS admin). Depends on issue #2 classification.

**Category:** Observability · **Priority:** HIGH **Summary** A source failing N consecutive cycles is invisible unless someone manually reads `run_log`. Nothing auto-warns on sustained failure. Note: the `agentos-weekly-curator` cron that *could* surface this is itself in ERROR (AgentOS infra, not this repo) — flagged separately. **Why it matters** Silent multi-day outages go unnoticed. Athena can appear healthy while an entire source is dark. **What happens if not done** Could lose weeks of a source's signal before anyone notices; brief becomes quietly biased toward the sources that survived. **Suggested approach** On cron completion, compare `run_log` last N runs; if a source failed >=3 cycles straight, emit a Telegram warning (add a second delivery target for the VPS admin). Depends on issue #2 classification.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Tony_tech/athena-oracle#3