# Athena-Oracle: Development Design Document

**Version:** 0.1.0
**Date:** 2026-07-08
**Status:** Draft — pre-implementation
**Source branch:** `MVP-milestone`

---

## 1. North Star

> *Surfacing cross-source convergence and using falsification to distinguish real momentum from noise.*

Athena is an autonomous research intelligence engine that ingests from multiple fragmented sources, detects when the same signals appear across independent channels, and uses decay-based falsification to separate genuine trends from one-day spikes. The system is model-agnostic, lightweight, and designed to run unattended on a resource-constrained VPS.

### How this design supports the north star

| North Star Principle | Design Decision | Why |
|---|---|---|
| Cross-source convergence | Keyword co-occurrence matrix (Phase 1), embeddings + vector search (Phase 2) | Keyword co-occurrence is the simplest convergence detector: if the same entity appears in ≥3 independent sources within a time window, it's converging. Embeddings Phase 2 adds semantic convergence for signals that use different words but mean the same thing. |
| Falsification over confirmation | Exponential decay scoring | A 7-day hard cutoff is blunt: some trends die in 48 hours, some take 60 days to validate. `score = base_score * e^(-λ * days_since_last_signal)` naturally scores dying trends low and sustained trends high without arbitrary day thresholds. |
| Autonomous operation | Cron/scheduled pipeline + graceful degradation | The pipeline runs daily without human intervention. If Ollama is down, ingestion continues and summarization defers to the next run. If one adapter fails, the rest still run. |
| Lightweight deployment | Python + SQLite + Flask + host-level Ollama | No PostgreSQL, no Elasticsearch, no Redis, no Kubernetes. A single Python process, a single SQLite file, and an external Ollama REST API. |

---

## 2. System Architecture

```
┌─────────────────────────────────────────────────────┐
│  Athena-Oracle Pipeline (Python, ~150-500MB)        │
│                                                     │
│  ┌────────────┐ ┌─────────┐ ┌──────────────────┐   │
│  │ GitHub     │ │ arXiv   │ │ Reddit           │   │
│  │ Adapter    │ │ Adapter │ │ Adapter          │   │
│  └──────┬─────┘ └────┬────┘ └────────┬─────────┘   │
│         │             │              │              │
│  ┌──────┴─────────────┴──────────────┴──────────┐  │
│  │  Deduplication (URL hash + title simhash)    │  │
│  └────────────────────┬─────────────────────────┘  │
│                       │                            │
│  ┌────────────────────┴─────────────────────────┐  │
│  │  Theme Tagging                                │  │
│  │  Phase 1: Keyword co-occurrence + fixed seeds│  │
│  │  Phase 2: all-MiniLM embeddings + BERTopic   │  │
│  └────────────────────┬─────────────────────────┘  │
│                       │                            │
│  ┌────────────────────┴─────────────────────────┐  │
│  │  Falsification Engine                        │  │
│  │  Exponential decay scoring per theme         │  │
│  │  Convergence threshold: ≥3 independent sources│ │
│  └────────────────────┬─────────────────────────┘  │
│                       │                            │
│  ┌────────────────────┴─────────────────────────┐  │
│  │  SQLite                                      │  │
│  │  entries table + run_log + FTS5 index        │  │
│  │  Phase 2: + sqlite-vec extension             │  │
│  └──────────────────────────────────────────────┘  │
│                                                     │
│  Output layer:                                      │
│  ┌──────────┐ ┌──────────────┐ ┌────────────────┐  │
│  │ Flask API│ │ File drops   │ │ MCP server     │  │
│  │ (Phase 4)│ │ (Phase 4)    │ │ (Phase 6)      │  │
│  └──────────┘ └──────────────┘ └────────────────┘  │
│                                                     │
│  Structured JSON logging → stdout + file rotation  │
│  Discord/Slack webhook on 2+ day adapter failure   │
└─────────────────────────────────────────────────────┘
                                                    │
                                                    │
                    ┌───────────────────────────────┘
                    │ HTTP REST API
                    ▼
          ┌─────────────────────┐
          │ Ollama (host-level) │
          │ llama3.2:1b         │
          │ (summarization)     │
          └─────────────────────┘
```

### Memory budget

| Component | Phase 1 | Phase 2 |
|---|---|---|
| Python runtime + deps | ~80 MB | ~80 MB |
| SQLite (in-process) | ~10 MB | ~10 MB |
| all-MiniLM embeddings | — | ~80 MB |
| sqlite-vec extension | — | ~2 MB |
| Flask | ~1 MB | ~1 MB |
| **Pipeline total** | **~90 MB** | **~173 MB** |
| Ollama + model (host-level) | ~2 GB | ~2 GB |
| **System total** | **~2.1 GB** | **~2.2 GB** |

The 150MB constraint applies to the pipeline process. The full system footprint including Ollama is ~2GB.

---

## 3. Data Layer

### 3.1 SQLite schema

Core tables (from `schema.sql`):

- **entries** — one row per ingested item, deduplicated by `(source, source_id)`
- **run_log** — one row per pipeline run, tracks per-source success/failure
- **FTS5 virtual table** — full-text search over `title` and `extracted_text`
- **Phase 2: sqlite-vec** — vector index for semantic similarity queries

### 3.2 Why SQLite

- Zero external dependency, single file, survives container restarts
- FTS5 is built-in (no separate search engine)
- Handles 100K-1M rows without performance issues
- sqlite-vec extension adds vector search without a separate database
- No connection pooling needed (single-writer pipeline)
- Postgres/pgvector is premature optimization at this scale

### 3.3 Data retention

- Raw entries: 90 days
- Summaries and convergence scores: 365 days
- Periodic `VACUUM` to reclaim space
- `archive.py` handles cold storage rotation (deferred to Phase 3)

---

## 4. Adapter Layer

### 4.1 Source adapters (HTTP-only)

| Adapter | API | Rate limit | Auth required |
|---|---|---|---|
| GitHub | REST API | 60/hr (unauth), 5000/hr (token) | `GITHUB_TOKEN` |
| arXiv | REST API | 1 req/sec (polite) | No |
| Reddit | RSS/JSON | ~100 req/min | No (but OAuth recommended) |
| Hacker News | Firebase API | Unofficial, ~30 req/sec | No |
| HuggingFace | REST API | Throttled if aggressive | `HUGGINGFACE_TOKEN` |
| RSS Feeds | RSS XML | Varies | No |

**Decision:** HTTP-only adapters, no Playwright/Selenium. All 6 sources have programmatic APIs. Playwright would add Chromium's 300MB+ overhead and fragility.

### 4.2 Deduplication

arXiv papers appear on HN, Reddit, and Twitter. Without deduplication, the same signal is counted 3× and produces false convergence.

- **Phase 1:** UNIQUE constraint on `(source, source_id)` + URL hash dedup across sources
- **Phase 2:** SimHash/MinHash content fingerprinting for near-duplicate detection

### 4.3 Rate limiting and retries

- Per-adapter rate limits enforced in the adapter class
- `tenacity` library for exponential backoff on transient failures (429, 503, timeout)
- One failing adapter does not kill the pipeline

---

## 5. Theme Tagging and Convergence Detection

### 5.1 Phase 1: Keyword co-occurrence

Pre-defined keyword dictionaries per theme. An entry is tagged if ≥2 keywords from a theme dictionary appear in its title or extracted text. A theme "converges" if it appears in ≥3 independent sources within the last 24 hours.

**Why keyword first at 150MB:** Keyword matching is zero-dependency, explainable, and works within the memory constraint. FTS5 provides fast retrieval.

**Fixed themes:** The initial 4 themes (`tool-call`, `context`, `compute`, `trust`) are seeds, not a hard limit. An "other" catch-all bucket captures signals that don't match predefined themes.

### 5.2 Phase 2: Embeddings + auto-discovery

- **all-MiniLM-L6-v2** (22M params, ~80MB) for sentence embeddings
- **sqlite-vec** for in-database ANN search
- **BERTopic** (or equivalent) for semi-supervised theme discovery, seeded from the Phase 1 dictionary
- Hybrid query: FTS5 for precision (keyword match) + vector for recall (semantic match), merged via Reciprocal Rank Fusion

**Why not keyword forever:** Keyword matching cannot detect semantic convergence (different words, same concept) and requires constant manual dictionary updates. Embeddings are the eventual target; Phase 1 is the bridge.

### 5.3 Convergence scoring

```
convergence_score = Σ(source_weights) × temporal_proximity × theme_entropy

where:
  source_weights: arXiv=2.0, GitHub=1.5, HN=1.0, Reddit=0.8, HF=1.2, RSS=0.5
  temporal_proximity: e^(-0.1 * hours_since_first_signal)
  theme_entropy: log2(number_of_independent_sources)
```

Thresholds:
- `≥ 3.0` → "confirmed" trend
- `≥ 1.5` → "emerging" signal
- `< 1.5` → "noise"

---

## 6. Falsification Engine

### 6.1 Exponential decay scoring

Replace the 7-day dead thesis rule with:

```
thesis_score = initial_score × e^(-λ × days_since_last_signal)

where λ = 0.1 (configurable)
```

A thesis is "dead" when its score falls below a configurable threshold (default: 0.1), not when it hits a fixed day count. This naturally handles:
- Fast-dying trends (score drops quickly)
- Slow-burn trends (score stays elevated)
- Revived trends (new signal resets the decay clock)

### 6.2 Cross-source validation

A signal is flagged "unverified" if:
- Only 1 source has primary (non-derivative) coverage
- The signal appears only in echo chambers (e.g., HN upvotes ≠ real adoption)
- A counter-narrative exists in the same time window

---

## 7. Output Layer

### 7.1 Consumer interfaces (progressive rollout)

| Consumer | Interface | Phase |
|---|---|---|
| **Bob** (trend tracker) | Flask REST API: `GET /trends?theme=&period=7d` | 4 |
| **Alice** (content creator) | Daily file drops: `/output/YYYY-MM-DD/trends.yaml` | 4 |
| **Sam** (Hermes agent) | MCP server: `oracle_search`, `oracle_trends`, `oracle_verdicts` | 6 |

### 7.2 REST API (Phase 4)

Flask endpoints:
- `GET /health` — pipeline status, last run time, adapter health
- `GET /trends` — active themes with convergence scores
- `GET /entries` — search entries (keyword + phase 2: semantic)
- `GET /verdicts` — confirmed/dead theses
- `GET /convergence` — cross-source convergence matrix

### 7.3 File drops (Phase 4)

Daily structured output at a known path:
```
/output/YYYY-MM-DD/
  trends.yaml       # Human-readable daily digest
  signals.json      # Structured machine-readable output
  verdicts.json     # Confirmed/dead thesis list
```

### 7.4 MCP server (Phase 6)

MCP tools for Hermes agent integration:
- `oracle_search(query, source, date_range)` — search entries
- `oracle_trends(theme, convergence_threshold)` — get active trends
- `oracle_verdicts(status)` — confirmed or dead theses
- `oracle_latest(source)` — most recent entry per source

---

## 8. Observability and Reliability

### 8.1 Logging

Structured JSON logging via stdlib `logging` with JSON formatter. Per-pipeline-stage logs (ingest, dedup, theme, falsification, summarize) with source-level granularity.

### 8.2 Health endpoint

`GET /health` returns:
```json
{
  "status": "ok",
  "last_run": "2026-07-08T13:00:00Z",
  "last_run_duration_sec": 245,
  "entries_since_last_run": 127,
  "adapters": {
    "github": {"status": "ok", "fetched": 20},
    "arxiv": {"status": "ok", "fetched": 15},
    "reddit": {"status": "error", "fetched": 0, "error": "429 rate limited"}
  }
}
```

### 8.3 Alerting

Discord/Slack webhook triggered when:
- An adapter fails for 2+ consecutive days
- Pipeline run exceeds 2× expected duration
- SQLite database integrity check fails

### 8.4 Graceful degradation

If Ollama is unreachable:
- Ingestion continues normally
- Summarization is skipped, entries stored with `summary = null`
- Deferral: next run summarizes pending entries
- Alert: "summarization deferred, N entries pending"

---

## 9. Scheduling

### 9.1 Phase 1: Cron

- `oracle-pipeline.sh` invoked by cron at 13:00 UTC daily
- `flock`/PID file prevents overlapping runs
- Exit codes: 0 = success, 1 = partial failure, 2 = total failure

### 9.2 Phase 2: systemd timers

- `Persistent=true` catches up on missed runs
- `RandomizedDelaySec` prevents thundering herd
- `OnFailureSec` for retry logic
- Better logging than cron (`journalctl -u athena-timer`)

### 9.3 Why not APScheduler (Phase 1)

APScheduler adds in-process async daemon overhead. Cron/systemd is OS-level, zero process memory cost, and sufficient for daily runs. APScheduler is the Phase 2 target if dynamic scheduling (user-configurable refresh rates, per-source intervals) is needed.

---

## 10. Inference

### 10.1 Summarization

**Model:** Ollama `llama3.2:1b` (or `qwen2.5:0.5b` for lower resource)
**Deployment:** Host-level Ollama service, pipeline calls via HTTP REST API
**Contract:** Model-agnostic — `summarize(text) → (summary, model)` interface
**Graceful degradation:** If Ollama is down, store raw text and defer summarization

### 10.2 Why not in-container Ollama

Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB container is physically impossible. Running it host-level means the pipeline process stays within budget and Ollama can share resources with other services.

---

## 11. Security

| Requirement | Implementation |
|---|---|
| No hardcoded secrets | `GITHUB_TOKEN`, `HUGGINGFACE_TOKEN` as env vars or mounted secret files |
| TLS for outbound | All HTTP adapters use `https://` |
| Least privilege | Pipeline runs as standard user (no sudo) |
| DB protection | `chmod 600 oracle.db` |
| Input sanitization | Parameterized SQL queries, no string concatenation |

---

## 12. Implementation Phases

| Phase | Scope | Deliverable |
|---|---|---|
| **P0: Foundation** | schema.sql + sqlite-vec design, adapter registry, oracle-pipeline.sh skeleton | Empty but valid pipeline |
| **P1: First data** | arXiv + RSS adapters, SQLite storage, keyword convergence, dedup | Live data flowing |
| **P2: Full ingest** | GitHub, HN, HF, Reddit adapters, rate limiting, structured logging | All 6 sources live |
| **P3: Falsification** | Exponential decay scoring, Ollama summarization, graceful degradation | Trend verdicts working |
| **P4: Consumption** | Flask API, daily file drops, health endpoint, alerting webhooks | Bob and Alice can consume |
| **P5: Validation** | 7-day UAT window, Hermes cron integration, exit codes | System runs unattended |
| **P6: Scale** | sqlite-vec + embeddings, MCP server, APScheduler, BERTopic themes | Research-grade system |

---

## 13. Backport to PRD: Requirements to Add

The following requirements are implied by this design and should be added to `docs/MVP-PRD.md`:

### 13.1 Platform requirements (REQ-PLT-XX)

| ID | Requirement |
|---|---|
| REQ-PLT-05 | All processes run as standard user (no sudo) — *already exists* |
| REQ-PLT-10 | The pipeline process shall not exceed 500MB of RSS memory (excluding host-level Ollama) |
| REQ-PLT-15 | The system shall support deployment on a VPS with 2GB total RAM (pipeline + Ollama + OS) |
| REQ-PLT-20 | Ollama inference shall run as a host-level service, not inside the pipeline container |
| REQ-PLT-25 | The pipeline shall use SQLite as the sole database (no PostgreSQL, no Elasticsearch, no Redis) |

### 13.2 Reliability requirements (REQ-REL-XX)

| ID | Requirement |
|---|---|
| REQ-REL-05 | The system operates autonomously without human interaction — *already exists* |
| REQ-REL-10 | Failed source fetches retry with exponential backoff — *already exists* |
| REQ-REL-15 | Previously stored data is not lost on restart or failure — *already exists* |
| REQ-REL-20 | The pipeline completes successfully even if 1+ sources are unavailable — *already exists* |
| REQ-REL-25 | If Ollama is unreachable, ingestion continues and summarization defers to the next run |
| REQ-REL-30 | The pipeline uses flock/PID file to prevent overlapping runs |

### 13.3 Observability requirements (REQ-DIAG-XX)

| ID | Requirement |
|---|---|
| REQ-DIAG-05 | Structured JSON logs with timestamps and severity — *already exists* |
| REQ-DIAG-10 | Health endpoint reports system status and last successful run — *already exists* |
| REQ-DIAG-15 | Per-source success/failure and fetch counts logged per run — *already exists* |
| REQ-DIAG-20 | Webhook alert (Discord/Slack) fires when an adapter fails for 2+ consecutive days |
| REQ-DIAG-25 | run_log table captures per-run metrics queryable via SQL |

### 13.4 Integration requirements (REQ-INT-XX)

| ID | Requirement |
|---|---|
| REQ-INT-05 | Structured, machine-readable output (JSON) consumable by external tools — *already exists* |
| REQ-INT-10 | Adapter layer supports adding new sources without modifying core pipeline logic — *already exists* |
| REQ-INT-15 | REST API exposes GET /trends, /entries, /verdicts endpoints |
| REQ-INT-20 | Daily file drop at configurable path with structured output (YAML + JSON) |
| REQ-INT-25 | MCP server exposes oracle_search, oracle_trends, oracle_verdicts tools (Phase 6) |

### 13.5 Data requirements (REQ-DATA-XX) *(new section)*

| ID | Requirement |
|---|---|
| REQ-DATA-05 | Entries are deduplicated by (source, source_id) with cross-source URL hash deduplication |
| REQ-DATA-10 | FTS5 full-text index on title and extracted_text for keyword search |
| REQ-DATA-15 | Convergence detection: theme appears in ≥3 independent sources within 24h window |
| REQ-DATA-20 | Falsification uses exponential decay scoring (configurable λ), not fixed day thresholds |
| REQ-DATA-25 | Data retention: raw entries 90 days, summaries/convergence 365 days, periodic VACUUM |

### 13.6 Security requirements (REQ-SEC-XX)

| ID | Requirement |
|---|---|
| REQ-SEC-05 | All outbound HTTP uses TLS — *already exists* |
| REQ-SEC-10 | No secrets hardcoded or stored in plaintext — *already exists* |
| REQ-SEC-15 | Least-privilege access for outbound API calls — *already exists* |
| REQ-SEC-20 | SQLite database file permissions set to 600 (owner-only read/write) |
| REQ-SEC-25 | All SQL queries use parameterized statements (no string concatenation) |

### 13.7 Scheduling requirements (REQ-SCH-XX) *(new section)*

| ID | Requirement |
|---|---|
| REQ-SCH-05 | Pipeline entry point is oracle-pipeline.sh (idempotent, single command) |
| REQ-SCH-10 | Default schedule: 13:00 UTC daily |
| REQ-SCH-15 | Exit codes: 0 = success, 1 = partial failure, 2 = total failure |
| REQ-SCH-20 | Overlapping run prevention via flock or PID file check |

---

## 14. Design Decisions Summary (Why)

| Decision | Why |
|---|---|
| **SQLite over PostgreSQL** | Zero external dependency, single file, FTS5 built-in, handles 1M rows fine. pgvector is premature at this scale. |
| **Ollama host-level** | 150MB container cannot fit Ollama + model (~2GB). Host-level lets pipeline stay within budget. |
| **Flask over FastAPI** | ~1MB vs ~100MB runtime overhead. FastAPI is Phase 2 target; Flask suffices for internal REST API. |
| **Keyword co-occurrence (Phase 1)** | Zero-dependency, explainable, works at 150MB. Embeddings (Phase 2) add semantic convergence. |
| **Fixed themes + catch-all** | BERTopic requires 4GB RAM. Fixed themes with "other" bucket is the pragmatic constraint choice. |
| **Cron over APScheduler (Phase 1)** | OS-level, zero process memory cost. APScheduler is Phase 2 for dynamic scheduling. |
| **HTTP-only adapters** | All 6 sources have programmatic APIs. Playwright adds 300MB+ overhead and fragility. |
| **Exponential decay over 7-day rule** | One-line formula, no fixed threshold. Handles fast-dying and slow-burn trends naturally. |
| **Deduplication required** | arXiv papers appear on HN/Reddit/Twitter. Without dedup, same signal counted 3× = false convergence. |
| **Graceful degradation on Ollama** | Ingestion must not depend on summarization. Store raw data, defer summaries. |

---

*Document prepared via multi-model analysis: Qwythos-9B (architectural critique), Qwen3.5-9B (implementation evaluation), and cross-review synthesis. 4 delegations, 2 rounds of debate. Raw reviews saved in the same directory.*