athena-oracle/docs/Dev-Design.md

# Athena-Oracle: Development Design Document

**Version:** 0.1.0
**Date:** 2026-07-08
**Status:** Draft — pre-implementation
**Source branch:** `MVP-milestone`

---

## 1. North Star

> *Surfacing cross-source convergence and using falsification to distinguish real momentum from noise.*

Athena is an autonomous research intelligence engine that ingests from multiple fragmented sources, detects when the same signals appear across independent channels, and uses decay-based falsification to separate genuine trends from one-day spikes. The system is model-agnostic, lightweight, and designed to run unattended on a resource-constrained VPS.

### How this design supports the north star

| North Star Principle | Design Decision | Why |
|---|---|---|
| Cross-source convergence | Keyword co-occurrence matrix (Phase 1), embeddings + vector search (Phase 2) | Keyword co-occurrence is the simplest convergence detector: if the same entity appears in ≥3 independent sources within a time window, it's converging. Embeddings Phase 2 adds semantic convergence for signals that use different words but mean the same thing. |
| Falsification over confirmation | Exponential decay scoring | A 7-day hard cutoff is blunt: some trends die in 48 hours, some take 60 days to validate. `score = base_score * e^(-λ * days_since_last_signal)` naturally scores dying trends low and sustained trends high without arbitrary day thresholds. |
| Autonomous operation | Cron/scheduled pipeline + graceful degradation | The pipeline runs daily without human intervention. If Ollama is down, ingestion continues and summarization defers to the next run. If one adapter fails, the rest still run. |
| Lightweight deployment | Python + SQLite + Flask + host-level Ollama | No PostgreSQL, no Elasticsearch, no Redis, no Kubernetes. A single Python process, a single SQLite file, and an external Ollama REST API. |

### MVP Success Criteria

The MVP is considered successful when:

1. **Research Time**: Bob answers 'what's trending this week' in <15 min of research time total (across all sources).

2. **Signal Detection**: Over 7 days, the theme scan correctly identifies ≥1 real trend that has genuine cross-source convergence.

3. **Noise Rejection**: Over the same 7-day period, the falsification engine kills ≥1 false signal (a one-day spike with no sustained arrivals) to demonstrate decay-based filtering is working.

---

## 2. System Architecture

```
┌─────────────────────────────────────────────────────┐
│  Athena-Oracle Pipeline (Python, ~150-500MB)        │
│                                                     │
│  ┌────────────┐ ┌─────────┐ ┌──────────────────┐   │
│  │ GitHub     │ │ arXiv   │ │ Reddit           │   │
│  │ Adapter    │ │ Adapter │ │ Adapter          │   │
│  └──────┬─────┘ └────┬────┘ └────────┬─────────┘   │
│         │             │              │              │
│  ┌──────┴─────────────┴──────────────┴──────────┐  │
│  │  Deduplication (URL hash + title simhash)    │  │
│  └────────────────────┬─────────────────────────┘  │
│                       │                            │
│  ┌────────────────────┴─────────────────────────┐  │
│  │  Theme Tagging                                │  │
│  │  Phase 1: Keyword co-occurrence + fixed seeds│  │
│  │  Phase 2: all-MiniLM embeddings + BERTopic   │  │
│  └────────────────────┬─────────────────────────┘  │
│                       │                            │
│  ┌────────────────────┴─────────────────────────┐  │
│  │  Falsification Engine                        │  │
│  │  Exponential decay scoring per theme         │  │
│  │  Convergence threshold: ≥3 independent sources│ │
│  └────────────────────┬─────────────────────────┘  │
│                       │                            │
│  ┌────────────────────┴─────────────────────────┐  │
│  │  SQLite                                      │  │
│  │  entries table + run_log + FTS5 index        │  │
│  │  Phase 2: + sqlite-vec extension             │  │
│  └──────────────────────────────────────────────┘  │
│                                                     │
│  Output layer:                                      │
│  ┌──────────┐ ┌──────────────┐ ┌────────────────┐  │
│  │ Flask API│ │ File drops   │ │ MCP server     │  │
│  │ (Phase 4)│ │ (Phase 4)    │ │ (Phase 6)      │  │
│  └──────────┘ └──────────────┘ └────────────────┘  │
│                                                     │
│  Structured JSON logging → stdout + file rotation  │
│  Discord/Slack webhook on 2+ day adapter failure   │
└─────────────────────────────────────────────────────┘
                                                    │
                                                    │
                    ┌───────────────────────────────┘
                    │ HTTP REST API
                    ▼
          ┌─────────────────────┐
          │ Ollama (host-level) │
          │ llama3.2:1b         │
          │ (summarization)     │
          └─────────────────────┘
```

### Memory budget

| Component | Phase 1 | Phase 2 |
|---|---|---|
| Python runtime + deps | ~80 MB | ~80 MB |
| SQLite (in-process) | ~10 MB | ~10 MB |
| all-MiniLM embeddings | — | ~80 MB |
| sqlite-vec extension | — | ~2 MB |
| Flask | ~1 MB | ~1 MB |
| **Pipeline total** | **~90 MB** | **~173 MB** |
| Ollama + model (host-level) | ~2 GB | ~2 GB |
| **System total** | **~2.1 GB** | **~2.2 GB** |

The 150MB constraint applies to the pipeline process. The full system footprint including Ollama is ~2GB.

### Daily Run Flow

1. **Preflight** — Check disk space, verify DB integrity (`PRAGMA integrity_check`), load adapter config
2. **Ingest** — Run all 6 adapters in parallel, collect raw items per source
3. **Dedup** — Hash URLs, match against existing entries, insert only new items
4. **Theme Tag** — Run keyword co-occurrence against new items, tag themes
5. **Falsification** — Recompute decay scores for all active themes, kill dead theses
6. **Archive** — Move entries older than 90 days to archive table
7. **Report** — Write daily summary to `/output/`, update run_log

### Failure Modes

| Component | Failure | Impact | Recovery |
|---|---|---|---|
| Adapter | Rate limit / 503 | Missing items from that source | Retry next cycle; pipeline continues |
| Ollama | Down | Summaries skipped | Entries stored with `summary = null`, deferred to next run |
| SQLite | Disk full | No writes | Alert via webhook; manual cleanup |
| SQLite | Corruption | Data loss | Restore from last backup |
| Network | Outbound blocked | All adapters fail | Alert; pipeline exits code 2 |

---

## 3. Data Layer

### 3.1 SQLite schema

Core tables (from `schema.sql`):

- **entries** — one row per ingested item, deduplicated by `(source, source_id)`
- **run_log** — one row per pipeline run, tracks per-source success/failure
- **FTS5 virtual table** — full-text search over `title` and `extracted_text`
- **Phase 2: sqlite-vec** — vector index for semantic similarity queries

### 3.2 Why SQLite

- Zero external dependency, single file, survives container restarts
- FTS5 is built-in (no separate search engine)
- Handles 100K-1M rows without performance issues
- sqlite-vec extension adds vector search without a separate database
- No connection pooling needed (single-writer pipeline)
- Postgres/pgvector is premature optimization at this scale

### 3.3 Data retention

- Raw entries: 90 days
- Summaries and convergence scores: 365 days
- Periodic `VACUUM` to reclaim space
- `archive.py` handles cold storage rotation (deferred to Phase 3)

### 3.4 Backup Strategy

- **Pre-schema-change backup**: Before any `ALTER TABLE` or schema modification:
  ```bash
  sqlite3 oracle.db '.backup oracle.db.bak'
  ```
  Store `.bak` files with date suffix in `/backup/` (`oracle.db.bak-YYYYMMDD`).

- **Daily compressed backup**: At 01:00 UTC (off-peak):
  ```bash
  tar -czf /backup/oracle.db.$(date +%Y%m%d).tar.gz oracle.db
  ```
  Retain 30 days of backups; purge older: `find /backup -name '*.tar.gz' -mtime +30 -delete`

- **Recovery**: Restore from backup with `cp /backup/oracle.db.bak-YYYYMMDD oracle.db`, verify with `PRAGMA integrity_check`

### 3.5 Schema Migration

- Versioned migration files in `migrations/` directory (e.g., `001_initial_schema.sql`, `002_add_theme_tags.sql`)
- Applied at startup: pipeline checks `schema_version` table, runs any unapplied migrations in order
- Each migration is a single atomic SQL file; no partial migrations
- Rollback: each migration includes a comment with the reverse SQL if needed

---

## 4. Adapter Layer

### 4.1 Source adapters (HTTP-only)

| Adapter | API | Rate limit | Auth required |
|---|---|---|---|
| GitHub | REST API | 60/hr (unauth), 5000/hr (token) | `GITHUB_TOKEN` |
| arXiv | REST API | 1 req/sec (polite) | No |
| Reddit | RSS/JSON | ~100 req/min | No (but OAuth recommended) |
| Hacker News | Firebase API | Unofficial, ~30 req/sec | No |
| HuggingFace | REST API | Throttled if aggressive | `HUGGINGFACE_TOKEN` |
| RSS Feeds | RSS XML | Varies | No |

**Decision:** HTTP-only adapters, no Playwright/Selenium. All 6 sources have programmatic APIs. Playwright would add Chromium's 300MB+ overhead and fragility.

### 4.2 Deduplication

arXiv papers appear on HN, Reddit, and Twitter. Without deduplication, the same signal is counted 3× and produces false convergence.

- **Phase 1:** UNIQUE constraint on `(source, source_id)` + URL hash dedup across sources
- **Phase 2:** SimHash/MinHash content fingerprinting for near-duplicate detection

### 4.3 Rate limiting and retries

- Per-adapter rate limits enforced in the adapter class
- `tenacity` library for exponential backoff on transient failures (429, 503, timeout)
- One failing adapter does not kill the pipeline

### 4.4 Adapter Interface Contract

All adapters must implement this minimal interface:

- **name() → str**: Unique identifier for the source (e.g., `"arxiv"`)
- **fetch(query, limit) → list[dict]**: Returns normalized items with required schema fields:
  - `source` (str): Source name matching `name()`
  - `source_id` (str): Unique per-source ID for deduplication
  - `title` (str): Human-readable title
  - `url` (str): Direct URL to the entry
  - `timestamp` (datetime): Publication/update time
  - `raw_score` (float): Source-specific signal strength (0.0–10.0)
  - `body` (str): Raw text content for summarization and theme tagging

**Custom Exceptions:**

- `RateLimitError`: Raised when source returns 429 or similar; includes `retry_after` (seconds)
- `SourceUnavailableError`: Raised when source is down (5xx) or unreachable

Adapters must not raise other exceptions on normal operation; unexpected errors should be logged with full traceback and surfaced via the run_log, not propagated to the pipeline.

---

## 5. Theme Tagging and Convergence Detection

### 5.1 Phase 1: Keyword co-occurrence

Pre-defined keyword dictionaries per theme. An entry is tagged if ≥2 keywords from a theme dictionary appear in its title or extracted text. A theme "converges" if it appears in ≥3 independent sources within the last 24 hours.

**Why keyword first at 150MB:** Keyword matching is zero-dependency, explainable, and works within the memory constraint. FTS5 provides fast retrieval.

**Fixed themes:** The initial 4 themes (`tool-call`, `context`, `compute`, `trust`) are seeds, not a hard limit. An "other" catch-all bucket captures signals that don't match predefined themes.

### 5.2 Phase 2: Embeddings + auto-discovery

- **all-MiniLM-L6-v2** (22M params, ~80MB) for sentence embeddings
- **sqlite-vec** for in-database ANN search
- **BERTopic** (or equivalent) for semi-supervised theme discovery, seeded from the Phase 1 dictionary
- Hybrid query: FTS5 for precision (keyword match) + vector for recall (semantic match), merged via Reciprocal Rank Fusion

**Why not keyword forever:** Keyword matching cannot detect semantic convergence (different words, same concept) and requires constant manual dictionary updates. Embeddings are the eventual target; Phase 1 is the bridge.

### 5.3 Convergence scoring

```
convergence_score = Σ(source_weights) × temporal_proximity × theme_entropy

where:
  source_weights: arXiv=2.0, GitHub=1.5, HN=1.0, Reddit=0.8, HF=1.2, RSS=0.5
  temporal_proximity: e^(-0.1 * hours_since_first_signal)
  theme_entropy: log2(number_of_independent_sources)
```

Thresholds:
- `≥ 3.0` → "confirmed" trend
- `≥ 1.5` → "emerging" signal
- `< 1.5` → "noise"

### 5.4 Theme Governance

Themes are defined in a single YAML file: `themes.yaml`. Each theme entry includes:

- `name`: human-readable identifier (e.g., `tool-call`)
- `keywords`: list of keyword patterns for Phase 1 regex matching
- `owner`: responsible person (for quarterly review)
- `created_at`: ISO date of creation
- `status`: `active` or `deprecated`

**Lifecycle:**

- **Propose**: New themes require owner nomination + approval from project lead
- **Review**: Active themes are reviewed quarterly; deprecated if <2 hits in 30 days
- **Retire**: Deprecated themes are excluded from convergence scoring after 90 days
- **Reinstate**: Deprecated themes can be reactivated if signals re-emerge

The "other" catch-all bucket captures signals that don't match any active theme and is reviewed during quarterly theme audits for potential new theme creation.

---

## 6. Falsification Engine

### 6.1 Exponential decay scoring

Replace the 7-day dead thesis rule with:

```
thesis_score = initial_score × e^(-λ × days_since_last_signal)

where λ = 0.1 (configurable)
```

A thesis is "dead" when its score falls below a configurable threshold (default: 0.1), not when it hits a fixed day count. This naturally handles:
- Fast-dying trends (score drops quickly)
- Slow-burn trends (score stays elevated)
- Revived trends (new signal resets the decay clock)

### 6.2 Cross-source validation

A signal is flagged "unverified" if:
- Only 1 source has primary (non-derivative) coverage
- The signal appears only in echo chambers (e.g., HN upvotes ≠ real adoption)
- A counter-narrative exists in the same time window

### 6.3 Calibration Process

- **Initial parameters**: Start with λ = 0.1 (half-life ~7 days) for all themes at deployment
- **Validation window**: After 30 days of live operation, validate against historical data
- **Adjustment triggers**:
  - If >20% of confirmed real trends were falsely killed → decrease λ (e.g., to 0.05, slower decay)
  - If >30% of noise signals were incorrectly confirmed → increase λ (e.g., to 0.15, faster decay)
- **Documentation**: Record calibration decisions in `calibration_log.md` with date, old/new λ values, and rationale

Re-calibrate quarterly or whenever a major theme dictionary change is made.

---

## 7. Output Layer

### 7.1 Consumer interfaces (progressive rollout)

| Consumer | Interface | Phase |
|---|---|---|
| **Bob** (trend tracker) | Flask REST API: `GET /trends?theme=&period=7d` | 4 |
| **Alice** (content creator) | Daily file drops: `/output/YYYY-MM-DD/trends.yaml` | 4 |
| **Sam** (Hermes agent) | MCP server: `oracle_search`, `oracle_trends`, `oracle_verdicts` | 6 |

### 7.2 REST API (Phase 4)

Flask endpoints:
- `GET /health` — pipeline status, last run time, adapter health
- `GET /trends` — active themes with convergence scores
- `GET /entries` — search entries (keyword + phase 2: semantic)
- `GET /verdicts` — confirmed/dead theses
- `GET /convergence` — cross-source convergence matrix

### 7.3 File drops (Phase 4)

Daily structured output at a known path:
```
/output/YYYY-MM-DD/
  trends.yaml       # Human-readable daily digest
  signals.json      # Structured machine-readable output
  verdicts.json     # Confirmed/dead thesis list
```

### 7.4 MCP server (Phase 6)

MCP tools for Hermes agent integration:
- `oracle_search(query, source, date_range)` — search entries
- `oracle_trends(theme, convergence_threshold)` — get active trends
- `oracle_verdicts(status)` — confirmed or dead theses
- `oracle_latest(source)` — most recent entry per source

### 7.5 MCP Tool Signatures

All MCP tools must implement these minimum signatures:

- **get_trends()**:
  - Request: `{}` (no params)
  - Response: `{ "trends": [{"name": str, "score": float, "sources": [str], "decay_score": float}] }`

- **search_entry(query: str, source: str | None = None)**:
  - Request: `{ "query": str, "source": str | null }`
  - Response: `{ "entries": [{"title": str, "url": str, "summary": str, "score": float}] }`

- **get_convergence_report()**:
  - Request: `{}`
  - Response: `{ "converged": [{"entity": str, "sources": [str], "confidence": float}] }`

Tools must validate input types and return empty arrays (not errors) for valid queries that yield no results.

### 7.6 API Authentication Model

- **Phase 4 (MVP)**: Simple API key in `X-API-Key` header. No expiration, stored in config file (`/etc/athena/api_keys.yaml`)
- **Phase 6 (Production)**: JWT bearer token with scopes (read-only, read-write, admin). Tokens expire after 24 hours; refresh via `/auth/token` endpoint

Auth failures return HTTP 401 with `{ "error": "unauthorized" }`. Rate limiting applies per-key: 100 requests/minute.

---

## 8. Observability and Reliability

### 8.1 Logging

Structured JSON logging via stdlib `logging` with JSON formatter. Per-pipeline-stage logs (ingest, dedup, theme, falsification, summarize) with source-level granularity.

### 8.2 Health endpoint

`GET /health` returns:
```json
{
  "status": "ok",
  "last_run": "2026-07-08T13:00:00Z",
  "last_run_duration_sec": 245,
  "entries_since_last_run": 127,
  "adapters": {
    "github": {"status": "ok", "fetched": 20},
    "arxiv": {"status": "ok", "fetched": 15},
    "reddit": {"status": "error", "fetched": 0, "error": "429 rate limited"}
  }
}
```

### 8.3 Alerting

Discord/Slack webhook triggered when:
- An adapter fails for 2+ consecutive days
- Pipeline run exceeds 2× expected duration
- SQLite database integrity check fails

### 8.4 Graceful degradation

If Ollama is unreachable:
- Ingestion continues normally
- Summarization is skipped, entries stored with `summary = null`
- **Deferral**: next run summarizes pending entries
- **Alert**: "summarization deferred, N entries pending"

### 8.5 Log Retention and SLIs

- **Log retention**: 30 days rolling; gzip-compressed after 7 days to save disk space
- **SLI definitions**:
  - **Pipeline success rate**: >95% of daily runs complete without critical failure (exit code 2)
  - **Adapter availability**: >90% of scheduled runs successfully fetch each source (per-source metric)
  - **Theme detection accuracy**: ≥80% of manually verified trends identified correctly in first week

### 8.6 Alerting Matrix

| Condition | Channel | Severity | Response Time |
|---|---|---|---|
| Adapter fails >2 consecutive runs | Discord webhook | P2 | Investigate within 1 hour |
| Pipeline exit code 2 | Discord webhook + email | P1 | Investigate within 30 minutes |
| DB disk usage >85% | Discord webhook | P2 | Investigate within 2 hours |
| Ollama unreachable >5 min | Discord webhook | P2 | Restart service if needed |
| Pipeline success rate <90% for 3 days | Email + dashboard | P3 | Review next cycle |

Alerts are deduplicated: same condition won't fire again until resolved.

---

## 9. Scheduling

### 9.1 Phase 1: Cron

- `oracle-pipeline.sh` invoked by cron at 13:00 UTC daily
- `flock`/PID file prevents overlapping runs
- Exit codes: 0 = success, 1 = partial failure, 2 = total failure

### 9.2 Phase 2: systemd timers

- `Persistent=true` catches up on missed runs
- `RandomizedDelaySec` prevents thundering herd
- `OnFailureSec` for retry logic
- Better logging than cron (`journalctl -u athena-timer`)

### 9.3 Why not APScheduler (Phase 1)

APScheduler adds in-process async daemon overhead. Cron/systemd is OS-level, zero process memory cost, and sufficient for daily runs. APScheduler is the Phase 2 target if dynamic scheduling (user-configurable refresh rates, per-source intervals) is needed.

### 9.4 Scheduling Recommendation

- **Phase 1 (MVP)**: Use system cron for daily runs at 13:00 UTC. Sufficient for fixed schedule, zero process memory cost, OS-level reliability.

- **Phase 2**: Switch to systemd timers if dynamic scheduling needed (skip runs on holidays, adjust time zones). Better observability and integration with monitoring tools.

- **APScheduler**: Only use if per-source intervals are required (e.g., arXiv every 6 hours, Reddit every 30 min). Adds ~50MB process overhead — not recommended for Phase 1.

**Recommendation**: Start with cron for Phase 1. Evaluate whether Phase 2 sources need different intervals before considering systemd timers or APScheduler.

---

## 10. Inference

### 10.1 Summarization

**Model:** Ollama `llama3.2:1b` (or `qwen2.5:0.5b` for lower resource)
**Deployment:** Host-level Ollama service, pipeline calls via HTTP REST API
**Contract:** Model-agnostic — `summarize(text) → (summary, model)` interface
**Graceful degradation:** If Ollama is down, store raw text and defer summarization

### 10.2 Why not in-container Ollama

Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB container is physically impossible. Running it host-level means the pipeline process stays within budget and Ollama can share resources with other services.

### 10.3 Model Upgrade Process

- **Swap model**: Replace model file; update systemd unit `ExecStart` path if needed
- **Restart service**: `systemctl restart qwythos-gpu0.service` (or equivalent unit)
- **Verify health**: `curl http://localhost:8081/v1/health` — should return `{ "status": "ready" }`
- **Quality check**: Run first summarization on known-good source (e.g., arXiv paper), compare output against baseline summary for content accuracy, length consistency, and hallucination rate
- **Rollback**: If quality degrades (e.g., summary length <50 tokens, hallucination rate >10%), revert to previous model file immediately

**Quality metrics**: Pass if summary is >50 tokens, no hallucinations on known entities, and theme detection matches golden samples. Only proceed with upgrade after successful verification.

---

## 11. Security

| Requirement | Implementation |
|---|---|
| No hardcoded secrets | `GITHUB_TOKEN`, `HUGGINGFACE_TOKEN` as env vars or mounted secret files |
| TLS for outbound | All HTTP adapters use `https://` |
| Least privilege | Pipeline runs as standard user (no sudo) |
| DB protection | `chmod 600 oracle.db` |
| Input sanitization | Parameterized SQL queries, no string concatenation |

### 11.2 MVP Security Baseline

> **Updated 2026-07-12:** Container-specific hardening below is *design intent*, not the current deployment. The live system runs as the `vpsadmin` standard user (no sudo) under Hermes cron. The equivalent protections that DO apply today: no hardcoded secrets (`.env` sourced by the pipeline scripts), DB file `chmod 600`, parameterized SQL. Container hardening is deferred to the future containerization task.

- **Current (bare-script) baseline**:
  - Runs as `vpsadmin` standard user (no sudo)
  - Secrets in `/home/vpsadmin/oracle/.env` (`chmod 600`), never in code
  - DB protected: `chmod 600 oracle.db`
  - All outbound adapters use `https://`
- **Deferred (container) hardening** *(design intent, not live)*:
  - Run as non-root user (`user: nobody` in Dockerfile)
  - Read-only filesystem where possible (except `/tmp`, `/var/log`)
  - No SSH access inside container; pipeline is cron-triggered, no interactive access needed
  - Minimal base image: `python:3.11-slim` (no dev tools, no git)

- **Dependency scanning**:
  - CI pipeline runs `pip audit` or `safety check` on every push to `MVP-milestone`
  - Fail build if critical vulnerabilities found (>CVSS 7.0)
  - Warn on medium/high vulnerabilities; require manual review before merging

- **Secret management**:
  - No secrets in code or config files (use environment variables at runtime)
  - API keys stored in `/etc/athena/secrets.yaml` with restricted permissions (`chmod 0600`)

**Enforcement**: Security checks are automated in CI; local development is permissive but container builds must pass all scans.

---

## 12. Implementation Phases

| Phase | Scope | Deliverable |
|---|---|---|
| **P0: Foundation** | schema.sql + sqlite-vec design, adapter registry, oracle-pipeline.sh skeleton | Empty but valid pipeline |
| **P1: First data** | arXiv + RSS adapters, SQLite storage, keyword convergence, dedup | Live data flowing |
| **P2: Full ingest** | GitHub, HN, HF, Reddit adapters, rate limiting, structured logging | All 6 sources live |
| **P3: Falsification** | Exponential decay scoring, Ollama summarization, graceful degradation | Trend verdicts working |
| **P4: Consumption** | Flask API, daily file drops, health endpoint, alerting webhooks | Bob and Alice can consume |
| **P5: Validation** | 7-day UAT window, Hermes cron integration, exit codes | System runs unattended |
| **P6: Scale** | sqlite-vec + embeddings, MCP server, APScheduler, BERTopic themes | Research-grade system |

### 12.4 Definition of Done per Phase

Each phase must pass all listed criteria before being marked complete:

**P0 (Foundation)**:
- `schema.sql` creates all tables without errors
- Empty pipeline runs cleanly with exit code 0
- `oracle-pipeline.sh` is idempotent (safe to run twice)
- Adapter registry loads all 6 adapters

**P1 (First data)**:
- arXiv + RSS adapters fetch successfully
- Entries stored in SQLite with correct schema
- Keyword convergence detects at least 1 theme
- Deduplication works (no duplicate entries)

**P2 (Full ingest)**:
- All 6 adapters fetch successfully in one run
- Rate limiting enforced per adapter
- Structured JSON logs emitted per pipeline stage
- No adapter failure kills the pipeline

**P3 (Falsification)**:
- Exponential decay scoring implemented
- Ollama summarization works with graceful degradation
- Trend verdicts computed: confirmed/emerging/dead
- Running over 7 days shows false signals dying

**P4 (Consumption)**:
- REST API endpoints return valid JSON
- Daily file drops written to `/output/`
- Health endpoint reports accurate status
- Alerting webhooks fire on simulated failures

**P5 (Validation)**:
- 7-day UAT: pipeline runs unattended without intervention
- Hermes cron integration works
- Exit codes correct: 0=success, 1=partial, 2=failure
- Pipeline completes <30 minutes end-to-end

**P6 (Scale)**:
- sqlite-vec + embeddings operational
- MCP server responds to all 3 tools
- APScheduler handles per-source intervals
- System stays within 150MB pipeline memory budget

**General**: No critical bugs open, all unit tests pass, CI green, security scan clean.

---

## 13. Backport to PRD: Requirements to Add

The following requirements are implied by this design and should be added to `docs/MVP-PRD.md`:

### 13.1 Platform requirements (REQ-PLT-XX)

| ID | Requirement |
|---|---|
| REQ-PLT-05 | All processes run as standard user (no sudo) — *already exists* |
| REQ-PLT-10 | The pipeline process shall not exceed 500MB of RSS memory (excluding host-level Ollama) |
| REQ-PLT-15 | The system shall support deployment on a VPS with 2GB total RAM (pipeline + Ollama + OS) |
| REQ-PLT-20 | Ollama inference shall run as a host-level service, not inside the pipeline container |
| REQ-PLT-25 | The pipeline shall use SQLite as the sole database (no PostgreSQL, no Elasticsearch, no Redis) |

### 13.2 Reliability requirements (REQ-REL-XX)

| ID | Requirement |
|---|---|
| REQ-REL-05 | The system operates autonomously without human interaction — *already exists* |
| REQ-REL-10 | Failed source fetches retry with exponential backoff — *already exists* |
| REQ-REL-15 | Previously stored data is not lost on restart or failure — *already exists* |
| REQ-REL-20 | The pipeline completes successfully even if 1+ sources are unavailable — *already exists* |
| REQ-REL-25 | If Ollama is unreachable, ingestion continues and summarization defers to the next run |
| REQ-REL-30 | The pipeline uses flock/PID file to prevent overlapping runs |

### 13.3 Observability requirements (REQ-DIAG-XX)

| ID | Requirement |
|---|---|
| REQ-DIAG-05 | Structured JSON logs with timestamps and severity — *already exists* |
| REQ-DIAG-10 | Health endpoint reports system status and last successful run — *already exists* |
| REQ-DIAG-15 | Per-source success/failure and fetch counts logged per run — *already exists* |
| REQ-DIAG-20 | Webhook alert (Discord/Slack) fires when an adapter fails for 2+ consecutive days |
| REQ-DIAG-25 | run_log table captures per-run metrics queryable via SQL |

### 13.4 Integration requirements (REQ-INT-XX)

| ID | Requirement |
|---|---|
| REQ-INT-05 | Structured, machine-readable output (JSON) consumable by external tools — *already exists* |
| REQ-INT-10 | Adapter layer supports adding new sources without modifying core pipeline logic — *already exists* |
| REQ-INT-15 | REST API exposes GET /trends, /entries, /verdicts endpoints |
| REQ-INT-20 | Daily file drop at configurable path with structured output (YAML + JSON) |
| REQ-INT-25 | MCP server exposes oracle_search, oracle_trends, oracle_verdicts tools (Phase 6) |

### 13.5 Data requirements (REQ-DATA-XX) *(new section)*

| ID | Requirement |
|---|---|
| REQ-DATA-05 | Entries are deduplicated by (source, source_id) with cross-source URL hash deduplication |
| REQ-DATA-10 | FTS5 full-text index on title and extracted_text for keyword search |
| REQ-DATA-15 | Convergence detection: theme appears in ≥3 independent sources within 24h window |
| REQ-DATA-20 | Falsification uses exponential decay scoring (configurable λ), not fixed day thresholds |
| REQ-DATA-25 | Data retention: raw entries 90 days, summaries/convergence 365 days, periodic VACUUM |

### 13.6 Security requirements (REQ-SEC-XX)

| ID | Requirement |
|---|---|
| REQ-SEC-05 | All outbound HTTP uses TLS — *already exists* |
| REQ-SEC-10 | No secrets hardcoded or stored in plaintext — *already exists* |
| REQ-SEC-15 | Least-privilege access for outbound API calls — *already exists* |
| REQ-SEC-20 | SQLite database file permissions set to 600 (owner-only read/write) |
| REQ-SEC-25 | All SQL queries use parameterized statements (no string concatenation) |

### 13.7 Scheduling requirements (REQ-SCH-XX) *(new section)*

| ID | Requirement |
|---|---|
| REQ-SCH-05 | Pipeline entry point is oracle-pipeline.sh (idempotent, single command) |
| REQ-SCH-10 | Default schedule: 13:00 UTC daily |
| REQ-SCH-15 | Exit codes: 0 = success, 1 = partial failure, 2 = total failure |
| REQ-SCH-20 | Overlapping run prevention via flock or PID file check |

---

## 14. Design Decisions Summary (Why)

| Decision | Why | Revisit |
|---|---|---|
| **SQLite over PostgreSQL** | Zero external dependency, single file, FTS5 built-in, handles 1M rows fine. pgvector is premature at this scale. | P6 (if scale demands pgvector) |
| **Ollama host-level** | 150MB container cannot fit Ollama + model (~2GB). Host-level lets pipeline stay within budget. | P2 (if inference needs change) |
| **Flask over FastAPI** | ~1MB vs ~100MB runtime overhead. FastAPI is Phase 2 target; Flask suffices for internal REST API. | P2 (when FastAPI becomes viable) |
| **Keyword co-occurrence (Phase 1)** | Zero-dependency, explainable, works at 150MB. Embeddings (Phase 2) add semantic convergence. | P2 (when embeddings ready) |
| **Fixed themes + catch-all** | BERTopic requires 4GB RAM. Fixed themes with "other" bucket is the pragmatic constraint choice. | P6 (when auto-discovery needed) |
| **Cron over APScheduler (Phase 1)** | OS-level, zero process memory cost. APScheduler is Phase 2 for dynamic scheduling. | P2 (if per-source intervals needed) |
| **HTTP-only adapters** | All 6 sources have programmatic APIs. Playwright adds 300MB+ overhead and fragility. | N/A (stable) |
| **Exponential decay over 7-day rule** | One-line formula, no fixed threshold. Handles fast-dying and slow-burn trends naturally. | N/A (stable) |
| **Deduplication required** | arXiv papers appear on HN/Reddit/Twitter. Without dedup, same signal counted 3× = false convergence. | N/A (stable) |
| **Graceful degradation on Ollama** | Ingestion must not depend on summarization. Store raw data, defer summaries. | N/A (stable) |

## 15. Testing Strategy

### 15.1 Unit Tests

- **Per adapter**: Test each of the 6 adapters with known-good endpoints. Verify:
  - Returns valid JSON with required schema fields (source, source_id, title, url, timestamp)
  - Handles rate limits gracefully (no infinite loops)
  - Correct error codes for 429/503
- **Scoring functions**: Unit tests for exponential decay, convergence scoring, and theme matching. Include edge cases (empty input, negative scores).

### 15.2 Integration Tests

- Run pipeline against seed dataset (100 entries from arXiv + RSS). Verify:
  - All adapters fetch successfully
  - Deduplication removes duplicates correctly
  - Theme detection identifies at least 3 themes
  - No critical errors in logs

### 15.3 E2E Tests

- **Cron-to-snapshot**: Run full pipeline via cron, verify output files match expected snapshot (golden file comparison)
- **Stress test**: Run 10 consecutive daily cycles with simulated failures (adapter timeout, Ollama down). Verify graceful degradation and recovery

**Test coverage goal**: 80% of critical paths covered by automated tests. Manual testing for theme quality and summary accuracy.

## 16. Deployment and CI/CD

### 16.1 CI Pipeline

> **Updated 2026-07-12:** CI originally built a Docker image per push. The live system has no CI/Docker build. The equivalent checks run manually/optionally: lint, `pip audit`, and `pytest` if tests exist. Container build is deferred.

On every push to `MVP-milestone`:
- **Lint**: `ruff check`, `mdlint docs/`
- **Test**: Run unit tests (`pytest tests/`), integration tests with seed dataset
- **Scan**: Run `pip audit`; fail if critical vulnerabilities (>CVSS 7.0) found
- *(Deferred: Docker image build/tag with commit SHA)*

### 16.2 Deployment

> **Deployment reality (updated 2026-07-12):** This document originally specified Docker containerization (150MB cap, non-root, `python:3.11-slim`). The deployed system does **not** use containers. It runs as bare Python scripts under the `vpsadmin` standard user, invoked by Hermes cron (`oracle-pipeline.sh` at 13:00 UTC). The 150MB budget, non-root, and read-only-filesystem hardening described below remain *design intent* and are tracked as a future containerization task — not the current state. Ollama is also not deployed; summarization is deferred (graceful degradation is the live behavior).

- **Local dev / Production**: Bare Python on the VPS, no Docker. `pip install` deps, run `schema.sql`, source `.env`, then Hermes cron drives `oracle-pipeline.sh`.
- **Ollama**: Not currently running. Pipeline stores raw text and defers summarization (see §10.2 / REQ-REL-25). Adding Ollama is a separate step, host-level.

### 16.3 Rollback Procedure

If deployment fails or quality degrades:
1. Stop services: `systemctl stop oracle-pipeline qwythos-gpu0`
2. Restore previous code: `git checkout <good-tag>`
3. Rebuild Docker image from restored code
4. Restart services: `systemctl start oracle-pipeline qwythos-gpu0`
5. Verify health: `curl http://localhost:8081/v1/health`

**Rollback window**: Must complete within 5 minutes of failure detection.

**Tagging**: Each deploy is tagged with semantic versioning (`v1.0.0`, `v1.1.0`) for easy rollback reference.

## 17. Operational Runbooks

### 17.1 Daily Pipeline Verification

At 13:00 UTC after each run:
1. Check logs: `journalctl -u oracle-pipeline --since "today" | grep ERROR`
2. Verify exit code in `/var/log/oracle-pipeline/run_log.txt` — should be 0
3. Confirm output: `ls -lh /output/$(date +%Y-%m-%d)/` — should have `summary.json` and `metrics.json`
4. If any check fails, investigate with the relevant runbook below

### 17.2 Database Recovery from Corruption

If `sqlite3 oracle.db 'PRAGMA integrity_check'` returns errors:
1. Stop pipeline: `systemctl stop oracle-pipeline`
2. Restore from last backup: `cp /backup/oracle.db.bak-YYYYMMDD oracle.db`
3. Verify integrity: `sqlite3 oracle.db 'PRAGMA integrity_check'` — should return "ok"
4. Restart pipeline: `systemctl start oracle-pipeline`

**Prevention**: Daily compressed backups to `/backup/`, retention 30 days.

### 17.3 Adapter Failure Investigation

If adapter fails repeatedly (>2 consecutive runs):
1. Check logs: `journalctl -u oracle-pipeline --since "today" | grep -A5 "Adapter"`
2. Test endpoint manually: `curl -X GET <adapter_url>` — verify HTTP status
3. Check rate limit headers: `curl -I <adapter_url> | grep -i 'x-ratelimit'`
4. If rate-limited: Wait `retry_after` seconds, pipeline retries next cycle
5. Escalate to project lead if >3 consecutive failures

### 17.4 Ollama Service Restart

If Ollama becomes unresponsive:
1. Check status: `systemctl status qwythos-gpu0.service`
2. View logs: `journalctl -u qwythos-gpu0.service --since "today"`
3. Restart service: `systemctl restart qwythos-gpu0.service`
4. Verify health: `curl http://localhost:8081/v1/health` — should return `{ "status": "ready" }`
5. Check GPU: `nvidia-smi` — ensure service actually loaded

**Escalation**: If issue persists after restart, notify project lead.

**Runbook maintenance**: Update runbooks when features change. Document changes in `runbook_changes.md`.

## 18. Risk Register and Assumptions

### 18.1 Risk Register

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Source API changes (rate limits, endpoint shifts) | High | High | Auto-retry with exponential backoff; monitor adapter health; log API changes for review |
| Model quality drift (summarization accuracy degrades) | Medium | Medium | Weekly golden sample comparison; rollback if hallucination rate >10% or summary <50 tokens |
| Disk space exhaustion (backups, logs) | High | High | Automated 30-day retention; alert at 85% usage; compress old backups |
| Single operator bottleneck (manual theme review) | Medium | Medium | Documented theme governance; quarterly review cycle; catch-all "other" bucket |
| Network outage (all adapters fail) | Low | High | Graceful degradation: store raw data without summaries; resume when network returns |
| SQLite corruption during schema migration | Low | Critical | Pre-migration backup; atomic migration scripts; integrity check after each migration |

### 18.2 Key Assumptions

- All 6 source APIs remain stable for at least 6 months (no breaking changes)
- GPU memory available for Qwythos inference (~15GB free on GPU0)
- Network connectivity to all sources is available during pipeline runs
- Single operator can complete manual theme review within 24 hours
- Disk space sufficient for 30-day backup retention (~10GB)

**Risk review**: Quarterly review of this register; update mitigations when new risks emerge.

**Assumption tracking**: If an assumption proves false, document the deviation in `assumption_deviations.md` and reassess risks.

## 19. Dependencies and Tooling

### 19.1 Python Runtime Dependencies

All dependencies pinned to exact versions in `requirements.txt`:

- **requests**: HTTP client for all source APIs
- **beautifulsoup4**: HTML parsing for RSS feeds
- **feedparser**: RSS/Atom feed handling (primary)
- **tenacity**: Retry logic with exponential backoff (used by all adapters)
- **flask**: Internal REST API for Phase 4+ (served on port 8081)

No external database dependencies — SQLite is built into Python.

### 19.2 Runtime System Dependencies

- **Ollama**: Host-level inference service (~2GB RAM, GPU acceleration)
- **systemd**: Service management for pipeline and inference (`oracle-pipeline.service`, `qwythos-gpu0.service`)
- **cron**: Daily schedule trigger (Phase 1); systemd timers for Phase 2
- **sqlite3**: Database CLI for backup/recovery commands

### 19.3 Development Tooling

- **ruff**: Linting and formatting (Python)
- **pytest**: Unit and integration test framework
- **docker**: Containerization for pipeline
- **pip audit / safety**: Dependency vulnerability scanning (CI checks)

### 19.4 Dependency Update Policy

- Critical security updates: Apply within 7 days of CVE disclosure
- Minor version updates: Test in staging before production deployment
- Major version upgrades: Require migration scripts and rollback plan

**Tooling policy**: No new dependencies without approval from project lead; document rationale in `dependency_justifications.md`.

**Lock file**: `requirements.txt` pinned to exact versions for reproducible builds.

---

*Document prepared via multi-model analysis: Qwythos-9B (architectural critique), Qwen3.5-9B (implementation evaluation), and cross-review synthesis. 4 delegations, 2 rounds of debate. Raw reviews saved in the same directory.*