> *Surfacing cross-source convergence and using falsification to distinguish real momentum from noise.*
Athena is an autonomous research intelligence engine that ingests from multiple fragmented sources, detects when the same signals appear across independent channels, and uses decay-based falsification to separate genuine trends from one-day spikes. The system is model-agnostic, lightweight, and designed to run unattended on a resource-constrained VPS.
### How this design supports the north star
| North Star Principle | Design Decision | Why |
|---|---|---|
| Cross-source convergence | Keyword co-occurrence matrix (Phase 1), embeddings + vector search (Phase 2) | Keyword co-occurrence is the simplest convergence detector: if the same entity appears in ≥3 independent sources within a time window, it's converging. Embeddings Phase 2 adds semantic convergence for signals that use different words but mean the same thing. |
| Falsification over confirmation | Exponential decay scoring | A 7-day hard cutoff is blunt: some trends die in 48 hours, some take 60 days to validate. `score = base_score * e^(-λ * days_since_last_signal)` naturally scores dying trends low and sustained trends high without arbitrary day thresholds. |
| Autonomous operation | Cron/scheduled pipeline + graceful degradation | The pipeline runs daily without human intervention. If Ollama is down, ingestion continues and summarization defers to the next run. If one adapter fails, the rest still run. |
| Lightweight deployment | Python + SQLite + Flask + host-level Ollama | No PostgreSQL, no Elasticsearch, no Redis, no Kubernetes. A single Python process, a single SQLite file, and an external Ollama REST API. |
### MVP Success Criteria
The MVP is considered successful when:
1.**Research Time**: Bob answers 'what's trending this week' in <15 min of research time total (across all sources).
2.**Signal Detection**: Over 7 days, the theme scan correctly identifies ≥1 real trend that has genuine cross-source convergence.
3.**Noise Rejection**: Over the same 7-day period, the falsification engine kills ≥1 false signal (a one-day spike with no sustained arrivals) to demonstrate decay-based filtering is working.
| Hacker News | Firebase API | Unofficial, ~30 req/sec | No |
| HuggingFace | REST API | Throttled if aggressive | `HUGGINGFACE_TOKEN` |
| RSS Feeds | RSS XML | Varies | No |
**Decision:** HTTP-only adapters, no Playwright/Selenium. All 6 sources have programmatic APIs. Playwright would add Chromium's 300MB+ overhead and fragility.
### 4.2 Deduplication
arXiv papers appear on HN, Reddit, and Twitter. Without deduplication, the same signal is counted 3× and produces false convergence.
- **Phase 1:** UNIQUE constraint on `(source, source_id)` + URL hash dedup across sources
- **Phase 2:** SimHash/MinHash content fingerprinting for near-duplicate detection
### 4.3 Rate limiting and retries
- Per-adapter rate limits enforced in the adapter class
- `tenacity` library for exponential backoff on transient failures (429, 503, timeout)
- One failing adapter does not kill the pipeline
### 4.4 Adapter Interface Contract
All adapters must implement this minimal interface:
- **name() → str**: Unique identifier for the source (e.g., `"arxiv"`)
- `source_id` (str): Unique per-source ID for deduplication
- `title` (str): Human-readable title
- `url` (str): Direct URL to the entry
- `timestamp` (datetime): Publication/update time
- `raw_score` (float): Source-specific signal strength (0.0–10.0)
- `body` (str): Raw text content for summarization and theme tagging
**Custom Exceptions:**
- `RateLimitError`: Raised when source returns 429 or similar; includes `retry_after` (seconds)
- `SourceUnavailableError`: Raised when source is down (5xx) or unreachable
Adapters must not raise other exceptions on normal operation; unexpected errors should be logged with full traceback and surfaced via the run_log, not propagated to the pipeline.
---
## 5. Theme Tagging and Convergence Detection
### 5.1 Phase 1: Keyword co-occurrence
Pre-defined keyword dictionaries per theme. An entry is tagged if ≥2 keywords from a theme dictionary appear in its title or extracted text. A theme "converges" if it appears in ≥3 independent sources within the last 24 hours.
**Why keyword first at 150MB:** Keyword matching is zero-dependency, explainable, and works within the memory constraint. FTS5 provides fast retrieval.
**Fixed themes:** The initial 4 themes (`tool-call`, `context`, `compute`, `trust`) are seeds, not a hard limit. An "other" catch-all bucket captures signals that don't match predefined themes.
### 5.2 Phase 2: Embeddings + auto-discovery
- **all-MiniLM-L6-v2** (22M params, ~80MB) for sentence embeddings
- **sqlite-vec** for in-database ANN search
- **BERTopic** (or equivalent) for semi-supervised theme discovery, seeded from the Phase 1 dictionary
- Hybrid query: FTS5 for precision (keyword match) + vector for recall (semantic match), merged via Reciprocal Rank Fusion
**Why not keyword forever:** Keyword matching cannot detect semantic convergence (different words, same concept) and requires constant manual dictionary updates. Embeddings are the eventual target; Phase 1 is the bridge.
- `keywords`: list of keyword patterns for Phase 1 regex matching
- `owner`: responsible person (for quarterly review)
- `created_at`: ISO date of creation
- `status`: `active` or `deprecated`
**Lifecycle:**
- **Propose**: New themes require owner nomination + approval from project lead
- **Review**: Active themes are reviewed quarterly; deprecated if <2 hits in 30 days
- **Retire**: Deprecated themes are excluded from convergence scoring after 90 days
- **Reinstate**: Deprecated themes can be reactivated if signals re-emerge
The "other" catch-all bucket captures signals that don't match any active theme and is reviewed during quarterly theme audits for potential new theme creation.
- Better logging than cron (`journalctl -u athena-timer`)
### 9.3 Why not APScheduler (Phase 1)
APScheduler adds in-process async daemon overhead. Cron/systemd is OS-level, zero process memory cost, and sufficient for daily runs. APScheduler is the Phase 2 target if dynamic scheduling (user-configurable refresh rates, per-source intervals) is needed.
### 9.4 Scheduling Recommendation
- **Phase 1 (MVP)**: Use system cron for daily runs at 13:00 UTC. Sufficient for fixed schedule, zero process memory cost, OS-level reliability.
- **Phase 2**: Switch to systemd timers if dynamic scheduling needed (skip runs on holidays, adjust time zones). Better observability and integration with monitoring tools.
- **APScheduler**: Only use if per-source intervals are required (e.g., arXiv every 6 hours, Reddit every 30 min). Adds ~50MB process overhead — not recommended for Phase 1.
**Recommendation**: Start with cron for Phase 1. Evaluate whether Phase 2 sources need different intervals before considering systemd timers or APScheduler.
---
## 10. Inference
### 10.1 Summarization
**Model:** Ollama `llama3.2:1b` (or `qwen2.5:0.5b` for lower resource)
**Deployment:** Host-level Ollama service, pipeline calls via HTTP REST API
**Graceful degradation:** If Ollama is down, store raw text and defer summarization
### 10.2 Why not in-container Ollama
Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB container is physically impossible. Running it host-level means the pipeline process stays within budget and Ollama can share resources with other services.
### 10.3 Model Upgrade Process
- **Swap model**: Replace model file; update systemd unit `ExecStart` path if needed
- **Quality check**: Run first summarization on known-good source (e.g., arXiv paper), compare output against baseline summary for content accuracy, length consistency, and hallucination rate
- **Rollback**: If quality degrades (e.g., summary length <50 tokens, hallucination rate >10%), revert to previous model file immediately
**Quality metrics**: Pass if summary is >50 tokens, no hallucinations on known entities, and theme detection matches golden samples. Only proceed with upgrade after successful verification.
---
## 11. Security
| Requirement | Implementation |
|---|---|
| No hardcoded secrets | `GITHUB_TOKEN`, `HUGGINGFACE_TOKEN` as env vars or mounted secret files |
| TLS for outbound | All HTTP adapters use `https://` |
| Least privilege | Pipeline runs as standard user (no sudo) |
| REQ-SCH-20 | Overlapping run prevention via flock or PID file check |
---
## 14. Design Decisions Summary (Why)
| Decision | Why | Revisit |
|---|---|---|
| **SQLite over PostgreSQL** | Zero external dependency, single file, FTS5 built-in, handles 1M rows fine. pgvector is premature at this scale. | P6 (if scale demands pgvector) |
| **Ollama host-level** | 150MB container cannot fit Ollama + model (~2GB). Host-level lets pipeline stay within budget. | P2 (if inference needs change) |
| **Flask over FastAPI** | ~1MB vs ~100MB runtime overhead. FastAPI is Phase 2 target; Flask suffices for internal REST API. | P2 (when FastAPI becomes viable) |
5. Check GPU: `nvidia-smi` — ensure service actually loaded
**Escalation**: If issue persists after restart, notify project lead.
**Runbook maintenance**: Update runbooks when features change. Document changes in `runbook_changes.md`.
## 18. Risk Register and Assumptions
### 18.1 Risk Register
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Source API changes (rate limits, endpoint shifts) | High | High | Auto-retry with exponential backoff; monitor adapter health; log API changes for review |
| Model quality drift (summarization accuracy degrades) | Medium | Medium | Weekly golden sample comparison; rollback if hallucination rate >10% or summary <50 tokens |
| Disk space exhaustion (backups, logs) | High | High | Automated 30-day retention; alert at 85% usage; compress old backups |
| Single operator bottleneck (manual theme review) | Medium | Medium | Documented theme governance; quarterly review cycle; catch-all "other" bucket |
| Network outage (all adapters fail) | Low | High | Graceful degradation: store raw data without summaries; resume when network returns |
| SQLite corruption during schema migration | Low | Critical | Pre-migration backup; atomic migration scripts; integrity check after each migration |
### 18.2 Key Assumptions
- All 6 source APIs remain stable for at least 6 months (no breaking changes)
- GPU memory available for Qwythos inference (~15GB free on GPU0)
- Network connectivity to all sources is available during pipeline runs
- Single operator can complete manual theme review within 24 hours
- Disk space sufficient for 30-day backup retention (~10GB)
**Risk review**: Quarterly review of this register; update mitigations when new risks emerge.
**Assumption tracking**: If an assumption proves false, document the deviation in `assumption_deviations.md` and reassess risks.
## 19. Dependencies and Tooling
### 19.1 Python Runtime Dependencies
All dependencies pinned to exact versions in `requirements.txt`:
- Critical security updates: Apply within 7 days of CVE disclosure
- Minor version updates: Test in staging before production deployment
- Major version upgrades: Require migration scripts and rollback plan
**Tooling policy**: No new dependencies without approval from project lead; document rationale in `dependency_justifications.md`.
**Lock file**: `requirements.txt` pinned to exact versions for reproducible builds.
---
*Document prepared via multi-model analysis: Qwythos-9B (architectural critique), Qwen3.5-9B (implementation evaluation), and cross-review synthesis. 4 delegations, 2 rounds of debate. Raw reviews saved in the same directory.*
**Recommendation**: Add DoD checklist for each phase.
### 13. Backport to PRD
**Strengths**: Useful.
**Recommendation**: Consider moving actual requirement text to PRD to avoid duplication.
### 14. Design Decisions
**Strengths**: Good.
**Recommendation**: Add "Revisit in Phase X" column for key decisions.
## Missing Sections to Add
1. **Testing Strategy** (unit, integration, E2E)
2. **Deployment & CI/CD**
3. **Operational Runbooks**
4. **Risk Register & Assumptions**
5. **Dependencies & Tooling**
## Priority for Next Revision
Focus on adding Testing Strategy, Runbooks, and DoD per phase first. This will make the document truly delegation-ready.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
ty
Ty