athena-oracle/whitepaper.md

# Athena: AI Research Intelligence Engine

## White Paper — System Architecture, Pattern Detection, and Opportunity Discovery

**Source:** `Tony_tech/athena-oracle` (public, Gitea — http://localhost:3000/Tony_tech/athena-oracle) · branch `main`

---

## Table of Contents

1. **Abstract**
2. **Problem Statement: The AI Signal Crisis**
3. **System Overview**
4. **Source Architecture**
5. **Scoring Methodology**
6. **Verification Discipline**
7. **Trend Detection & Falsification**
8. **From Signal to Opportunity**
9. **Agentic Capability Assessment**
10. **Technical Specifications**
11. **Roadmap**
12. **Appendix**

---

## 1. Abstract

Athena is an autonomous AI research intelligence engine that continuously ingests, scores, and analyzes signals from five primary sources covering the full AI landscape: code adoption (GitHub), academic research (arXiv), practitioner sentiment (Reddit), industry news (Hacker News), and model releases (Hugging Face).

The system ingests ~100 entries per daily cycle, applies rule-based scoring with per-source normalization, generates structured summaries, and runs a falsification-based trend detection engine that answers the question: *"Is this pattern a real trend or a one-day coincidence?"*

Built in 3,731 lines of Python, Athena runs on a single VPS, requires no external LLM, and operates autonomously via daily cron. It is designed to surface cross-source convergence signals — the kind that matter before they become obvious — and to translate structured signal into actionable opportunity intelligence.

**Current state:** Five sources live and verified. Theme tracking operational. Awaiting 5–7 cron cycles for trend validation.

---

## 2. Problem Statement: The AI Signal Crisis

### 2.1 Volume

The AI ecosystem generates thousands of new signals daily:

- **GitHub:** 10,000+ new AI/ML repositories per week
- **arXiv:** 300+ AI/ML papers per day across cs.AI, cs.LG, cs.CL
- **Reddit:** 500+ AI-related posts daily across r/MachineLearning, r/LocalLLaMA, r/artificial
- **Hacker News:** 50+ AI stories in the top 500 daily
- **Hugging Face:** 2,000+ new model uploads daily

No human can read this volume. No single feed captures the full picture.

### 2.2 Fragmentation

Each source tells only a partial story:

- GitHub shows what's being built (code adoption)
- arXiv shows what's being researched (academic)
- Reddit shows what practitioners care about (sentiment)
- Hacker News shows what's being discussed (industry)
- Hugging Face shows what's being downloaded (model adoption)

**The signal that matters exists only across all five.** A tool on GitHub with 77K stars (ponytail), discussed on Reddit by four independent practitioners, trending on Hugging Face downloads, and covered in Hacker News — that is convergence. That is signal.

### 2.3 The Consequence

Without structured ingestion and cross-source analysis:

- **Opportunities are missed** because they surface on one feed you don't monitor
- **Trends are confused with noise** because one viral day doesn't mean a movement
- **Decisions are backward-looking** because by the time a trend is obvious, it's already priced in
- **Competition is blind** because others who see convergence act first

---

## 3. System Overview

Athena is a five-stage pipeline that runs daily at 13:00 UTC via cron:

```
                    ┌─────────────┐
                    │  5 Sources   │  GitHub, arXiv, Reddit, HN, HF
                    └──────┬──────┘
                           │  fetch(limit=20)
                           ▼
                    ┌─────────────┐
              ┌─────►│  Adapters   │  Per-source AI relevance filtering
              │      └──────┬──────┘
              │             │  structured entries
              │             ▼
              │      ┌─────────────┐
              │      │   Pipeline   │  Dedup → Score → Store (SQLite)
              │      └──────┬──────┘
              │             │  entries in DB
              │             ▼
              │      ┌─────────────┐
              │      │  Summarizer  │  Rule-based extraction (no LLM)
              │      └──────┬──────┘
              │             │  summaries written
              │             ▼
              │      ┌─────────────┐
              │      │  Theme Scan  │  Classify by 4 themes, count new arrivals
              │      └──────┬──────┘
              │             │  theme tags
              │             ▼
              │      ┌─────────────┐
              └──────┤   Archive   │  Soft-cap: >30d or >5000 entries → archive
                     └─────────────┘
```

**Key design decisions:**

1. **SQLite as source of truth** — single file, no infrastructure, queryable
2. **Rule-based summarization** — no LLM required, eliminates hallucination
3. **Score-type flagging** — `actual` vs `estimated` prevents false cross-source comparison
4. **Verification discipline** — every adapter's scores spot-checked against live data
5. **Theme-based trend detection** — tags concepts, not entries; counts fresh arrivals per cycle
6. **Soft-cap archiving** — preserves history without unbounded DB growth

---

## 4. Source Architecture

### 4.1 The SourceAdapter Interface

Every source implements a uniform interface:

```python
class SourceAdapter(ABC):
    @abstractmethod
    def name(self) -> str:
        """Source identifier: 'github', 'arxiv', 'reddit'."""
        pass

    @abstractmethod
    def fetch(self, query: str = "", limit: int = 20) -> list[dict]:
        """Return entries matching the unified DB schema."""
        pass
```

Adding a new source is **one line of registration** in the pipeline's adapter dictionary. Each adapter is an independent file (~300–500 lines) that:

1. Fetches from the source's primary API
2. Filters for AI relevance
3. Scores entries
4. Returns structured dictionaries matching the DB schema

### 4.2 Source Details

| Source | API | Score Type | What It Captures | Entries/Cycle |
|--------|-----|-----------|------------------|---------------|
| **GitHub** | GitHub REST API | `actual` (stars) | Code adoption, viral tools, agent frameworks | 20 |
| **arXiv** | arXiv API | `estimated` | Academic research, papers, breakthroughs | 20 |
| **Reddit** | RSS (JSON endpoints rate-limited) | `estimated` | Practitioner sentiment, discussion clusters | 20 |
| **Hacker News** | Firebase API | `actual` (points) | Industry news, practitioner announcements | 18–20 |
| **Hugging Face** | HF API | `actual` (likes) | Model releases, adoption, downloads | 20 |
| **Total** | | | **Full spectrum: code → research → discussion → news → models** | **~98** |

### 4.3 Why Five Sources?

The five-source coverage is deliberate — each captures a different signal layer:

| Layer | Source | Example |
|-------|--------|---------|
| **Code adoption** | GitHub | ponytail (77K⭐ in 25 days) |
| **Academic research** | arXiv | "Doomed from the Start: Early Abort of LLM Agent Episodes" |
| **Practitioner sentiment** | Reddit | "Competence Gate: gating tool-use on internal confidence" |
| **Industry news** | Hacker News | "GLM 5.2 and the coming AI margin collapse" (669 pts) |
| **Model adoption** | Hugging Face | GLM-5.2 (3,607 likes, 281K downloads) |

**Cross-source convergence** is the system's primary value. GLM-5.2 appearing on both HN (#1 story) and HF (#1 model) is independent validation — two sources, same signal, different audiences. That is genuinely more compelling than either source alone.

---

## 5. Scoring Methodology

### 5.1 The Score-Type Flag

Every entry carries a `score_type` flag:

- **`actual`** — real numbers from the source (GitHub stars, HN points, HF likes)
- **`estimated`** — heuristic-based scoring where no native popularity metric exists (arXiv papers, Reddit posts)

This prevents the fundamental error of comparing an arXiv score of 5.5 against a GitHub score of 5.5 — they mean different things. The pipeline explicitly ranks within-source, not cross-source.

### 5.2 Per-Source Scoring

#### GitHub (actual)
```
score = log10(stars)
```
Capped at 10.0. A repo with 100K stars scores ~5.0; 1M stars scores ~6.0.

#### arXiv (estimated)
Composite of paper-specific signals:
- Abstract relevance to AI/ML topics
- Category matching (cs.AI, cs.LG, cs.CL)
- Author count (proxy for collaborative effort)
- Recency bonus

**Deliberate design choice:** structural metadata (author count, abstract length) was removed after review revealed it was measuring paper properties, not AI relevance. Relevance signals now dominate.

#### Reddit (estimated)
Composite of:
- Upvote ratio (upvotes / total votes)
- Comment depth (descendants)
- Subreddit authority (r/MachineLearning > r/Startups)
- Post type (project posts score higher than links)

#### Hacker News (actual)
```
score = log10(points) + log10(comments)
```
Capped at 10.0. Both metrics are real numbers from the Firebase API.

#### Hugging Face (actual)
Composite of:
- Likes (log scale, primary adoption signal)
- Downloads (log scale, secondary — can be inflated by programmatic pulls)
- Pipeline tag relevance (text-generation, conversational, etc.)
- Library ecosystem (transformers, diffusers)
- AI-specific tag matching (agent, reasoning, alignment)
- Recency bonus (newer models get slight boost)

**Adoption ≠ relevance** — a 1M-download fine-tune is less interesting than a 500-download novel architecture. Pipeline tag and library signals carry weight to ensure content-relevance is not drowned out by raw popularity.

---

## 6. Verification Discipline

### 6.1 The Rule

Every adapter's scores must be verified against the live source before the adapter is trusted. This is not optional process — it caught the arXiv structural metadata bug (H. pylori scoring error) and the confidence-length bug in the summarizer.

### 6.2 Process

1. **Run the adapter** — fetch entries, store to DB
2. **Select 3 high-signal entries** from the DB
3. **Fetch the same entries from the live API**
4. **Compare stored values against live values**
5. **Document drift** — acceptable drift is ≤20 points for HN (real-time scoring), ≤10 likes for HF

### 6.3 Verification History

| Adapter | Stories Verified | Max Drift | Status |
|---------|-----------------|-----------|--------|
| GitHub | 3 repos | 0 stars | ✓ exact match |
| arXiv | 3 papers | N/A (estimated) | ✓ schema validated |
| Reddit | 3 posts | N/A (RSS) | ✓ RSS validated |
| Hacker News | 3 stories | 0 points | ✓ exact match |
| Hugging Face | 2 models | 0 likes | ✓ exact match |

### 6.4 Word-Boundary Fix (Hacker News)

A unit bug was discovered in the HN adapter: the keyword "ai" was matching inside "Great Britain" and "Guinea." Fixed with `re.search(r'\bkeyword\b', title_lower)` word-boundary matching. Verified:

| Input | Match? | Reason |
|-------|--------|--------|
| "Great Britain's AI strategy" | ❌ | "ai" not standalone word |
| "Guinea pig in ML" | ❌ | "ai" inside "Guinea" |
| "AI agents taking over" | ✅ | "ai" is standalone |
| "Deep learning advances" | ✅ | "deep learning" is standalone |

---

## 7. Trend Detection & Falsification

### 7.1 The Problem

A single day of data can look like anything. On July 8, 2026, four Reddit practitioners independently posted about:

1. **Tool-call gating** (Competence Gate)
2. **Context compression** (semantic compression for oversized sessions)
3. **Compute optimization** (CPU TTS, smallest-sufficient-model routing)
4. **Trust boundaries** (trusted LoRA adapters)

**Was this a real trend or a busy Tuesday?**

### 7.2 The Theme Scan

Athena's `theme_scan.py` answers this with a falsification-based approach:

**Design:**
- Tag entries by **theme** (tool-call, context, compute, trust), not by entry ID
- Count **NEW theme-tagged arrivals** per cron cycle (only unclassified rows)
- Idempotent: re-running on the same data returns 0 new arrivals (verified)

**Falsification rule:**
- If new theme arrivals stay ~0 per cycle over 7 days → **coincidence, kill the thesis**
- If 2–4+ new relevant items per cycle across sources → **trend, justify further investment**

### 7.3 Current Baseline (as of 2026-07-08)

| Theme | Cumulative Tags | What It Tracks |
|-------|-----------------|----------------|
| tool-call | 3 | Gating tool use on confidence/reliability |
| context | 6 | Context compression, session overflow |
| compute | 9 | Smallest-sufficient-model routing, inference cost |
| trust | 3 | Trust boundaries, vetted adapters |

**Adequacy check:** `compute` at 9 tags is the strongest signal — multiple sources (Reddit benchmarks, HF models, HN coverage) all touching the same theme. This needs 5–7 more cycles to determine if the signal is sustained or decaying.

### 7.4 Why Falsification Beats Confirmation

Most trend-detection systems confirm: *"We found 47 matches, this is a trend."*

Athena falsifies: *"We'll prove this is NOT a trend unless fresh evidence arrives each cycle."*

The burden of proof is on the data. A thesis is guilty until proven innocent across multiple independent cycles. This prevents the system from inflating one-day noise into a "discovered trend."

---

## 8. From Signal to Opportunity

### 8.1 The Original Thesis

The starting question was not *"What AI news is there?"* — it was *"Where are the use-case profit opportunities?"*

Athena is designed to answer this by:

1. **Detecting practitioner clusters** — when multiple independent actors signal the same problem, a market exists
2. **Validating cross-source convergence** — when the same signal appears on HN, Reddit, and HF, it's real
3. **Tracking trend persistence** — when the signal survives 7+ days, it's a trend worth acting on
4. **Mapping to capability gaps** — when a trend is detected, what agentic capability would solve it?

### 8.2 Example: The Competence Gate Opportunity

**Signal detected (July 8, 2026):**

| Source | Signal |
|--------|--------|
| Reddit | "Competence Gate: gating tool-use on a small model's internal confidence" |
| GitHub | ponytail (77K⭐) — "Makes your AI agent think like the laziest senior dev" |
| HN | "We charge $10k a week to delete AI-generated code" (250 pts) |
| Reddit | "What if a model could only learn what trusted LoRA adapters can express?" |

**Convergence:** Four independent sources, same underlying problem — AI agents acting without confidence gating, generating unusable code, operating without trust boundaries.

**Opportunity mapping:**

| Capability | Market Signal | Build Readiness |
|------------|--------------|-----------------|
| **Tool-call gating** | Competence Gate (Reddit), trusted LoRA (Reddit) | MVP pattern-proof built (9 tests, `discipline.py`) |
| **Context compression** | Semantic compression (Reddit) | MVP pattern-proof built |
| **Compute routing** | CPU TTS benchmark (Reddit), small model (Reddit) | Pattern identified, not yet built |
| **Trust boundaries** | Trusted LoRA (Reddit) | Pattern identified, not yet built |

### 8.3 The Resource Discipline Skill

A pattern-proof was built (`~/.hermes/profiles/leonard/skills/resource-discipline/`) demonstrating:

- **Host-agnostic ruleset** — gate tool calls based on confidence, compress context before hitting token ceilings
- **Self-test suite** — 9 tests, all passing
- **Behavioral demo** — blocked low-confidence tool calls, compressed context from 10,415 to 6,232 tokens

**Critical distinction:** This is a **pattern-proof** (tagging/detection mechanism, ~100 lines, 9 tests), not a production-ready discipline layer. It demonstrates the concept is viable; it is not yet the product.

### 8.4 From Pattern to Product

The progression is:

1. **Pattern detected** ← Athena surface signal
2. **Trend validated** ← theme_scan confirms 2–4+ arrivals/cycle over 7 days
3. **Pattern-proof built** ← `discipline.py` demonstrates concept viability
4. **Product scoped** ← real MVP with tool-call gating + context compression implementation
5. **Distribution built** ← adapter layer for host platforms (Hermes, Ollama, etc.)

**We are at step 3.** Step 2 (trend validation) is pending 5–7 more cron cycles.

---

## 9. Agentic Capability Assessment

### 9.1 What Athena Can Do Autonomously

| Capability | Status | Notes |
|------------|--------|-------|
| **Daily ingestion** | ✅ Live | 5 sources, ~98 entries/cycle |
| **Scoring** | ✅ Live | Per-source, with actual/estimated flags |
| **Summarization** | ✅ Live | Rule-based, no LLM required |
| **Trend detection** | ✅ Live | Theme-based, falsification-driven |
| **Cross-source convergence** | ✅ Live | Manual query (`python3 query.py snapshot`) |
| **Failure visibility** | ✅ Live | run_log with per-source status |
| **Soft-cap archiving** | ✅ Live | >30d or >5000 entries → archive |

### 9.2 What Athena Cannot Do (Yet)

| Capability | Status | Roadmap |
|------------|--------|---------|
| **Automated opportunity scoring** | ❌ | Requires LLM to assess market fit |
| **Competitor gap analysis** | ❌ | Manual process (Phase 6A, done once) |
| **Cross-platform adapter deployment** | ❌ | Post-pattern-proof (step 5) |
| **Natural language querying** | ❌ | `query.py` is CLI-based; LLM query layer deferred |
| **Discord monitoring** | ❌ | Requires bot token + server invite |
| **Product Hunt monitoring** | ❌ | OAuth token setup pending |
| **Techmeme/industry news** | ❌ | Low priority, HTML scraping required |

### 9.3 The LLM Question

Athena is deliberately LLM-free at ingestion time. The summarizer uses rule-based extraction to eliminate hallucination. This is a design choice:

- **Ingestion** = deterministic (no hallucination, no API cost)
- **Analysis** = LLM-assisted (future: Claude/Codex for opportunity scoring, market fit assessment)

When a local LLM becomes available, the analysis layer swaps in via `--llm` flag. The DB schema is identical.

---

## 10. Technical Specifications

### 10.1 Codebase

| Component | Lines | Purpose |
|-----------|-------|---------|
| `pipeline.py` | 334 | Pipeline orchestrator, dedup, scoring |
| `summarize.py` | 552 | Rule-based summarization engine |
| `query.py` | 459 | Snapshot, filter, theme history CLI |
| `archive.py` | 94 | Soft-cap archival logic |
| `theme_scan.py` | 128 | Trend detection, theme classification |
| `schema.sql` | 46 | SQLite schema (entries, run_log, theme_tags) |
| `adapters/__init__.py` | 17 | SourceAdapter interface |
| `adapters/github.py` | 311 | GitHub REST API adapter |
| `adapters/arxiv.py` | 527 | arXiv API adapter |
| `adapters/reddit.py` | 535 | Reddit RSS adapter |
| `adapters/hackernews.py` | 321 | HN Firebase API adapter |
| `adapters/huggingface.py` | 407 | HF API adapter |
| **Total** | **3,731** | |

### 10.2 Database Schema

```sql
-- Core entries
entries (id, source, source_id, url, title, extracted_text,
         summary, category_tags, signal_score, raw_metadata,
         first_seen, last_updated)

-- Run log (failure visibility)
run_log (id, run_time, total_fetched, total_stored,
         sources_ok, sources_failed, notes)

-- Theme tags (trend tracking)
theme_tags (entry_id, theme, first_seen_cycle)
```

### 10.3 Cron Configuration

- **Schedule:** Daily at 13:00 UTC
- **Script:** `oracle-pipeline.sh` (trampoline → `~/oracle/oracle-pipeline.sh`)
- **Chain:** pipeline → summarize → theme_scan → archive
- **Job ID:** `a992954d6233`

### 10.4 Infrastructure

- **Host:** Linux VPS (Python 3.11.15)
- **Database:** SQLite (single file, `oracle.db`)
- **Version control:** Git (5 commits, clean tree)
- **External dependencies:** None (stdlib only — `urllib`, `json`, `sqlite3`, `re`)
- **API keys required:** None (all sources use public APIs)

---

## 11. Roadmap

### 11.1 Immediate (0–14 days)

| Task | Status | Notes |
|------|--------|-------|
| Accumulate 5–7 cron cycles | In progress | Theme scan needs data |
| Validate theme trend | Pending | Falsification check |
| Build real MVP if trend confirmed | Pending | Tool-call gating + context compression |
| Expand to Product Hunt | Backlog | OAuth token pending |
| Expand to Discord | Backlog | Bot token + server invite pending |
| Expand to Techmeme | Backlog | Low priority, HTML scraping |

### 11.2 Near-term (14–90 days)

| Task | Dependency |
|------|-----------|
| LLM-assisted analysis layer | Local LLM available |
| Natural language querying | LLM layer |
| Opportunity scoring engine | LLM layer + trend validation |
| Competitor gap refresh | 90-day cadence |
| Cross-platform adapter deployment | MVP built |

### 11.3 Long-term (90+ days)

| Vision | Status |
|--------|--------|
| Full agentic research assistant | Pattern-proof exists; product pending |
| Automated startup opportunity identification | Requires LLM analysis layer |
| Multi-platform distribution (Hermes, Ollama, etc.) | Post-MVP |
| Community-sourced themes and signals | Post-90-day |

---

## 12. Appendix

### A. Git History

| Commit | Message |
|--------|---------|
| `67c002b` | Initial: Oracle AI research pipeline (adapters, pipeline, summarize, query, schema) |
| `b51be6e` | Phase 5: cron entry script, soft-cap archive, run_log zero-fetch degradation |
| `729760f` | Phase 6: theme trend-scan (B) + competitor gap research (A) |
| `6396b66` | Hacker News adapter + wire into pipeline |
| `4ba270c` | Hugging Face adapter — model releases & adoption signal |

### B. Verification Log

| Date | Adapter | Entries Verified | Drift | Result |
|------|---------|-----------------|-------|--------|
| 2026-07-08 | GitHub | 3 repos | 0 stars | ✓ |
| 2026-07-08 | Hacker News | 3 stories | 0 points | ✓ |
| 2026-07-08 | Hugging Face | 2 models | 0 likes | ✓ |
| 2026-07-08 | Word-boundary | 4 test cases | N/A | ✓ |

### C. Current Data Snapshot (2026-07-08T05:54 UTC)

- **Total entries:** 99 (20 per source, ~19 HN)
- **Confidence distribution:** 43 high, 5 medium, 2 low
- **Security flags:** 1 (dual-use tooling)
- **Theme tags:** 15 rows across 4 themes (compute=9, context=6, tool-call=3, trust=3)
- **Run log entries:** 5

### D. Resource Discipline Pattern-Proof

| File | Lines | Purpose |
|------|-------|---------|
| `discipline.py` | 106 | Host-agnostic ruleset (gate, compress) |
| `tests/test_discipline.py` | 71 | 9 tests |
| `demo.py` | 57 | Behavioral demo |
| `SKILL.md` | 59 | Skill definition |
| `adapters/hermes.md` | 24 | Hermes adapter notes |

---

*Whitepaper generated: 2026-07-08*
*Athena v0.1 — AI Research Intelligence Engine*