Tony_tech/athena-oracle

Fork 0

Files

T

Epictetus fd669c31bb Add README + source link to whitepaper (Gitea repo location)

2026-07-08 16:11:41 +00:00

22 KiB

Raw Blame History

Athena: AI Research Intelligence Engine

White Paper — System Architecture, Pattern Detection, and Opportunity Discovery

Source: Leonard/athena-oracle (private, Gitea — http://localhost:3000/Leonard/athena-oracle) · branch main

Abstract
Problem Statement: The AI Signal Crisis
System Overview
Source Architecture
Scoring Methodology
Verification Discipline
Trend Detection & Falsification
From Signal to Opportunity
Agentic Capability Assessment
Technical Specifications
Roadmap
Appendix

1. Abstract

Athena is an autonomous AI research intelligence engine that continuously ingests, scores, and analyzes signals from five primary sources covering the full AI landscape: code adoption (GitHub), academic research (arXiv), practitioner sentiment (Reddit), industry news (Hacker News), and model releases (Hugging Face).

The system ingests ~100 entries per daily cycle, applies rule-based scoring with per-source normalization, generates structured summaries, and runs a falsification-based trend detection engine that answers the question: "Is this pattern a real trend or a one-day coincidence?"

Built in 3,731 lines of Python, Athena runs on a single VPS, requires no external LLM, and operates autonomously via daily cron. It is designed to surface cross-source convergence signals — the kind that matter before they become obvious — and to translate structured signal into actionable opportunity intelligence.

Current state: Five sources live and verified. Theme tracking operational. Awaiting 5–7 cron cycles for trend validation.

2. Problem Statement: The AI Signal Crisis

2.1 Volume

The AI ecosystem generates thousands of new signals daily:

GitHub: 10,000+ new AI/ML repositories per week
arXiv: 300+ AI/ML papers per day across cs.AI, cs.LG, cs.CL
Reddit: 500+ AI-related posts daily across r/MachineLearning, r/LocalLLaMA, r/artificial
Hacker News: 50+ AI stories in the top 500 daily
Hugging Face: 2,000+ new model uploads daily

No human can read this volume. No single feed captures the full picture.

2.2 Fragmentation

Each source tells only a partial story:

GitHub shows what's being built (code adoption)
arXiv shows what's being researched (academic)
Reddit shows what practitioners care about (sentiment)
Hacker News shows what's being discussed (industry)
Hugging Face shows what's being downloaded (model adoption)

The signal that matters exists only across all five. A tool on GitHub with 77K stars (ponytail), discussed on Reddit by four independent practitioners, trending on Hugging Face downloads, and covered in Hacker News — that is convergence. That is signal.

2.3 The Consequence

Without structured ingestion and cross-source analysis:

Opportunities are missed because they surface on one feed you don't monitor
Trends are confused with noise because one viral day doesn't mean a movement
Decisions are backward-looking because by the time a trend is obvious, it's already priced in
Competition is blind because others who see convergence act first

3. System Overview

Athena is a five-stage pipeline that runs daily at 13:00 UTC via cron:

                    ┌─────────────┐
                    │  5 Sources   │  GitHub, arXiv, Reddit, HN, HF
                    └──────┬──────┘
                           │  fetch(limit=20)
                           ▼
                    ┌─────────────┐
              ┌─────►│  Adapters   │  Per-source AI relevance filtering
              │      └──────┬──────┘
              │             │  structured entries
              │             ▼
              │      ┌─────────────┐
              │      │   Pipeline   │  Dedup → Score → Store (SQLite)
              │      └──────┬──────┘
              │             │  entries in DB
              │             ▼
              │      ┌─────────────┐
              │      │  Summarizer  │  Rule-based extraction (no LLM)
              │      └──────┬──────┘
              │             │  summaries written
              │             ▼
              │      ┌─────────────┐
              │      │  Theme Scan  │  Classify by 4 themes, count new arrivals
              │      └──────┬──────┘
              │             │  theme tags
              │             ▼
              │      ┌─────────────┐
              └──────┤   Archive   │  Soft-cap: >30d or >5000 entries → archive
                     └─────────────┘

Key design decisions:

SQLite as source of truth — single file, no infrastructure, queryable
Rule-based summarization — no LLM required, eliminates hallucination
Score-type flagging — actual vs estimated prevents false cross-source comparison
Verification discipline — every adapter's scores spot-checked against live data
Theme-based trend detection — tags concepts, not entries; counts fresh arrivals per cycle
Soft-cap archiving — preserves history without unbounded DB growth

4. Source Architecture

4.1 The SourceAdapter Interface

Every source implements a uniform interface:

class SourceAdapter(ABC):
    @abstractmethod
    def name(self) -> str:
        """Source identifier: 'github', 'arxiv', 'reddit'."""
        pass

    @abstractmethod
    def fetch(self, query: str = "", limit: int = 20) -> list[dict]:
        """Return entries matching the unified DB schema."""
        pass

Adding a new source is one line of registration in the pipeline's adapter dictionary. Each adapter is an independent file (~300–500 lines) that:

Fetches from the source's primary API
Filters for AI relevance
Scores entries
Returns structured dictionaries matching the DB schema

4.2 Source Details

Source	API	Score Type	What It Captures	Entries/Cycle
GitHub	GitHub REST API	`actual` (stars)	Code adoption, viral tools, agent frameworks	20
arXiv	arXiv API	`estimated`	Academic research, papers, breakthroughs	20
Reddit	RSS (JSON endpoints rate-limited)	`estimated`	Practitioner sentiment, discussion clusters	20
Hacker News	Firebase API	`actual` (points)	Industry news, practitioner announcements	18–20
Hugging Face	HF API	`actual` (likes)	Model releases, adoption, downloads	20
Total			Full spectrum: code → research → discussion → news → models	~98

4.3 Why Five Sources?

The five-source coverage is deliberate — each captures a different signal layer:

Layer	Source	Example
Code adoption	GitHub	ponytail (77K⭐ in 25 days)
Academic research	arXiv	"Doomed from the Start: Early Abort of LLM Agent Episodes"
Practitioner sentiment	Reddit	"Competence Gate: gating tool-use on internal confidence"
Industry news	Hacker News	"GLM 5.2 and the coming AI margin collapse" (669 pts)
Model adoption	Hugging Face	GLM-5.2 (3,607 likes, 281K downloads)

Cross-source convergence is the system's primary value. GLM-5.2 appearing on both HN (#1 story) and HF (#1 model) is independent validation — two sources, same signal, different audiences. That is genuinely more compelling than either source alone.

5. Scoring Methodology

5.1 The Score-Type Flag

Every entry carries a score_type flag:

actual — real numbers from the source (GitHub stars, HN points, HF likes)
estimated — heuristic-based scoring where no native popularity metric exists (arXiv papers, Reddit posts)

This prevents the fundamental error of comparing an arXiv score of 5.5 against a GitHub score of 5.5 — they mean different things. The pipeline explicitly ranks within-source, not cross-source.

5.2 Per-Source Scoring

GitHub (actual)

score = log10(stars)

Capped at 10.0. A repo with 100K stars scores ~5.0; 1M stars scores ~6.0.

arXiv (estimated)

Composite of paper-specific signals:

Abstract relevance to AI/ML topics
Category matching (cs.AI, cs.LG, cs.CL)
Author count (proxy for collaborative effort)
Recency bonus

Deliberate design choice: structural metadata (author count, abstract length) was removed after review revealed it was measuring paper properties, not AI relevance. Relevance signals now dominate.

Reddit (estimated)

Composite of:

Upvote ratio (upvotes / total votes)
Comment depth (descendants)
Subreddit authority (r/MachineLearning > r/Startups)
Post type (project posts score higher than links)

Hacker News (actual)

score = log10(points) + log10(comments)

Capped at 10.0. Both metrics are real numbers from the Firebase API.

Hugging Face (actual)

Composite of:

Likes (log scale, primary adoption signal)
Downloads (log scale, secondary — can be inflated by programmatic pulls)
Pipeline tag relevance (text-generation, conversational, etc.)
Library ecosystem (transformers, diffusers)
AI-specific tag matching (agent, reasoning, alignment)
Recency bonus (newer models get slight boost)

Adoption ≠ relevance — a 1M-download fine-tune is less interesting than a 500-download novel architecture. Pipeline tag and library signals carry weight to ensure content-relevance is not drowned out by raw popularity.

6. Verification Discipline

6.1 The Rule

Every adapter's scores must be verified against the live source before the adapter is trusted. This is not optional process — it caught the arXiv structural metadata bug (H. pylori scoring error) and the confidence-length bug in the summarizer.

6.2 Process

Run the adapter — fetch entries, store to DB
Select 3 high-signal entries from the DB
Fetch the same entries from the live API
Compare stored values against live values
Document drift — acceptable drift is ≤20 points for HN (real-time scoring), ≤10 likes for HF

6.3 Verification History

Adapter	Stories Verified	Max Drift	Status
GitHub	3 repos	0 stars	✓ exact match
arXiv	3 papers	N/A (estimated)	✓ schema validated
Reddit	3 posts	N/A (RSS)	✓ RSS validated
Hacker News	3 stories	0 points	✓ exact match
Hugging Face	2 models	0 likes	✓ exact match

6.4 Word-Boundary Fix (Hacker News)

A unit bug was discovered in the HN adapter: the keyword "ai" was matching inside "Great Britain" and "Guinea." Fixed with re.search(r'\bkeyword\b', title_lower) word-boundary matching. Verified:

Input	Match?	Reason
"Great Britain's AI strategy"	❌	"ai" not standalone word
"Guinea pig in ML"	❌	"ai" inside "Guinea"
"AI agents taking over"	✅	"ai" is standalone
"Deep learning advances"	✅	"deep learning" is standalone

7. Trend Detection & Falsification

7.1 The Problem

A single day of data can look like anything. On July 8, 2026, four Reddit practitioners independently posted about:

Tool-call gating (Competence Gate)
Context compression (semantic compression for oversized sessions)
Compute optimization (CPU TTS, smallest-sufficient-model routing)
Trust boundaries (trusted LoRA adapters)

Was this a real trend or a busy Tuesday?

7.2 The Theme Scan

Athena's theme_scan.py answers this with a falsification-based approach:

Design:

Tag entries by theme (tool-call, context, compute, trust), not by entry ID
Count NEW theme-tagged arrivals per cron cycle (only unclassified rows)
Idempotent: re-running on the same data returns 0 new arrivals (verified)

Falsification rule:

If new theme arrivals stay ~0 per cycle over 7 days → coincidence, kill the thesis
If 2–4+ new relevant items per cycle across sources → trend, justify further investment

7.3 Current Baseline (as of 2026-07-08)

Theme	Cumulative Tags	What It Tracks
tool-call	3	Gating tool use on confidence/reliability
context	6	Context compression, session overflow
compute	9	Smallest-sufficient-model routing, inference cost
trust	3	Trust boundaries, vetted adapters

Adequacy check: compute at 9 tags is the strongest signal — multiple sources (Reddit benchmarks, HF models, HN coverage) all touching the same theme. This needs 5–7 more cycles to determine if the signal is sustained or decaying.

7.4 Why Falsification Beats Confirmation

Most trend-detection systems confirm: "We found 47 matches, this is a trend."

Athena falsifies: "We'll prove this is NOT a trend unless fresh evidence arrives each cycle."

The burden of proof is on the data. A thesis is guilty until proven innocent across multiple independent cycles. This prevents the system from inflating one-day noise into a "discovered trend."

8. From Signal to Opportunity

8.1 The Original Thesis

The starting question was not "What AI news is there?" — it was "Where are the use-case profit opportunities?"

Athena is designed to answer this by:

Detecting practitioner clusters — when multiple independent actors signal the same problem, a market exists
Validating cross-source convergence — when the same signal appears on HN, Reddit, and HF, it's real
Tracking trend persistence — when the signal survives 7+ days, it's a trend worth acting on
Mapping to capability gaps — when a trend is detected, what agentic capability would solve it?

8.2 Example: The Competence Gate Opportunity

Signal detected (July 8, 2026):

Source	Signal
Reddit	"Competence Gate: gating tool-use on a small model's internal confidence"
GitHub	ponytail (77K⭐) — "Makes your AI agent think like the laziest senior dev"
HN	"We charge $10k a week to delete AI-generated code" (250 pts)
Reddit	"What if a model could only learn what trusted LoRA adapters can express?"

Convergence: Four independent sources, same underlying problem — AI agents acting without confidence gating, generating unusable code, operating without trust boundaries.

Opportunity mapping:

Capability	Market Signal	Build Readiness
Tool-call gating	Competence Gate (Reddit), trusted LoRA (Reddit)	MVP pattern-proof built (9 tests, `discipline.py`)
Context compression	Semantic compression (Reddit)	MVP pattern-proof built
Compute routing	CPU TTS benchmark (Reddit), small model (Reddit)	Pattern identified, not yet built
Trust boundaries	Trusted LoRA (Reddit)	Pattern identified, not yet built

8.3 The Resource Discipline Skill

A pattern-proof was built (~/.hermes/profiles/leonard/skills/resource-discipline/) demonstrating:

Host-agnostic ruleset — gate tool calls based on confidence, compress context before hitting token ceilings
Self-test suite — 9 tests, all passing
Behavioral demo — blocked low-confidence tool calls, compressed context from 10,415 to 6,232 tokens

Critical distinction: This is a pattern-proof (tagging/detection mechanism, ~100 lines, 9 tests), not a production-ready discipline layer. It demonstrates the concept is viable; it is not yet the product.

8.4 From Pattern to Product

The progression is:

Pattern detected ← Athena surface signal
Trend validated ← theme_scan confirms 2–4+ arrivals/cycle over 7 days
Pattern-proof built ← discipline.py demonstrates concept viability
Product scoped ← real MVP with tool-call gating + context compression implementation
Distribution built ← adapter layer for host platforms (Hermes, Ollama, etc.)

We are at step 3. Step 2 (trend validation) is pending 5–7 more cron cycles.

9. Agentic Capability Assessment

9.1 What Athena Can Do Autonomously

Capability	Status	Notes
Daily ingestion	✅ Live	5 sources, ~98 entries/cycle
Scoring	✅ Live	Per-source, with actual/estimated flags
Summarization	✅ Live	Rule-based, no LLM required
Trend detection	✅ Live	Theme-based, falsification-driven
Cross-source convergence	✅ Live	Manual query (`python3 query.py snapshot`)
Failure visibility	✅ Live	run_log with per-source status
Soft-cap archiving	✅ Live	>30d or >5000 entries → archive

9.2 What Athena Cannot Do (Yet)

Capability	Status	Roadmap
Automated opportunity scoring	❌	Requires LLM to assess market fit
Competitor gap analysis	❌	Manual process (Phase 6A, done once)
Cross-platform adapter deployment	❌	Post-pattern-proof (step 5)
Natural language querying	❌	`query.py` is CLI-based; LLM query layer deferred
Discord monitoring	❌	Requires bot token + server invite
Product Hunt monitoring	❌	OAuth token setup pending
Techmeme/industry news	❌	Low priority, HTML scraping required

9.3 The LLM Question

Athena is deliberately LLM-free at ingestion time. The summarizer uses rule-based extraction to eliminate hallucination. This is a design choice:

Ingestion = deterministic (no hallucination, no API cost)
Analysis = LLM-assisted (future: Claude/Codex for opportunity scoring, market fit assessment)

When a local LLM becomes available, the analysis layer swaps in via --llm flag. The DB schema is identical.

10. Technical Specifications

10.1 Codebase

Component	Lines	Purpose
`pipeline.py`	334	Pipeline orchestrator, dedup, scoring
`summarize.py`	552	Rule-based summarization engine
`query.py`	459	Snapshot, filter, theme history CLI
`archive.py`	94	Soft-cap archival logic
`theme_scan.py`	128	Trend detection, theme classification
`schema.sql`	46	SQLite schema (entries, run_log, theme_tags)
`adapters/__init__.py`	17	SourceAdapter interface
`adapters/github.py`	311	GitHub REST API adapter
`adapters/arxiv.py`	527	arXiv API adapter
`adapters/reddit.py`	535	Reddit RSS adapter
`adapters/hackernews.py`	321	HN Firebase API adapter
`adapters/huggingface.py`	407	HF API adapter
Total	3,731

10.2 Database Schema

-- Core entries
entries (id, source, source_id, url, title, extracted_text,
         summary, category_tags, signal_score, raw_metadata,
         first_seen, last_updated)

-- Run log (failure visibility)
run_log (id, run_time, total_fetched, total_stored,
         sources_ok, sources_failed, notes)

-- Theme tags (trend tracking)
theme_tags (entry_id, theme, first_seen_cycle)

10.3 Cron Configuration

Schedule: Daily at 13:00 UTC
Script: oracle-pipeline.sh (trampoline → ~/oracle/oracle-pipeline.sh)
Chain: pipeline → summarize → theme_scan → archive
Job ID: a992954d6233

10.4 Infrastructure

Host: Linux VPS (Python 3.11.15)
Database: SQLite (single file, oracle.db)
Version control: Git (5 commits, clean tree)
External dependencies: None (stdlib only — urllib, json, sqlite3, re)
API keys required: None (all sources use public APIs)

11. Roadmap

11.1 Immediate (0–14 days)

Task	Status	Notes
Accumulate 5–7 cron cycles	In progress	Theme scan needs data
Validate theme trend	Pending	Falsification check
Build real MVP if trend confirmed	Pending	Tool-call gating + context compression
Expand to Product Hunt	Backlog	OAuth token pending
Expand to Discord	Backlog	Bot token + server invite pending
Expand to Techmeme	Backlog	Low priority, HTML scraping

11.2 Near-term (14–90 days)

Task	Dependency
LLM-assisted analysis layer	Local LLM available
Natural language querying	LLM layer
Opportunity scoring engine	LLM layer + trend validation
Competitor gap refresh	90-day cadence
Cross-platform adapter deployment	MVP built

11.3 Long-term (90+ days)

Vision	Status
Full agentic research assistant	Pattern-proof exists; product pending
Automated startup opportunity identification	Requires LLM analysis layer
Multi-platform distribution (Hermes, Ollama, etc.)	Post-MVP
Community-sourced themes and signals	Post-90-day

12. Appendix

A. Git History

Commit	Message
`67c002b`	Initial: Oracle AI research pipeline (adapters, pipeline, summarize, query, schema)
`b51be6e`	Phase 5: cron entry script, soft-cap archive, run_log zero-fetch degradation
`729760f`	Phase 6: theme trend-scan (B) + competitor gap research (A)
`6396b66`	Hacker News adapter + wire into pipeline
`4ba270c`	Hugging Face adapter — model releases & adoption signal

B. Verification Log

Date	Adapter	Entries Verified	Drift	Result
2026-07-08	GitHub	3 repos	0 stars	✓
2026-07-08	Hacker News	3 stories	0 points	✓
2026-07-08	Hugging Face	2 models	0 likes	✓
2026-07-08	Word-boundary	4 test cases	N/A	✓

C. Current Data Snapshot (2026-07-08T05:54 UTC)

Total entries: 99 (20 per source, ~19 HN)
Confidence distribution: 43 high, 5 medium, 2 low
Security flags: 1 (dual-use tooling)
Theme tags: 15 rows across 4 themes (compute=9, context=6, tool-call=3, trust=3)
Run log entries: 5

D. Resource Discipline Pattern-Proof

File	Lines	Purpose
`discipline.py`	106	Host-agnostic ruleset (gate, compress)
`tests/test_discipline.py`	71	9 tests
`demo.py`	57	Behavioral demo
`SKILL.md`	59	Skill definition
`adapters/hermes.md`	24	Hermes adapter notes

Whitepaper generated: 2026-07-08 Athena v0.1 — AI Research Intelligence Engine

22 KiB Raw Blame History Unescape Escape