athena-oracle/README.md

Athena Oracle — AI Research Intelligence Pipeline

Multi-source AI news aggregation, scoring, and multi-variant edition rendering. One pipeline → multiple audience-specific editions.

Live site: https://ai-oracle.com (rendered from this pipeline)

Architecture

┌─────────────────────────────────────────────────────┐
│  adapters/                                          │
│  arxiv  ·  github  ·  hackernews  ·  reddit        │
│  huggingface  ·  rss_feeds                          │
└───────────────┬─────────────────────────────────────┘
                │ fetch(limit, timeout=10s)
                ▼
┌─────────────────────────────────────────────────────┐
│  oracle/ — Core Package                             │
│                                                     │
│  scoring.py    Pure-rule component scoring (0-10)   │
│  dedup.py      Content-hash dedup + verdict engine  │
│  variants.py   Multi-variant edition engine         │
│  render.py     HTML/JSON variant renderer           │
│  summarize.py  Source-aware text summarization      │
│  recency.py    Age-based freshness gate             │
│  themes.py     Theme-based trend tracking           │
│  archive.py    Soft-cap entry archival              │
│  db.py         Schema management + migrations       │
│  config.py     Centralized configuration            │
│  cli.py        Unified CLI (python -m oracle)       │
└───────────────┬─────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────┐
│  oracle.db  — SQLite (784+ entries)                 │
│  Columns: signal_score · final_score · content_hash │
│          · verdict · source_tier · summary           │
└─────────────────────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────┐
│  variants/ — Edition Configs (YAML)                 │
│                                                     │
│  default.yaml  Full feed, clickability-ranked       │
│  research.yaml arXiv + HF papers, signal-ranked     │
│  devops.yaml   Shipping tools, 7-day window         │
│  brief.yaml    PUBLISH verdict only, top 8          │
└─────────────────────────────────────────────────────┘

Quick Start

# Run the ingestion pipeline
python -m oracle ingest

# Generate summaries for pending entries
python -m oracle summarize

# Render all variant editions (preview)
python -m oracle render --all --dry-run

# Render a single variant
python -m oracle render --variant research

# Deploy to production
python -m oracle render --all --webroot /var/www/html

# Check system health
python -m oracle health

CLI Commands

Command	Description
ingest	Fetch from adapters, score, store to oracle.db
summarize	Generate summaries for entries without one
query top N	Top N entries by signal score
query search "text"	Keyword search across titles/summaries
query recent --hours 24	Recent entries
render	Render variant editions (HTML + JSON)
archive	Soft-cap archival of old entries
themes	Theme-based trend tracking
dedup	Content-hash dedup + verdict management
health	System health check

Ingest Options

python -m oracle ingest --sources github,arxiv    # Specific sources only
python -m oracle ingest --limit 5                 # 5 entries per source
python -m oracle ingest --dry-run                 # Fetch but don't store

Render Options

python -m oracle render --list                    # Show available variants
python -m oracle render --variant brief            # Single variant
python -m oracle render --all                     # All variants
python -m oracle render --all --dry-run            # Preview mode
python -m oracle render --all --webroot /var/www   # Production deploy

Scoring Engine

Pure-rule component scoring — no embeddings or LLM required.

Component	Weight	Description
Shipping	20%	Code releases, benchmarks, working demos
Utility	20%	Practical tools, frameworks, integrations
Replication	25%	Reproducible research, open datasets
Enthusiast	20%	Community buzz, notable figures
Novelty	15%	First-of-its-kind, paradigm shifts

Hype Penalty: Caps at 45% to prevent buzzwords from dominating.

Signal Score: 0–10 scale (per-adapter normalization → final composite).

Source Tiers (World Monitor Pattern)

Tier	Sources	Rationale
Tier 1 (PRIMARY)	arxiv, github, huggingface	Peer-reviewed research, official code releases, model registry
Tier 2 (SECONDARY)	rss, hackernews	Curated tech media, curated community
Tier 3 (TERTIARY)	reddit	User-generated discussion

Composite Verdicts

Entries are classified based on signal score + age:

Verdict	Minimum Score	Maximum Age	Meaning
PUBLISH	≥ 6.0	≤ 48h	High-signal, fresh — front page material
WATCH	≥ 4.0	≤ 168h	Solid signal — worth tracking
ARCHIVE	≥ 2.0	≤ 720h	Historical value — keep for reference
DROP	any	> 720h	Stale — exclude from active feeds

Content-Hash Dedup

SHA-256 content hashing (first 16 hex chars) for cross-source duplicate detection:

• Normalized whitespace before hashing
• Applied atomically at ingest time via adapters/_store.py
• Prevents the same story from appearing multiple times across sources

Variant Editions

Each variant is a YAML config defining:

• Filters: verdicts, sources, min_score, max_age_h, max_items
• Ranking: clickability | signal_score | verdict_priority | freshness (with half-life decay)
• Display: theme, accent color, logo, show/hide score/tier/verdict badges

Create a new edition by adding a YAML to variants/ — no code changes needed.

Database

SQLite (oracle.db) with 784+ entries across 6 sources. Schema includes:

• Entry metadata (title, url, source, extracted_text, raw_metadata)
• Scoring (signal_score, component scores, final_score, actionability_score)
• Dedup/verdict (content_hash, verdict, source_tier)
• Summarization (summary JSON with one_liner, key_points, implications)
• Categorization (category_tags, bucket, narrative_id, topic_id)

Sprint Log

Sprint 0 — Foundation (2026-07-22) ✅

• Package restructure: oracle/ + python -m oracle CLI
• Per-adapter timeout (10s) + threading fallback
• Source confidence tiers (3-tier system)
• Content-hash dedup (SHA-256[:16])
• Composite verdicts (PUBLISH/WATCH/ARCHIVE/DROP)
• Wired into pipeline: atomic hash/tier/verdict at insert time
• Consolidated 12 root scripts → thin wrappers + oracle/ package

Sprint 1 — Multi-variant Editions (2026-07-22) ✅

• Variant engine: oracle/variants.py
• 4 default editions: default, research, devops, brief
• Variant-aware HTML/JSON renderer with theme support
• CLI: render --variant, --all, --list, --webroot
• Verified: all 4 variants render correctly with proper filtering

Dependencies

• Python 3.10+
• PyYAML (pip install pyyaml)
• Trafilatura (for text extraction in adapters)
• Feedparser (RSS feeds)

Backward Compatibility

Old scripts (pipeline.py, summarize.py, query.py, etc.) are thin wrappers that delegate to the unified CLI. Existing cron jobs calling python3 pipeline.py continue to work without changes.

License

AGPL v3 (see LICENSE for details)