Commit Graph

3 Commits

Author SHA1 Message Date
Epictetus a9fbe27178 refactor(adapters): centralize curation in config/queries.json (issue #7)
- adapters/__init__.py: add load_queries() + source_config() (stdlib json,
  safe fallback to {} on missing/corrupt config so pipeline never crashes).
- config/queries.json: per-adapter blocks (hackernews.keywords, arxiv.categories,
  reddit.subreddits, rss.feeds+keywords, github.search_terms). JSON (not
  yaml) to honor Athena's dependency-free runtime; PyYAML avoided.
- hackernews: drop class AI_KEYWORDS + the DUPLICATE inline list inside
  _is_ai_relevant() (the internal drift Ty flagged). Now loads self.ai_keywords
  from config. Simplified matching to single substring pass (boundary variants
  'ai ',' ai','ai-','-ai' approximate word-boundary; dropped the niche
  'compute+tech-context' guard as not worth centralizing).
- reddit: DEFAULT_SUBREDDITS kept as fallback; __init__ prefers config.
- arxiv: DEFAULT_CATEGORIES kept as fallback; __init__ prefers config.
- rss: FEEDS + AI_KEYWORDS kept as module fallbacks; __init__ prefers config.
  Keywords stay regex form (re.search) as in original.
- github: trending queries moved to config search_terms; fallback retained.
- DELETE reddit_proof.py: standalone PoC v5 at repo root, own main()+init_db()
  + direct INSERT OR REPLACE, NOT in cron, NOT imported anywhere -> dead
  code. Also removes its byte-duplicate SUBREDDITS.

NOTE: fallback class constants remain intentionally (issue #7 cut #5: safe
rollout). Curation VALUES now live in one file; the constants are inert
unless config/queries.json is missing.

Verified: all adapters compile; config loads (HN 46 kw, RSS 10 feeds);
full dry-run fetches all 6 sources; grep confirms HN internal dup list gone.
2026-07-10 17:42:06 +00:00
Epictetus df5226859e Fix Reddit adapter: fast-bail when blocked (403/429)
Bug: Reddit was taking 120s+ when rate-limited (3 attempts × 4 subreddits × backoff).
Fix: 1) Try JSON on first subreddit; if blocked, test one RSS.
     2) If both fail, return empty immediately (<3s).
     3) Reduce default rate_limit 3→1s.
     4) 403 now immediate fail in RSS (was retrying).
     5) Max RSS retries 3→2, single 2s backoff.

Result: 120s timeout → 2.1s when blocked. Zero entries returned but no hang.
2026-07-08 15:36:50 +00:00
Epictetus 67c002b665 Initial commit: Oracle AI research pipeline (adapters, pipeline, summarize, query)
Source-controlled baseline before Phase 5 cron. Excludes oracle.db,
logs/, and __pycache__ via .gitignore. Pipeline verified running
clean end-to-end (run_log write confirmed before conn.close()).
2026-07-08 04:03:36 +00:00