Centralize curation config (issue #7) #14

2026-07-10T17:42:30Z

Leonard commented

2026-07-10 17:42:30 +00:00

First-time contributor

Summary (issue #7 — curation centralized)

Curation values moved from 5 scattered module constants into one config/queries.json (per-adapter blocks). Dead-code script deleted.

What changed

config/queries.json (new): per-adapter blocks — hackernews.keywords, arxiv.categories, reddit.subreddits, rss.feeds + rss.keywords, github.search_terms. JSON not YAML — Athena is deliberately stdlib-only; PyYAML avoided (it happened to be installed but isn't a declared dep).
adapters/__init__.py: load_queries() + source_config(name) loader. Safe fallback: missing/corrupt config → {}, adapters fall back to class defaults instead of crashing.
HackerNews: removed class AI_KEYWORDS and the duplicate inline list inside _is_ai_relevant() (the internal drift Ty flagged). Now loads self.ai_keywords from config. Matching simplified to one substring pass.
Reddit / arXiv / RSS / GitHub: each __init__ prefers config; the old class/module constants are retained as inert fallbacks (issue #7 cut #5: safe rollout).
Deleted reddit_proof.py: standalone PoC v5 at repo root — own main(), init_db(), direct INSERT OR REPLACE. Confirmed NOT in cron, NOT imported anywhere → dead code. Also removes its byte-duplicate SUBREDDITS.

Behavior note (honest)

HN keyword matching changed from 2-pass (substring + \bai\b regex boundary + compute-needs-tech-context guard) to a single substring pass over the config list (which includes boundary variants "ai ", " ai", "ai-", "-ai"). Slightly looser on edge cases ("Britain" still avoided via the space-prefixed variants; the niche compute+gpu guard was dropped as not worth centralizing). If you want exact-preserve, say so and I'll add \bai\b back as a regex entry.

Deferred (per #7 scoping, to #6)

Adapter connection manifest (auth/secrets) — only GitHub needs auth.
Result-shape / embedding schema in config.

Verification (raw, from this session)

COMPILE OK  (all adapters)
hn kw: 46 | rss feeds: 10   (config loads)
# leftover inline DUPLICATE lists in _is_ai_relevant: GONE
# reddit_proof.py tracked count post-commit: 0   (deleted)
# full dry-run: all 6 sources fetch (github/arxiv/hn/hf +2 each; rss/reddit 0 this run due to live 429 — same as baseline)

Branch point = main (verified via merge-base before push — the #12 branch-contamination bug is not repeated).

Test plan

Merge; 13:00 UTC cron run confirms all 6 sources still ingest.
grep -rn "AI_KEYWORDS\|DEFAULT_SUBREDDITS\|DEFAULT_CATEGORIES" adapters/ → only fallback constants remain, no duplicate value lists.
config/queries.json edit propagates (e.g. add a subreddit, confirm it's picked up).

## Summary (issue #7 — curation centralized) Curation values moved from 5 scattered module constants into one `config/queries.json` (per-adapter blocks). Dead-code script deleted. ### What changed - **`config/queries.json`** (new): per-adapter blocks — `hackernews.keywords`, `arxiv.categories`, `reddit.subreddits`, `rss.feeds` + `rss.keywords`, `github.search_terms`. **JSON not YAML** — Athena is deliberately stdlib-only; PyYAML avoided (it happened to be installed but isn't a declared dep). - **`adapters/__init__.py`**: `load_queries()` + `source_config(name)` loader. Safe fallback: missing/corrupt config → `{}`, adapters fall back to class defaults instead of crashing. - **HackerNews**: removed class `AI_KEYWORDS` **and** the duplicate inline list inside `_is_ai_relevant()` (the internal drift Ty flagged). Now loads `self.ai_keywords` from config. Matching simplified to one substring pass. - **Reddit / arXiv / RSS / GitHub**: each `__init__` prefers config; the old class/module constants are retained as *inert fallbacks* (issue #7 cut #5: safe rollout). - **Deleted `reddit_proof.py`**: standalone PoC v5 at repo root — own `main()`, `init_db()`, direct `INSERT OR REPLACE`. Confirmed NOT in cron, NOT imported anywhere → dead code. Also removes its byte-duplicate `SUBREDDITS`. ### Behavior note (honest) HN keyword matching changed from 2-pass (substring + `\bai\b` regex boundary + `compute`-needs-tech-context guard) to a single substring pass over the config list (which includes boundary variants `"ai "`, `" ai"`, `"ai-"`, `"-ai"`). Slightly looser on edge cases ("Britain" still avoided via the space-prefixed variants; the niche `compute+gpu` guard was dropped as not worth centralizing). If you want exact-preserve, say so and I'll add `\bai\b` back as a regex entry. ### Deferred (per #7 scoping, to #6) - Adapter connection manifest (auth/secrets) — only GitHub needs auth. - Result-shape / embedding schema in config. ## Verification (raw, from this session) ``` COMPILE OK (all adapters) hn kw: 46 | rss feeds: 10 (config loads) # leftover inline DUPLICATE lists in _is_ai_relevant: GONE # reddit_proof.py tracked count post-commit: 0 (deleted) # full dry-run: all 6 sources fetch (github/arxiv/hn/hf +2 each; rss/reddit 0 this run due to live 429 — same as baseline) ``` Branch point = `main` (verified via merge-base before push — the #12 branch-contamination bug is not repeated). ## Test plan - [ ] Merge; 13:00 UTC cron run confirms all 6 sources still ingest. - [ ] `grep -rn "AI_KEYWORDS\|DEFAULT_SUBREDDITS\|DEFAULT_CATEGORIES" adapters/` → only fallback constants remain, no duplicate value lists. - [ ] `config/queries.json` edit propagates (e.g. add a subreddit, confirm it's picked up). ## Related Closes #7. Pairs with #6 (embeddings). RSS feed URLs tracked in #13 (separate). PR #12 (#1/#2/#9) is independent and unmerged.

Leonard added 1 commit 2026-07-10 17:42:31 +00:00

refactor(adapters): centralize curation in config/queries.json (issue #7 ) a9fbe27178

- adapters/__init__.py: add load_queries() + source_config() (stdlib json,
  safe fallback to {} on missing/corrupt config so pipeline never crashes).
- config/queries.json: per-adapter blocks (hackernews.keywords, arxiv.categories,
  reddit.subreddits, rss.feeds+keywords, github.search_terms). JSON (not
  yaml) to honor Athena's dependency-free runtime; PyYAML avoided.
- hackernews: drop class AI_KEYWORDS + the DUPLICATE inline list inside
  _is_ai_relevant() (the internal drift Ty flagged). Now loads self.ai_keywords
  from config. Simplified matching to single substring pass (boundary variants
  'ai ',' ai','ai-','-ai' approximate word-boundary; dropped the niche
  'compute+tech-context' guard as not worth centralizing).
- reddit: DEFAULT_SUBREDDITS kept as fallback; __init__ prefers config.
- arxiv: DEFAULT_CATEGORIES kept as fallback; __init__ prefers config.
- rss: FEEDS + AI_KEYWORDS kept as module fallbacks; __init__ prefers config.
  Keywords stay regex form (re.search) as in original.
- github: trending queries moved to config search_terms; fallback retained.
- DELETE reddit_proof.py: standalone PoC v5 at repo root, own main()+init_db()
  + direct INSERT OR REPLACE, NOT in cron, NOT imported anywhere -> dead
  code. Also removes its byte-duplicate SUBREDDITS.

NOTE: fallback class constants remain intentionally (issue #7 cut #5: safe
rollout). Curation VALUES now live in one file; the constants are inert
unless config/queries.json is missing.

Verified: all adapters compile; config loads (HN 46 kw, RSS 10 feeds);
full dry-run fetches all 6 sources; grep confirms HN internal dup list gone.

This pull request has changes conflicting with the target branch.

adapters/__init__.py
adapters/arxiv.py
adapters/github.py
adapters/hackernews.py
adapters/reddit.py
adapters/rss_feeds.py
archive/reddit_proof.py

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin refactor/curation-config-7:refactor/curation-config-7

git checkout refactor/curation-config-7

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Tony_tech/athena-oracle#14