Tony_tech/athena-oracle
3
0
Fork 0
Code Issues 6 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Curation logic decentralized across adapters #7

New Issue
Open
opened 2026-07-10 15:39:27 +00:00 by Leonard · 1 comment
Leonard commented 2026-07-10 15:39:27 +00:00
Copy Link
Copy Source

Category: Maintainability · Priority: MED
Status: Scoped for narrow cut (confirmed 2026-07-10). Defer adapter-manifest / result-schema to #6.

Problem (verified against code)

Curation state is scattered and already drifting:

  • hackernews.py: AI_KEYWORDS = plain substrings (~40 entries).
  • rss_feeds.py: separate AI_KEYWORDS = \b-wrapped regex — different syntax from HN's.
  • arxiv.py: DEFAULT_CATEGORIES = ["cs.AI","cs.LG","cs.CL"].
  • reddit.py: DEFAULT_SUBREDDITS = ["MachineLearning","artificial","LocalLLaMA","Startups"].
  • Internal HN drift: hackernews._is_ai_relevant() does NOT use the class AI_KEYWORDS — it hardcodes a near-duplicate "unambiguous" list inline (lines 81-91). Two keyword lists in one file, already diverged.
  • reddit_proof.py (repo root): SUBREDDITS = byte-identical duplicate of reddit.DEFAULT_SUBREDDITS.

Why it matters

Adding a theme means editing 4+ files; HN/RSS already use different keyword formats; HN alone maintains two lists. Drift compounds.

Decision (confirmed): per-adapter blocks in ONE file — NOT monolithic

Six sources have genuinely different query semantics (arXiv=categories, Reddit=subreddits, HN/RSS=keywords, GitHub=search terms). Force-flattening into one schema is wrong. Use config/queries.yaml with per-source blocks:

sources:
  hackernews:
    keywords: [...]      # regex \b form
  arxiv:
    categories: ["cs.AI","cs.LG","cs.CL"]
  reddit:
    subreddits: [...]
  rss:
    feeds: [...]        # currently in rss_feeds.FEEDS
    keywords: [...]      # unify to regex \b
  github:
    search_terms: [...]

Narrow cut (do now)

  1. Create config/queries.yaml with per-adapter blocks above.
  2. Unify keyword format to regex \b for HN + RSS (RSS already there; convert HN's substrings).
  3. Fix HN internal drift: _is_ai_relevant() references AI_KEYWORDS from YAML instead of the inline duplicate list.
  4. Retire reddit_proof.py: standalone PoC v5 at repo root, its own main() + init_db() + direct INSERT OR REPLACE INTO entries (line 223). NOT in cron, NOT imported anywhere → dead code. Delete it outright (also removes the byte-duplicate SUBREDDITS). NOTE: it bypasses SourceAdapter + PR #12's retry/failure_class — if ever revived it needs a full adapter rewrite, not a merge.
  5. Adapters load their block from YAML at init (keep constructor None-fallback to class default for safety during rollout).

Explicitly DEFERRED (to #6, not here)

  • Adapter connection manifest (name/URL/query-method/auth/secrets-in-.env) — only GitHub needs auth; mostly-empty for 5/6 sources = ceremony now.
  • Result-shape / field-mapping / nomic-embedding schema in YAML — that's #6's territory (Ty flagged as feature-creep risk; agrees).

Verification

  • python3 pipeline.py --dry-run still fetches all 6 sources post-refactor.
  • grep -rn "AI_KEYWORDS\|DEFAULT_SUBREDDITS\|DEFAULT_CATEGORIES" adapters/ → only YAML loads remain; no inline duplicates.
  • reddit_proof.py gone; git ls-files confirms removal.
  • HN + RSS keyword lists identical in YAML (single source of truth).

Related

  • Pairs with #6 (embeddings) once curation is centralized.
  • RSS feed URLs (#13) stay separate.
  • PR #12 (#1/#2/#9) lives independently; #7 refactor reads those adapters' new structure.
**Category:** Maintainability · **Priority:** MED **Status:** Scoped for narrow cut (confirmed 2026-07-10). Defer adapter-manifest / result-schema to #6. ## Problem (verified against code) Curation state is scattered and already drifting: - `hackernews.py`: `AI_KEYWORDS` = plain substrings (~40 entries). - `rss_feeds.py`: separate `AI_KEYWORDS` = `\b`-wrapped **regex** — *different syntax* from HN's. - `arxiv.py`: `DEFAULT_CATEGORIES = ["cs.AI","cs.LG","cs.CL"]`. - `reddit.py`: `DEFAULT_SUBREDDITS = ["MachineLearning","artificial","LocalLLaMA","Startups"]`. - **Internal HN drift:** `hackernews._is_ai_relevant()` does NOT use the class `AI_KEYWORDS` — it hardcodes a near-duplicate "unambiguous" list inline (lines 81-91). Two keyword lists in one file, already diverged. - `reddit_proof.py` (repo root): `SUBREDDITS` = **byte-identical duplicate** of `reddit.DEFAULT_SUBREDDITS`. ## Why it matters Adding a theme means editing 4+ files; HN/RSS already use *different* keyword formats; HN alone maintains two lists. Drift compounds. ## Decision (confirmed): per-adapter blocks in ONE file — NOT monolithic Six sources have genuinely different query semantics (arXiv=categories, Reddit=subreddits, HN/RSS=keywords, GitHub=search terms). Force-flattening into one schema is wrong. Use `config/queries.yaml` with per-source blocks: ```yaml sources: hackernews: keywords: [...] # regex \b form arxiv: categories: ["cs.AI","cs.LG","cs.CL"] reddit: subreddits: [...] rss: feeds: [...] # currently in rss_feeds.FEEDS keywords: [...] # unify to regex \b github: search_terms: [...] ``` ## Narrow cut (do now) 1. Create `config/queries.yaml` with per-adapter blocks above. 2. **Unify keyword format** to regex `\b` for HN + RSS (RSS already there; convert HN's substrings). 3. **Fix HN internal drift:** `_is_ai_relevant()` references `AI_KEYWORDS` from YAML instead of the inline duplicate list. 4. **Retire `reddit_proof.py`:** standalone PoC v5 at repo root, its own `main()` + `init_db()` + direct `INSERT OR REPLACE INTO entries` (line 223). NOT in cron, NOT imported anywhere → **dead code.** Delete it outright (also removes the byte-duplicate `SUBREDDITS`). NOTE: it bypasses SourceAdapter + PR #12's retry/`failure_class` — if ever revived it needs a full adapter rewrite, not a merge. 5. Adapters load their block from YAML at init (keep constructor `None`-fallback to class default for safety during rollout). ## Explicitly DEFERRED (to #6, not here) - Adapter connection manifest (name/URL/query-method/auth/secrets-in-.env) — only GitHub needs auth; mostly-empty for 5/6 sources = ceremony now. - Result-shape / field-mapping / nomic-embedding schema in YAML — that's #6's territory (Ty flagged as feature-creep risk; agrees). ## Verification - `python3 pipeline.py --dry-run` still fetches all 6 sources post-refactor. - `grep -rn "AI_KEYWORDS\|DEFAULT_SUBREDDITS\|DEFAULT_CATEGORIES" adapters/` → only YAML loads remain; no inline duplicates. - `reddit_proof.py` gone; `git ls-files` confirms removal. - HN + RSS keyword lists identical in YAML (single source of truth). ## Related - Pairs with #6 (embeddings) once curation is centralized. - RSS feed URLs (#13) stay separate. - PR #12 (#1/#2/#9) lives independently; #7 refactor reads those adapters' new structure.
Ty added reference main 2026-07-10 16:21:09 +00:00
Ty commented 2026-07-10 16:36:02 +00:00
Owner
Copy Link
Copy Source

This issue tracks centralizing the keyword list and categories, and adapter manifests outside of the python and into a central configuration mechanism - .yaml recommended. The final implementation needs to -

  1. Adapter (I assume this ties to a specific source feed URL?) properties -Name, desc, URL, query method (curl, xurl, etc?), Auth method, Auth info (secrets stored in a separate .env)
  2. Query content - Categories>search words/phrases.

Worth discussing (feature creep) - within the Adapter class, this file can have a section detailing how results are returned and processed into the data / infra. Is just the title retrieved? Abstract? metadata about author/date published/etc? This question really asks how data is collected, field mapped, normalized during ingestion, and indexed. within that scope, should the YAML include schema to define it. having this discussion now is important because you need to decide if there is one monolithic YAML for all, or a per-adapter YAML design - both have trade offs.

This issue tracks centralizing the keyword list and categories, and adapter manifests outside of the python and into a central configuration mechanism - .yaml recommended. The final implementation needs to - 1. Adapter (I assume this ties to a specific source feed URL?) properties -Name, desc, URL, query method (curl, xurl, etc?), Auth method, Auth info (secrets stored in a separate .env) 2. Query content - Categories>search words/phrases. Worth discussing (feature creep) - within the Adapter class, this file can have a <results> section detailing how results are returned and processed into the data / <future nomic> infra. Is just the title retrieved? Abstract? metadata about author/date published/etc? This question really asks how data is collected, field mapped, normalized during ingestion, and indexed. within that scope, should the YAML include schema to define it. having this discussion now is important because you need to decide if there is one monolithic YAML for all, or a per-adapter YAML design - both have trade offs.
Sign in to join this conversation.
main
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
HA_Researcher Ty tonyjbala (Tony Balascio)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Tony_tech/athena-oracle#7
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?