refactor(adapters): centralize curation in config/queries.json (issue #7)

- adapters/__init__.py: add load_queries() + source_config() (stdlib json,
  safe fallback to {} on missing/corrupt config so pipeline never crashes).
- config/queries.json: per-adapter blocks (hackernews.keywords, arxiv.categories,
  reddit.subreddits, rss.feeds+keywords, github.search_terms). JSON (not
  yaml) to honor Athena's dependency-free runtime; PyYAML avoided.
- hackernews: drop class AI_KEYWORDS + the DUPLICATE inline list inside
  _is_ai_relevant() (the internal drift Ty flagged). Now loads self.ai_keywords
  from config. Simplified matching to single substring pass (boundary variants
  'ai ',' ai','ai-','-ai' approximate word-boundary; dropped the niche
  'compute+tech-context' guard as not worth centralizing).
- reddit: DEFAULT_SUBREDDITS kept as fallback; __init__ prefers config.
- arxiv: DEFAULT_CATEGORIES kept as fallback; __init__ prefers config.
- rss: FEEDS + AI_KEYWORDS kept as module fallbacks; __init__ prefers config.
  Keywords stay regex form (re.search) as in original.
- github: trending queries moved to config search_terms; fallback retained.
- DELETE reddit_proof.py: standalone PoC v5 at repo root, own main()+init_db()
  + direct INSERT OR REPLACE, NOT in cron, NOT imported anywhere -> dead
  code. Also removes its byte-duplicate SUBREDDITS.

NOTE: fallback class constants remain intentionally (issue #7 cut #5: safe
rollout). Curation VALUES now live in one file; the constants are inert
unless config/queries.json is missing.

Verified: all adapters compile; config loads (HN 46 kw, RSS 10 feeds);
full dry-run fetches all 6 sources; grep confirms HN internal dup list gone.
This commit is contained in:
Epictetus
2026-07-10 17:42:06 +00:00
parent 7ee1af3d7b
commit a9fbe27178
8 changed files with 115 additions and 386 deletions
+8 -7
View File
@@ -17,7 +17,7 @@ import urllib.error
import urllib.parse
from datetime import datetime, timedelta, timezone
from adapters import SourceAdapter
from adapters import SourceAdapter, source_config
class GitHubAdapter(SourceAdapter):
@@ -29,6 +29,11 @@ class GitHubAdapter(SourceAdapter):
"""Initialize with optional read-only token (5000 req/hr vs 60)."""
self.token = token or os.environ.get("GITHUB_TOKEN", "")
self.cache = {}
# Curation centralized (issue #7): trending queries from config
cfg = source_config("github")
self.search_terms = cfg.get("search_terms") or [
"ai agent", "llm OR inference OR rag", "autonomous agent OR AI tool",
]
def name(self) -> str:
return "github"
@@ -158,12 +163,8 @@ class GitHubAdapter(SourceAdapter):
cutoff = (now - timedelta(days=30)).strftime("%Y-%m-%d")
# Three queries for breadth: agents, LLM/infra, and security/tools
repos = []
for q in [
f"ai agent created:>{cutoff}",
f"llm OR inference OR rag created:>{cutoff}",
f"autonomous agent OR AI tool created:>{cutoff}",
]:
batch = self._search_repos(q, sort="stars", per_page=30)
for q in self.search_terms:
batch = self._search_repos(f"{q} created:>{cutoff}", sort="stars", per_page=30)
repos.extend(batch)
time.sleep(1) # polite spacing