refactor(adapters): centralize curation in config/queries.json (issue #7)
- adapters/__init__.py: add load_queries() + source_config() (stdlib json,
safe fallback to {} on missing/corrupt config so pipeline never crashes).
- config/queries.json: per-adapter blocks (hackernews.keywords, arxiv.categories,
reddit.subreddits, rss.feeds+keywords, github.search_terms). JSON (not
yaml) to honor Athena's dependency-free runtime; PyYAML avoided.
- hackernews: drop class AI_KEYWORDS + the DUPLICATE inline list inside
_is_ai_relevant() (the internal drift Ty flagged). Now loads self.ai_keywords
from config. Simplified matching to single substring pass (boundary variants
'ai ',' ai','ai-','-ai' approximate word-boundary; dropped the niche
'compute+tech-context' guard as not worth centralizing).
- reddit: DEFAULT_SUBREDDITS kept as fallback; __init__ prefers config.
- arxiv: DEFAULT_CATEGORIES kept as fallback; __init__ prefers config.
- rss: FEEDS + AI_KEYWORDS kept as module fallbacks; __init__ prefers config.
Keywords stay regex form (re.search) as in original.
- github: trending queries moved to config search_terms; fallback retained.
- DELETE reddit_proof.py: standalone PoC v5 at repo root, own main()+init_db()
+ direct INSERT OR REPLACE, NOT in cron, NOT imported anywhere -> dead
code. Also removes its byte-duplicate SUBREDDITS.
NOTE: fallback class constants remain intentionally (issue #7 cut #5: safe
rollout). Curation VALUES now live in one file; the constants are inert
unless config/queries.json is missing.
Verified: all adapters compile; config loads (HN 46 kw, RSS 10 feeds);
full dry-run fetches all 6 sources; grep confirms HN internal dup list gone.
This commit is contained in:
+15
-3
@@ -24,7 +24,7 @@ import feedparser
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from email.utils import parsedate_to_datetime
|
||||
|
||||
from adapters import SourceAdapter
|
||||
from adapters import SourceAdapter, source_config
|
||||
|
||||
|
||||
# Curated feed list — AI-focused, reliable, diverse publishers.
|
||||
@@ -83,6 +83,18 @@ AI_KEYWORDS = [
|
||||
class RSSFeedsAdapter(SourceAdapter):
|
||||
"""RSS feed aggregator for commercial AI news."""
|
||||
|
||||
# Module-level fallbacks (used only if config/queries.json is missing)
|
||||
FEEDS = [
|
||||
("rss:techcrunch", "TechCrunch AI", "https://techcrunch.com/category/artificial-intelligence/feed/"),
|
||||
]
|
||||
AI_KEYWORDS = [r"\bai\b"]
|
||||
|
||||
def __init__(self):
|
||||
# Curation centralized (issue #7): config wins, fallbacks otherwise
|
||||
cfg = source_config("rss")
|
||||
self.feeds = cfg.get("feeds") or list(self.FEEDS)
|
||||
self.ai_keywords = cfg.get("keywords") or list(self.AI_KEYWORDS)
|
||||
|
||||
def name(self) -> str:
|
||||
return "rss"
|
||||
|
||||
@@ -92,7 +104,7 @@ class RSSFeedsAdapter(SourceAdapter):
|
||||
tag_text = " ".join(tags).lower()
|
||||
combined = text + " " + tag_text
|
||||
|
||||
for pattern in AI_KEYWORDS:
|
||||
for pattern in self.ai_keywords:
|
||||
if re.search(pattern, combined):
|
||||
return True
|
||||
return False
|
||||
@@ -147,7 +159,7 @@ class RSSFeedsAdapter(SourceAdapter):
|
||||
all_entries = []
|
||||
feed_failures = []
|
||||
|
||||
for source_key, label, url in FEEDS:
|
||||
for source_key, label, url in self.feeds:
|
||||
try:
|
||||
d = feedparser.parse(url)
|
||||
if d.status not in (200, 301, 302, 307, 308) or not d.entries:
|
||||
|
||||
Reference in New Issue
Block a user