- adapters/__init__.py: add load_queries() + source_config() (stdlib json,
safe fallback to {} on missing/corrupt config so pipeline never crashes).
- config/queries.json: per-adapter blocks (hackernews.keywords, arxiv.categories,
reddit.subreddits, rss.feeds+keywords, github.search_terms). JSON (not
yaml) to honor Athena's dependency-free runtime; PyYAML avoided.
- hackernews: drop class AI_KEYWORDS + the DUPLICATE inline list inside
_is_ai_relevant() (the internal drift Ty flagged). Now loads self.ai_keywords
from config. Simplified matching to single substring pass (boundary variants
'ai ',' ai','ai-','-ai' approximate word-boundary; dropped the niche
'compute+tech-context' guard as not worth centralizing).
- reddit: DEFAULT_SUBREDDITS kept as fallback; __init__ prefers config.
- arxiv: DEFAULT_CATEGORIES kept as fallback; __init__ prefers config.
- rss: FEEDS + AI_KEYWORDS kept as module fallbacks; __init__ prefers config.
Keywords stay regex form (re.search) as in original.
- github: trending queries moved to config search_terms; fallback retained.
- DELETE reddit_proof.py: standalone PoC v5 at repo root, own main()+init_db()
+ direct INSERT OR REPLACE, NOT in cron, NOT imported anywhere -> dead
code. Also removes its byte-duplicate SUBREDDITS.
NOTE: fallback class constants remain intentionally (issue #7 cut #5: safe
rollout). Curation VALUES now live in one file; the constants are inert
unless config/queries.json is missing.
Verified: all adapters compile; config loads (HN 46 kw, RSS 10 feeds);
full dry-run fetches all 6 sources; grep confirms HN internal dup list gone.
Bug: Reddit was taking 120s+ when rate-limited (3 attempts × 4 subreddits × backoff).
Fix: 1) Try JSON on first subreddit; if blocked, test one RSS.
2) If both fail, return empty immediately (<3s).
3) Reduce default rate_limit 3→1s.
4) 403 now immediate fail in RSS (was retrying).
5) Max RSS retries 3→2, single 2s backoff.
Result: 120s timeout → 2.1s when blocked. Zero entries returned but no hang.
Bug: sort=stars returned the same viral repos every cycle (ponytail 77K⭐
dominated for 26 days). A 5-day-old repo with 3K stars (600/d) was
ranked below it despite having nearly 2x the growth velocity.
Fix:
- Fetch repos created in last 30 days (was 14)
- Re-sort by velocity (stars/day) instead of total stars
- Score formula: 70% velocity + 30% absolute stars
- Added 3rd query for breadth (autonomous agent OR AI tool)
- Added stars_per_day to raw_metadata for queryability
- Added velocity field to metadata
- adapters/huggingface.py: HF API adapter (models + datasets)
- Dual fetch: sort=likes (popularity) + sort=lastModified (fresh)
- AI relevance filter: pipeline_tag, library_name, tag matching
- Score: adoption (likes/downloads log-scale) + relevance (pipeline/library/tags)
- score_type: actual (real likes/downloads from HF API)
- Cross-source signal: GLM-5.2 top on both HN and HF
- Wired into ENABLED_SOURCES + verification in pipeline.py
- Live verify: DB likes match live API exactly (GLM-5.2: 3607, DeepSeek-R1: 13448)
- 99 total entries across 5 sources, all pipeline green
- adapters/hackernews.py: HN Firebase API adapter with AI keyword filtering
- Word-boundary matching to avoid substring traps (Britain/Guinea)
- Score: log(points) + log(comments), actual HN scores
- Wired into ENABLED_SOURCES + verification in pipeline.py
- Live test: 10 AI/ML stories fetched, all clean
Epictetus