 Epictetus
|
641d531d88
|
Sprint 3: Anti-bot retrieval layer + metrics module
Anti-bot changes (all 6 adapters):
- Browser-grade User-Agent rotation (Chrome/Firefox on Linux/Windows)
- Shared browser_headers() with Accept, Accept-Language, DNT
- Session-consistent UA fingerprint (picked once, not per-request)
- jitter_sleep() replaces fixed time.sleep() on all adapters
- Exponential backoff on 429/503 already on reddit, now consistent
New shared module:
- adapters/__init__.py: browser_user_agent(), browser_headers(), jitter_sleep()
- adapters/_http.py: HTTPClient class for future browser-mode adapters
Metrics module (from Sprint 2 carry):
- oracle/metrics.py: MetricsRun for log_adapter/log_verdicts/log_scores
- oracle/weekly.py: SYSTEM HEALTH section wired to adapter_health
- oracle/cli.py: metrics subparser with --adapters/--publish/--scores/--alerts
Before: bot signatures like 'ai-oracle/0.1', 'python:athena:v0.1'
After: 'Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0'
|
2026-07-22 14:26:42 +00:00 |
|
|
Epictetus
|
2c6701f5a3
|
Add RSS Feeds adapter (6th source) — commercial AI news from 10 feeds
Bug fix: date parsing was ISO-only (RFC 2822 feeds filtered out).
Added email.utils.parsedate_to_datetime() fallback for RSS dates.
10 feeds: TechCrunch AI, VentureBeat AI, The Verge AI, AI News,
The Decoder, MIT Tech Review AI, OpenAI Blog, Anthropic, Google AI, Meta AI.
3 dead feeds (Anthropic/Google/Meta 404), 7 working.
Score: authority (primary blogs 1.5x, industry 1.3x) × recency decay (48h half-life)
Age cutoff: 14 days. Score type: estimated.
Wired into pipeline.py ENABLED_SOURCES. Added RSS URL verification.
3/3 spot-checks passed (all URLs reachable, HTTP 200).
Commit: 62031be→c24a89f
|
2026-07-08 15:27:59 +00:00 |
|