Sprint 3: Anti-bot retrieval layer + metrics module

Anti-bot changes (all 6 adapters):
- Browser-grade User-Agent rotation (Chrome/Firefox on Linux/Windows)
- Shared browser_headers() with Accept, Accept-Language, DNT
- Session-consistent UA fingerprint (picked once, not per-request)
- jitter_sleep() replaces fixed time.sleep() on all adapters
- Exponential backoff on 429/503 already on reddit, now consistent

New shared module:
- adapters/__init__.py: browser_user_agent(), browser_headers(), jitter_sleep()
- adapters/_http.py: HTTPClient class for future browser-mode adapters

Metrics module (from Sprint 2 carry):
- oracle/metrics.py: MetricsRun for log_adapter/log_verdicts/log_scores
- oracle/weekly.py: SYSTEM HEALTH section wired to adapter_health
- oracle/cli.py: metrics subparser with --adapters/--publish/--scores/--alerts

Before: bot signatures like 'ai-oracle/0.1', 'python:athena:v0.1'
After: 'Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0'
This commit is contained in:
Epictetus
2026-07-22 14:26:42 +00:00
parent 3ec955e143
commit 641d531d88
11 changed files with 694 additions and 21 deletions
+4 -4
View File
@@ -26,7 +26,7 @@ import xml.etree.ElementTree as ET
from datetime import datetime, timezone
from html import unescape
from adapters import SourceAdapter
from adapters import SourceAdapter, browser_user_agent, jitter_sleep
from adapters._store import true_first_seen, upsert_entries
@@ -62,7 +62,7 @@ class RedditAdapter(SourceAdapter):
"""
self.subreddits = subreddits or self.DEFAULT_SUBREDDITS
self.rate_limit = rate_limit
self.user_agent = user_agent or "python:ai-oracle:v0.1 (by tony_tech)"
self.user_agent = user_agent or browser_user_agent()
def name(self) -> str:
return "reddit"
@@ -212,7 +212,7 @@ class RedditAdapter(SourceAdapter):
if e.code in (429, 403, 404):
return [] # JSON endpoint blocked, fall back to RSS
if attempt < 1:
time.sleep(5)
jitter_sleep(5)
continue
return []
except Exception:
@@ -417,7 +417,7 @@ class RedditAdapter(SourceAdapter):
if not json_worked:
print(" JSON endpoints blocked, using RSS fallback")
# Brief cooldown before RSS barrage
time.sleep(3)
jitter_sleep(3)
for sub in self.subreddits:
entries = self._fetch_rss(sub)
for e in entries: