Sprint 3: Anti-bot retrieval layer + metrics module

Anti-bot changes (all 6 adapters):
- Browser-grade User-Agent rotation (Chrome/Firefox on Linux/Windows)
- Shared browser_headers() with Accept, Accept-Language, DNT
- Session-consistent UA fingerprint (picked once, not per-request)
- jitter_sleep() replaces fixed time.sleep() on all adapters
- Exponential backoff on 429/503 already on reddit, now consistent

New shared module:
- adapters/__init__.py: browser_user_agent(), browser_headers(), jitter_sleep()
- adapters/_http.py: HTTPClient class for future browser-mode adapters

Metrics module (from Sprint 2 carry):
- oracle/metrics.py: MetricsRun for log_adapter/log_verdicts/log_scores
- oracle/weekly.py: SYSTEM HEALTH section wired to adapter_health
- oracle/cli.py: metrics subparser with --adapters/--publish/--scores/--alerts

Before: bot signatures like 'ai-oracle/0.1', 'python:athena:v0.1'
After: 'Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0'
This commit is contained in:
Epictetus
2026-07-22 14:26:42 +00:00
parent 3ec955e143
commit 641d531d88
11 changed files with 694 additions and 21 deletions
+4 -3
View File
@@ -32,9 +32,10 @@ import xml.etree.ElementTree as ET
from datetime import datetime, timedelta, timezone
from html import unescape
from adapters import SourceAdapter
from adapters import SourceAdapter, browser_headers, jitter_sleep
from adapters._store import true_first_seen, upsert_entries
# arXiv API
ARXIV_API = "http://export.arxiv.org/api/query"
@@ -152,7 +153,7 @@ class ArxivAdapter(SourceAdapter):
f"&max_results={max_results}"
)
req = urllib.request.Request(url, headers={"User-Agent": "ai-oracle/0.1"})
req = urllib.request.Request(url, headers=browser_headers())
try:
with urllib.request.urlopen(req, timeout=30) as resp:
@@ -389,7 +390,7 @@ class ArxivAdapter(SourceAdapter):
q = f"cat:{cat}"
cat_papers = self._request(q, max_results=limit, sort_by="submittedDate")
all_papers.extend(cat_papers)
time.sleep(self.rate_limit)
jitter_sleep(self.rate_limit)
# Deduplicate by arxiv_id
seen = set()