ce440149f0
Fix arxiv/reddit timeouts: batch queries, retry logic, reduced backoff
2026-07-22 14:51:51 +00:00
641d531d88
Sprint 3: Anti-bot retrieval layer + metrics module
...
Anti-bot changes (all 6 adapters):
- Browser-grade User-Agent rotation (Chrome/Firefox on Linux/Windows)
- Shared browser_headers() with Accept, Accept-Language, DNT
- Session-consistent UA fingerprint (picked once, not per-request)
- jitter_sleep() replaces fixed time.sleep() on all adapters
- Exponential backoff on 429/503 already on reddit, now consistent
New shared module:
- adapters/__init__.py: browser_user_agent(), browser_headers(), jitter_sleep()
- adapters/_http.py: HTTPClient class for future browser-mode adapters
Metrics module (from Sprint 2 carry):
- oracle/metrics.py: MetricsRun for log_adapter/log_verdicts/log_scores
- oracle/weekly.py: SYSTEM HEALTH section wired to adapter_health
- oracle/cli.py: metrics subparser with --adapters/--publish/--scores/--alerts
Before: bot signatures like 'ai-oracle/0.1', 'python:athena:v0.1'
After: 'Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0'
2026-07-22 14:26:42 +00:00
07c5f9a5c2
Sprint 0+1: Package restructure, source tiers, verdicts, multi-variant editions
...
- New oracle/ package (11 modules) with unified CLI (python -m oracle)
- Source tiers: Tier 1 (arxiv/github/hf), Tier 2 (rss/hn), Tier 3 (reddit)
- Composite verdicts: PUBLISH/WATCH/ARCHIVE/DROP based on signal score + age
- Content-hash dedup: SHA-256[:16] normalized, atomic at insert time
- Multi-variant editions: 4 YAML configs (default/research/devops/brief)
- Variant engine: filter → rank → render (HTML + JSON, themed)
- Per-adapter timeout (10s) + threading fallback
- Consolidated 12 root scripts → thin wrappers + oracle/ package
- Archived stale scripts (_engagement, _live_compare, reddit_proof)
- Updated .gitignore, README.md, schema.sql
2026-07-22 13:32:15 +00:00
424da91364
FIX: first_seen re-stamp bug — persist true source publish date, idempotent upsert, backfill DB
2026-07-13 22:16:59 +00:00
df5226859e
Fix Reddit adapter: fast-bail when blocked (403/429)
...
Bug: Reddit was taking 120s+ when rate-limited (3 attempts × 4 subreddits × backoff).
Fix: 1) Try JSON on first subreddit; if blocked, test one RSS.
2) If both fail, return empty immediately (<3s).
3) Reduce default rate_limit 3→1s.
4) 403 now immediate fail in RSS (was retrying).
5) Max RSS retries 3→2, single 2s backoff.
Result: 120s timeout → 2.1s when blocked. Zero entries returned but no hang.
2026-07-08 15:36:50 +00:00
67c002b665
Initial commit: Oracle AI research pipeline (adapters, pipeline, summarize, query)
...
Source-controlled baseline before Phase 5 cron. Excludes oracle.db,
logs/, and __pycache__ via .gitignore. Pipeline verified running
clean end-to-end (run_log write confirmed before conn.close()).
2026-07-08 04:03:36 +00:00
Epictetus