Tony_tech/athena-oracle
3
0
Fork 0
Code Issues 6 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Inconsistent transient-error (5xx) retry across adapters #1

New Issue
Closed
opened 2026-07-10 15:39:21 +00:00 by Leonard · 2 comments
Leonard commented 2026-07-10 15:39:21 +00:00
Copy Link
Copy Source

Category: Reliability · Priority: HIGH

Summary
Adapters handle HTTP errors inconsistently. hackernews.py and huggingface.py retry 5xx with exponential backoff; arxiv.py, github.py, and reddit.py catch HTTPError, print the code, and drop the batch with no retry. Same failure class, different behavior per file.

Why it matters
A transient 503/500 from an upstream (common on arXiv) silently yields a zero-fetch cycle. The data is lost with no recovery attempt.

What happens if not done
Recurring gaps in the research record; trends missed because a source blipped that day. Compounds with issue #2 (no failure classification) and #3 (no degradation alerting).

Suggested approach
Add a shared retry helper in adapters/__init__.py: exponential backoff on 429/5xx only, 2 attempts, capped wait. Route all five adapters through it.

**Category:** Reliability · **Priority:** HIGH **Summary** Adapters handle HTTP errors inconsistently. `hackernews.py` and `huggingface.py` retry `5xx` with exponential backoff; `arxiv.py`, `github.py`, and `reddit.py` catch `HTTPError`, print the code, and drop the batch with no retry. Same failure class, different behavior per file. **Why it matters** A transient 503/500 from an upstream (common on arXiv) silently yields a zero-fetch cycle. The data is lost with no recovery attempt. **What happens if not done** Recurring gaps in the research record; trends missed because a source blipped that day. Compounds with issue #2 (no failure classification) and #3 (no degradation alerting). **Suggested approach** Add a shared retry helper in `adapters/__init__.py`: exponential backoff on 429/5xx only, 2 attempts, capped wait. Route all five adapters through it.
Leonard commented 2026-07-12 16:33:50 +00:00
Author
Copy Link
Copy Source

Resolved by PR #12 (commit feba2fe). Shared retry helper in adapters/__init__.py with exponential backoff on 429/5xx, 2 attempts, capped wait. All adapters now route through it.

Resolved by PR #12 (commit `feba2fe`). Shared retry helper in `adapters/__init__.py` with exponential backoff on 429/5xx, 2 attempts, capped wait. All adapters now route through it.
Leonard commented 2026-07-12 16:41:16 +00:00
Author
Copy Link
Copy Source

Resolved by PR #12 (commit feba2fe). Shared retry helper in adapters/__init__.py with exponential backoff on 429/5xx, 2 attempts, capped wait. All adapters now route through it.

Resolved by PR #12 (commit feba2fe). Shared retry helper in adapters/__init__.py with exponential backoff on 429/5xx, 2 attempts, capped wait. All adapters now route through it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
HA_Researcher Ty tonyjbala (Tony Balascio)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Tony_tech/athena-oracle#1
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?