Adapter health + feed management (issues #1, #2, #9) #12

2026-07-10T16:34:35Z

Leonard commented

2026-07-10 16:34:35 +00:00

First-time contributor

Summary

Foundational reliability + feed-health work, kept off MVP-milestone per the scope decision (MVP stays lean; this is the reliability floor it sits on).

Issue #1 — Shared retry helper

adapters/__init__.py: new http_get() with a unified retry policy — retries only 429/5xx (max 2 attempts, capped exponential backoff). 4xx other than 429 are NOT retried (config problems).
All five enabled adapters (arxiv, github, huggingface, hackernews, reddit) now route through it. Previously arxiv/github/reddit dropped 5xx with zero retry while HN/HF retried — inconsistent.
Behavior preserved: GitHub 403 rate-limit retry (via retry_403_ratelimit=True, reads X-RateLimit-Reset); Reddit 403 fast-bail + single 2s 429 retry (not slower than before).

Issue #2 — Failure classification in run_log

schema.sql: new run_log.failure_class column.
pipeline.py: captures adapter.last_failure_class (set by http_get on failure) and rolls up the most-severe class across sources (5xx > 4xx > 429 > error > zero_fetch > ok).
Every failure row now has a non-null, response-derived class — distinguishes "bad token" (4xx, fix now) from "upstream hiccup" (5xx, retry).

Issue #9 — Re-enable RSS adapter

ENABLED_SOURCES now includes rss (was registered, disabled). All 6 adapters run.
Smoke run surfaced 3 broken feeds: anthropic (404), googleai (404), metaai (301). Logged via existing feed_failures capture — left as-is; URL correction is a separate discovery task (not guessed).

Verification

python3 -c "import pipeline" → OK
Full dry-run: all 6 sources fetch; github live fetch returns 1 entry; last_failure_class=None on success
Reddit 429 → fast-bail preserved; GitHub rate-limit-low flag intact
No import/syntax errors

Test plan

Merge, then let the 13:00 UTC cron run; confirm run_log has failure_class populated
Confirm RSS appears in ingest counts
Separate task: fix anthropic/googleai/metaai RSS URLs

## Summary Foundational reliability + feed-health work, kept off `MVP-milestone` per the scope decision (MVP stays lean; this is the reliability floor it sits on). ### Issue #1 — Shared retry helper - `adapters/__init__.py`: new `http_get()` with a unified retry policy — retries **only** 429/5xx (max 2 attempts, capped exponential backoff). 4xx other than 429 are NOT retried (config problems). - All five enabled adapters (arxiv, github, huggingface, hackernews, reddit) now route through it. Previously arxiv/github/reddit dropped 5xx with zero retry while HN/HF retried — inconsistent. - **Behavior preserved**: GitHub 403 rate-limit retry (via `retry_403_ratelimit=True`, reads `X-RateLimit-Reset`); Reddit 403 fast-bail + single 2s 429 retry (not slower than before). ### Issue #2 — Failure classification in run_log - `schema.sql`: new `run_log.failure_class` column. - `pipeline.py`: captures `adapter.last_failure_class` (set by `http_get` on failure) and rolls up the most-severe class across sources (5xx > 4xx > 429 > error > zero_fetch > ok). - Every failure row now has a non-null, response-derived class — distinguishes "bad token" (4xx, fix now) from "upstream hiccup" (5xx, retry). ### Issue #9 — Re-enable RSS adapter - `ENABLED_SOURCES` now includes `rss` (was registered, disabled). All 6 adapters run. - Smoke run surfaced **3 broken feeds**: `anthropic` (404), `googleai` (404), `metaai` (301). Logged via existing `feed_failures` capture — left as-is; URL correction is a separate discovery task (not guessed). ## Verification - `python3 -c "import pipeline"` → OK - Full dry-run: all 6 sources fetch; github live fetch returns 1 entry; `last_failure_class=None` on success - Reddit 429 → fast-bail preserved; GitHub rate-limit-low flag intact - No import/syntax errors ## Test plan - [ ] Merge, then let the 13:00 UTC cron run; confirm `run_log` has `failure_class` populated - [ ] Confirm RSS appears in ingest counts - [ ] Separate task: fix anthropic/googleai/metaai RSS URLs ## Related Closes #1, Closes #2, Closes #9 (on merge).

Leonard referenced this pull request

2026-07-10 16:38:06 +00:00

Fix 3 broken RSS feed URLs (anthropic / googleai / metaai) #13

Leonard added 1 commit 2026-07-10 16:39:04 +00:00

fix(adapters): shared retry helper + run_log failure_class + enable RSS (issues #1 #2 #9 ) 23cce4d609

- adapters/__init__.py: add http_get() unified retry (429/5xx only, max 2
  attempts, capped exp backoff) + AdapterHTTPError carrying failure_class;
  SourceAdapter.last_failure_class set on failure for pipeline capture.
- arxiv/github/huggingface/hackernews/reddit: route HTTP through http_get.
  Preserves GitHub 403 rate-limit retry and Reddit 403/429 fast-bail.
- schema.sql + pipeline.py: add run_log.failure_class column; rollup most-
  severe class across sources (5xx>4xx>429>error>zero_fetch>ok).
- pipeline.py: ENABLE RSS in ENABLED_SOURCES (was registered, disabled).
- RSS smoke test surfaced 3 broken feeds (anthropic 404, googleai 404,
  metaai 301) — left as-is, captured in feed_failures; URL fix is separate
  discovery task, not guessed.

Verified: full dry-run fetches all 6 sources; github live fetch OK;
Reddit 429 fast-bail preserved; no import/syntax errors.

Leonard force-pushed fix/adapter-health-1-2-9 from 8017ded3ba to 23cce4d609

2026-07-10 16:39:04 +00:00

Compare

Leonard referenced this pull request

2026-07-10 17:27:48 +00:00

Curation logic decentralized across adapters #7

Leonard referenced this pull request

2026-07-10 17:42:30 +00:00

Centralize curation config (issue #7) #14

Leonard referenced this pull request

2026-07-12 16:33:50 +00:00

Inconsistent transient-error (5xx) retry across adapters #1