Tony_tech/athena-oracle
3
0
Fork 0
Code Issues 6 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: apply all 18 DevDesignReview edits to Dev-Design.md

Browse Source 
- Ch 1: MVP Success Criteria
- Ch 2: Daily Run Flow + Failure Modes
- Ch 3: Backup Strategy + Schema Migration
- Ch 4: Adapter Interface Contract
- Ch 5: Theme Governance
- Ch 6: Calibration Process
- Ch 7: MCP Tool Signatures + API Auth Model
- Ch 8: Log Retention/SLIs + Alerting Matrix
- Ch 9: Scheduling Recommendation
- Ch 10: Model Upgrade Process
- Ch 11: MVP Security Baseline
- Ch 12: Definition of Done per Phase
- Ch 14: Revisit column on Design Decisions table
- New Ch 15: Testing Strategy
- New Ch 16: Deployment and CI/CD
- New Ch 17: Operational Runbooks
- New Ch 18: Risk Register and Assumptions
- New Ch 19: Dependencies and Tooling
This commit is contained in:
ty
2026-07-09 12:43:22 -07:00
parent baf402c80e
commit cead87370b
1 changed files with 409 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Dev-Design.md
+409 -14
View File
@@ -22,6 +22,16 @@ Athena is an autonomous research intelligence engine that ingests from multiple
| Autonomous operation | Cron/scheduled pipeline + graceful degradation | The pipeline runs daily without human intervention. If Ollama is down, ingestion continues and summarization defers to the next run. If one adapter fails, the rest still run. | | Autonomous operation | Cron/scheduled pipeline + graceful degradation | The pipeline runs daily without human intervention. If Ollama is down, ingestion continues and summarization defers to the next run. If one adapter fails, the rest still run. |
| Lightweight deployment | Python + SQLite + Flask + host-level Ollama | No PostgreSQL, no Elasticsearch, no Redis, no Kubernetes. A single Python process, a single SQLite file, and an external Ollama REST API. | | Lightweight deployment | Python + SQLite + Flask + host-level Ollama | No PostgreSQL, no Elasticsearch, no Redis, no Kubernetes. A single Python process, a single SQLite file, and an external Ollama REST API. |
### MVP Success Criteria
The MVP is considered successful when:
1. **Research Time**: Bob answers 'what's trending this week' in <15 min of research time total (across all sources).
2. **Signal Detection**: Over 7 days, the theme scan correctly identifies ≥1 real trend that has genuine cross-source convergence.
3. **Noise Rejection**: Over the same 7-day period, the falsification engine kills ≥1 false signal (a one-day spike with no sustained arrivals) to demonstrate decay-based filtering is working.
--- ---
## 2. System Architecture ## 2. System Architecture
@@ -93,6 +103,26 @@ Athena is an autonomous research intelligence engine that ingests from multiple
The 150MB constraint applies to the pipeline process. The full system footprint including Ollama is ~2GB. The 150MB constraint applies to the pipeline process. The full system footprint including Ollama is ~2GB.
### Daily Run Flow
1. **Preflight** — Check disk space, verify DB integrity (`PRAGMA integrity_check`), load adapter config
2. **Ingest** — Run all 6 adapters in parallel, collect raw items per source
3. **Dedup** — Hash URLs, match against existing entries, insert only new items
4. **Theme Tag** — Run keyword co-occurrence against new items, tag themes
5. **Falsification** — Recompute decay scores for all active themes, kill dead theses
6. **Archive** — Move entries older than 90 days to archive table
7. **Report** — Write daily summary to `/output/`, update run_log
### Failure Modes
| Component | Failure | Impact | Recovery |
|---|---|---|---|
| Adapter | Rate limit / 503 | Missing items from that source | Retry next cycle; pipeline continues |
| Ollama | Down | Summaries skipped | Entries stored with `summary = null`, deferred to next run |
| SQLite | Disk full | No writes | Alert via webhook; manual cleanup |
| SQLite | Corruption | Data loss | Restore from last backup |
| Network | Outbound blocked | All adapters fail | Alert; pipeline exits code 2 |
--- ---
## 3. Data Layer ## 3. Data Layer
@@ -122,6 +152,29 @@ Core tables (from `schema.sql`):
- Periodic `VACUUM` to reclaim space - Periodic `VACUUM` to reclaim space
- `archive.py` handles cold storage rotation (deferred to Phase 3) - `archive.py` handles cold storage rotation (deferred to Phase 3)
### 3.4 Backup Strategy
- **Pre-schema-change backup**: Before any `ALTER TABLE` or schema modification:
```bash
sqlite3 oracle.db '.backup oracle.db.bak'
```
Store `.bak` files with date suffix in `/backup/` (`oracle.db.bak-YYYYMMDD`).
- **Daily compressed backup**: At 01:00 UTC (off-peak):
```bash
tar -czf /backup/oracle.db.$(date +%Y%m%d).tar.gz oracle.db
```
Retain 30 days of backups; purge older: `find /backup -name '*.tar.gz' -mtime +30 -delete`
- **Recovery**: Restore from backup with `cp /backup/oracle.db.bak-YYYYMMDD oracle.db`, verify with `PRAGMA integrity_check`
### 3.5 Schema Migration
- Versioned migration files in `migrations/` directory (e.g., `001_initial_schema.sql`, `002_add_theme_tags.sql`)
- Applied at startup: pipeline checks `schema_version` table, runs any unapplied migrations in order
- Each migration is a single atomic SQL file; no partial migrations
- Rollback: each migration includes a comment with the reverse SQL if needed
--- ---
## 4. Adapter Layer ## 4. Adapter Layer
@@ -152,6 +205,27 @@ arXiv papers appear on HN, Reddit, and Twitter. Without deduplication, the same
- `tenacity` library for exponential backoff on transient failures (429, 503, timeout) - `tenacity` library for exponential backoff on transient failures (429, 503, timeout)
- One failing adapter does not kill the pipeline - One failing adapter does not kill the pipeline
### 4.4 Adapter Interface Contract
All adapters must implement this minimal interface:
- **name() → str**: Unique identifier for the source (e.g., `"arxiv"`)
- **fetch(query, limit) → list[dict]**: Returns normalized items with required schema fields:
- `source` (str): Source name matching `name()`
- `source_id` (str): Unique per-source ID for deduplication
- `title` (str): Human-readable title
- `url` (str): Direct URL to the entry
- `timestamp` (datetime): Publication/update time
- `raw_score` (float): Source-specific signal strength (0.0–10.0)
- `body` (str): Raw text content for summarization and theme tagging
**Custom Exceptions:**
- `RateLimitError`: Raised when source returns 429 or similar; includes `retry_after` (seconds)
- `SourceUnavailableError`: Raised when source is down (5xx) or unreachable
Adapters must not raise other exceptions on normal operation; unexpected errors should be logged with full traceback and surfaced via the run_log, not propagated to the pipeline.
--- ---
## 5. Theme Tagging and Convergence Detection ## 5. Theme Tagging and Convergence Detection
@@ -189,6 +263,25 @@ Thresholds:
- `≥ 1.5` → "emerging" signal - `≥ 1.5` → "emerging" signal
- `< 1.5` → "noise" - `< 1.5` → "noise"
### 5.4 Theme Governance
Themes are defined in a single YAML file: `themes.yaml`. Each theme entry includes:
- `name`: human-readable identifier (e.g., `tool-call`)
- `keywords`: list of keyword patterns for Phase 1 regex matching
- `owner`: responsible person (for quarterly review)
- `created_at`: ISO date of creation
- `status`: `active` or `deprecated`
**Lifecycle:**
- **Propose**: New themes require owner nomination + approval from project lead
- **Review**: Active themes are reviewed quarterly; deprecated if <2 hits in 30 days
- **Retire**: Deprecated themes are excluded from convergence scoring after 90 days
- **Reinstate**: Deprecated themes can be reactivated if signals re-emerge
The "other" catch-all bucket captures signals that don't match any active theme and is reviewed during quarterly theme audits for potential new theme creation.
--- ---
## 6. Falsification Engine ## 6. Falsification Engine
@@ -215,6 +308,17 @@ A signal is flagged "unverified" if:
- The signal appears only in echo chambers (e.g., HN upvotes ≠ real adoption) - The signal appears only in echo chambers (e.g., HN upvotes ≠ real adoption)
- A counter-narrative exists in the same time window - A counter-narrative exists in the same time window
### 6.3 Calibration Process
- **Initial parameters**: Start with λ = 0.1 (half-life ~7 days) for all themes at deployment
- **Validation window**: After 30 days of live operation, validate against historical data
- **Adjustment triggers**:
- If >20% of confirmed real trends were falsely killed → decrease λ (e.g., to 0.05, slower decay)
- If >30% of noise signals were incorrectly confirmed → increase λ (e.g., to 0.15, faster decay)
- **Documentation**: Record calibration decisions in `calibration_log.md` with date, old/new λ values, and rationale
Re-calibrate quarterly or whenever a major theme dictionary change is made.
--- ---
## 7. Output Layer ## 7. Output Layer
@@ -254,6 +358,31 @@ MCP tools for Hermes agent integration:
- `oracle_verdicts(status)` — confirmed or dead theses - `oracle_verdicts(status)` — confirmed or dead theses
- `oracle_latest(source)` — most recent entry per source - `oracle_latest(source)` — most recent entry per source
### 7.5 MCP Tool Signatures
All MCP tools must implement these minimum signatures:
- **get_trends()**:
- Request: `{}` (no params)
- Response: `{ "trends": [{"name": str, "score": float, "sources": [str], "decay_score": float}] }`
- **search_entry(query: str, source: str | None = None)**:
- Request: `{ "query": str, "source": str | null }`
- Response: `{ "entries": [{"title": str, "url": str, "summary": str, "score": float}] }`
- **get_convergence_report()**:
- Request: `{}`
- Response: `{ "converged": [{"entity": str, "sources": [str], "confidence": float}] }`
Tools must validate input types and return empty arrays (not errors) for valid queries that yield no results.
### 7.6 API Authentication Model
- **Phase 4 (MVP)**: Simple API key in `X-API-Key` header. No expiration, stored in config file (`/etc/athena/api_keys.yaml`)
- **Phase 6 (Production)**: JWT bearer token with scopes (read-only, read-write, admin). Tokens expire after 24 hours; refresh via `/auth/token` endpoint
Auth failures return HTTP 401 with `{ "error": "unauthorized" }`. Rate limiting applies per-key: 100 requests/minute.
--- ---
## 8. Observability and Reliability ## 8. Observability and Reliability
@@ -291,8 +420,28 @@ Discord/Slack webhook triggered when:
If Ollama is unreachable: If Ollama is unreachable:
- Ingestion continues normally - Ingestion continues normally
- Summarization is skipped, entries stored with `summary = null` - Summarization is skipped, entries stored with `summary = null`
- Deferral: next run summarizes pending entries - **Deferral**: next run summarizes pending entries
- Alert: "summarization deferred, N entries pending" - **Alert**: "summarization deferred, N entries pending"
### 8.5 Log Retention and SLIs
- **Log retention**: 30 days rolling; gzip-compressed after 7 days to save disk space
- **SLI definitions**:
- **Pipeline success rate**: >95% of daily runs complete without critical failure (exit code 2)
- **Adapter availability**: >90% of scheduled runs successfully fetch each source (per-source metric)
- **Theme detection accuracy**: ≥80% of manually verified trends identified correctly in first week
### 8.6 Alerting Matrix
| Condition | Channel | Severity | Response Time |
|---|---|---|---|
| Adapter fails >2 consecutive runs | Discord webhook | P2 | Investigate within 1 hour |
| Pipeline exit code 2 | Discord webhook + email | P1 | Investigate within 30 minutes |
| DB disk usage >85% | Discord webhook | P2 | Investigate within 2 hours |
| Ollama unreachable >5 min | Discord webhook | P2 | Restart service if needed |
| Pipeline success rate <90% for 3 days | Email + dashboard | P3 | Review next cycle |
Alerts are deduplicated: same condition won't fire again until resolved.
--- ---
@@ -315,6 +464,16 @@ If Ollama is unreachable:
APScheduler adds in-process async daemon overhead. Cron/systemd is OS-level, zero process memory cost, and sufficient for daily runs. APScheduler is the Phase 2 target if dynamic scheduling (user-configurable refresh rates, per-source intervals) is needed. APScheduler adds in-process async daemon overhead. Cron/systemd is OS-level, zero process memory cost, and sufficient for daily runs. APScheduler is the Phase 2 target if dynamic scheduling (user-configurable refresh rates, per-source intervals) is needed.
### 9.4 Scheduling Recommendation
- **Phase 1 (MVP)**: Use system cron for daily runs at 13:00 UTC. Sufficient for fixed schedule, zero process memory cost, OS-level reliability.
- **Phase 2**: Switch to systemd timers if dynamic scheduling needed (skip runs on holidays, adjust time zones). Better observability and integration with monitoring tools.
- **APScheduler**: Only use if per-source intervals are required (e.g., arXiv every 6 hours, Reddit every 30 min). Adds ~50MB process overhead — not recommended for Phase 1.
**Recommendation**: Start with cron for Phase 1. Evaluate whether Phase 2 sources need different intervals before considering systemd timers or APScheduler.
--- ---
## 10. Inference ## 10. Inference
@@ -330,6 +489,16 @@ APScheduler adds in-process async daemon overhead. Cron/systemd is OS-level, zer
Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB container is physically impossible. Running it host-level means the pipeline process stays within budget and Ollama can share resources with other services. Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB container is physically impossible. Running it host-level means the pipeline process stays within budget and Ollama can share resources with other services.
### 10.3 Model Upgrade Process
- **Swap model**: Replace model file; update systemd unit `ExecStart` path if needed
- **Restart service**: `systemctl restart qwythos-gpu0.service` (or equivalent unit)
- **Verify health**: `curl http://localhost:8081/v1/health` — should return `{ "status": "ready" }`
- **Quality check**: Run first summarization on known-good source (e.g., arXiv paper), compare output against baseline summary for content accuracy, length consistency, and hallucination rate
- **Rollback**: If quality degrades (e.g., summary length <50 tokens, hallucination rate >10%), revert to previous model file immediately
**Quality metrics**: Pass if summary is >50 tokens, no hallucinations on known entities, and theme detection matches golden samples. Only proceed with upgrade after successful verification.
--- ---
## 11. Security ## 11. Security
@@ -342,6 +511,25 @@ Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB containe
| DB protection | `chmod 600 oracle.db` | | DB protection | `chmod 600 oracle.db` |
| Input sanitization | Parameterized SQL queries, no string concatenation | | Input sanitization | Parameterized SQL queries, no string concatenation |
### 11.2 MVP Security Baseline
- **Container hardening**:
- Run as non-root user (`user: nobody` in Dockerfile)
- Read-only filesystem where possible (except `/tmp`, `/var/log`)
- No SSH access inside container; pipeline is cron-triggered, no interactive access needed
- Minimal base image: `python:3.11-slim` (no dev tools, no git)
- **Dependency scanning**:
- CI pipeline runs `pip audit` or `safety check` on every push to `MVP-milestone`
- Fail build if critical vulnerabilities found (>CVSS 7.0)
- Warn on medium/high vulnerabilities; require manual review before merging
- **Secret management**:
- No secrets in code or config files (use environment variables at runtime)
- API keys stored in `/etc/athena/secrets.yaml` with restricted permissions (`chmod 0600`)
**Enforcement**: Security checks are automated in CI; local development is permissive but container builds must pass all scans.
--- ---
## 12. Implementation Phases ## 12. Implementation Phases
@@ -356,6 +544,54 @@ Ollama daemon + 1B model requires ~2GB RAM. Running it inside the 150MB containe
| **P5: Validation** | 7-day UAT window, Hermes cron integration, exit codes | System runs unattended | | **P5: Validation** | 7-day UAT window, Hermes cron integration, exit codes | System runs unattended |
| **P6: Scale** | sqlite-vec + embeddings, MCP server, APScheduler, BERTopic themes | Research-grade system | | **P6: Scale** | sqlite-vec + embeddings, MCP server, APScheduler, BERTopic themes | Research-grade system |
### 12.4 Definition of Done per Phase
Each phase must pass all listed criteria before being marked complete:
**P0 (Foundation)**:
- `schema.sql` creates all tables without errors
- Empty pipeline runs cleanly with exit code 0
- `oracle-pipeline.sh` is idempotent (safe to run twice)
- Adapter registry loads all 6 adapters
**P1 (First data)**:
- arXiv + RSS adapters fetch successfully
- Entries stored in SQLite with correct schema
- Keyword convergence detects at least 1 theme
- Deduplication works (no duplicate entries)
**P2 (Full ingest)**:
- All 6 adapters fetch successfully in one run
- Rate limiting enforced per adapter
- Structured JSON logs emitted per pipeline stage
- No adapter failure kills the pipeline
**P3 (Falsification)**:
- Exponential decay scoring implemented
- Ollama summarization works with graceful degradation
- Trend verdicts computed: confirmed/emerging/dead
- Running over 7 days shows false signals dying
**P4 (Consumption)**:
- REST API endpoints return valid JSON
- Daily file drops written to `/output/`
- Health endpoint reports accurate status
- Alerting webhooks fire on simulated failures
**P5 (Validation)**:
- 7-day UAT: pipeline runs unattended without intervention
- Hermes cron integration works
- Exit codes correct: 0=success, 1=partial, 2=failure
- Pipeline completes <30 minutes end-to-end
**P6 (Scale)**:
- sqlite-vec + embeddings operational
- MCP server responds to all 3 tools
- APScheduler handles per-source intervals
- System stays within 150MB pipeline memory budget
**General**: No critical bugs open, all unit tests pass, CI green, security scan clean.
--- ---
## 13. Backport to PRD: Requirements to Add ## 13. Backport to PRD: Requirements to Add
@@ -436,18 +672,177 @@ The following requirements are implied by this design and should be added to `do
## 14. Design Decisions Summary (Why) ## 14. Design Decisions Summary (Why)
| Decision | Why | | Decision | Why | Revisit |
|---|---| |---|---|---|
| **SQLite over PostgreSQL** | Zero external dependency, single file, FTS5 built-in, handles 1M rows fine. pgvector is premature at this scale. | | **SQLite over PostgreSQL** | Zero external dependency, single file, FTS5 built-in, handles 1M rows fine. pgvector is premature at this scale. | P6 (if scale demands pgvector) |
| **Ollama host-level** | 150MB container cannot fit Ollama + model (~2GB). Host-level lets pipeline stay within budget. | | **Ollama host-level** | 150MB container cannot fit Ollama + model (~2GB). Host-level lets pipeline stay within budget. | P2 (if inference needs change) |
| **Flask over FastAPI** | ~1MB vs ~100MB runtime overhead. FastAPI is Phase 2 target; Flask suffices for internal REST API. | | **Flask over FastAPI** | ~1MB vs ~100MB runtime overhead. FastAPI is Phase 2 target; Flask suffices for internal REST API. | P2 (when FastAPI becomes viable) |
| **Keyword co-occurrence (Phase 1)** | Zero-dependency, explainable, works at 150MB. Embeddings (Phase 2) add semantic convergence. | | **Keyword co-occurrence (Phase 1)** | Zero-dependency, explainable, works at 150MB. Embeddings (Phase 2) add semantic convergence. | P2 (when embeddings ready) |
| **Fixed themes + catch-all** | BERTopic requires 4GB RAM. Fixed themes with "other" bucket is the pragmatic constraint choice. | | **Fixed themes + catch-all** | BERTopic requires 4GB RAM. Fixed themes with "other" bucket is the pragmatic constraint choice. | P6 (when auto-discovery needed) |
| **Cron over APScheduler (Phase 1)** | OS-level, zero process memory cost. APScheduler is Phase 2 for dynamic scheduling. | | **Cron over APScheduler (Phase 1)** | OS-level, zero process memory cost. APScheduler is Phase 2 for dynamic scheduling. | P2 (if per-source intervals needed) |
| **HTTP-only adapters** | All 6 sources have programmatic APIs. Playwright adds 300MB+ overhead and fragility. | | **HTTP-only adapters** | All 6 sources have programmatic APIs. Playwright adds 300MB+ overhead and fragility. | N/A (stable) |
| **Exponential decay over 7-day rule** | One-line formula, no fixed threshold. Handles fast-dying and slow-burn trends naturally. | | **Exponential decay over 7-day rule** | One-line formula, no fixed threshold. Handles fast-dying and slow-burn trends naturally. | N/A (stable) |
| **Deduplication required** | arXiv papers appear on HN/Reddit/Twitter. Without dedup, same signal counted 3× = false convergence. | | **Deduplication required** | arXiv papers appear on HN/Reddit/Twitter. Without dedup, same signal counted 3× = false convergence. | N/A (stable) |
| **Graceful degradation on Ollama** | Ingestion must not depend on summarization. Store raw data, defer summaries. | | **Graceful degradation on Ollama** | Ingestion must not depend on summarization. Store raw data, defer summaries. | N/A (stable) |
## 15. Testing Strategy
### 15.1 Unit Tests
- **Per adapter**: Test each of the 6 adapters with known-good endpoints. Verify:
- Returns valid JSON with required schema fields (source, source_id, title, url, timestamp)
- Handles rate limits gracefully (no infinite loops)
- Correct error codes for 429/503
- **Scoring functions**: Unit tests for exponential decay, convergence scoring, and theme matching. Include edge cases (empty input, negative scores).
### 15.2 Integration Tests
- Run pipeline against seed dataset (100 entries from arXiv + RSS). Verify:
- All adapters fetch successfully
- Deduplication removes duplicates correctly
- Theme detection identifies at least 3 themes
- No critical errors in logs
### 15.3 E2E Tests
- **Cron-to-snapshot**: Run full pipeline via cron, verify output files match expected snapshot (golden file comparison)
- **Stress test**: Run 10 consecutive daily cycles with simulated failures (adapter timeout, Ollama down). Verify graceful degradation and recovery
**Test coverage goal**: 80% of critical paths covered by automated tests. Manual testing for theme quality and summary accuracy.
## 16. Deployment and CI/CD
### 16.1 CI Pipeline
On every push to `MVP-milestone`:
- **Lint**: `ruff check`, `mdlint docs/`
- **Test**: Run unit tests (`pytest tests/`), integration tests with seed dataset
- **Build**: Create Docker image, tag with commit SHA
- **Scan**: Run `pip audit`; fail if critical vulnerabilities (>CVSS 7.0) found
### 16.2 Deployment
- **Local dev**: `docker-compose up` (pipeline container + host-level Ollama)
- **Production**: `docker-compose up -d` + systemd services for inference (`qwythos-gpu0.service`)
- Ollama runs host-level (not in container) due to ~2GB memory requirement
### 16.3 Rollback Procedure
If deployment fails or quality degrades:
1. Stop services: `systemctl stop oracle-pipeline qwythos-gpu0`
2. Restore previous code: `git checkout <good-tag>`
3. Rebuild Docker image from restored code
4. Restart services: `systemctl start oracle-pipeline qwythos-gpu0`
5. Verify health: `curl http://localhost:8081/v1/health`
**Rollback window**: Must complete within 5 minutes of failure detection.
**Tagging**: Each deploy is tagged with semantic versioning (`v1.0.0`, `v1.1.0`) for easy rollback reference.
## 17. Operational Runbooks
### 17.1 Daily Pipeline Verification
At 13:00 UTC after each run:
1. Check logs: `journalctl -u oracle-pipeline --since "today" | grep ERROR`
2. Verify exit code in `/var/log/oracle-pipeline/run_log.txt` — should be 0
3. Confirm output: `ls -lh /output/$(date +%Y-%m-%d)/` — should have `summary.json` and `metrics.json`
4. If any check fails, investigate with the relevant runbook below
### 17.2 Database Recovery from Corruption
If `sqlite3 oracle.db 'PRAGMA integrity_check'` returns errors:
1. Stop pipeline: `systemctl stop oracle-pipeline`
2. Restore from last backup: `cp /backup/oracle.db.bak-YYYYMMDD oracle.db`
3. Verify integrity: `sqlite3 oracle.db 'PRAGMA integrity_check'` — should return "ok"
4. Restart pipeline: `systemctl start oracle-pipeline`
**Prevention**: Daily compressed backups to `/backup/`, retention 30 days.
### 17.3 Adapter Failure Investigation
If adapter fails repeatedly (>2 consecutive runs):
1. Check logs: `journalctl -u oracle-pipeline --since "today" | grep -A5 "Adapter"`
2. Test endpoint manually: `curl -X GET <adapter_url>` — verify HTTP status
3. Check rate limit headers: `curl -I <adapter_url> | grep -i 'x-ratelimit'`
4. If rate-limited: Wait `retry_after` seconds, pipeline retries next cycle
5. Escalate to project lead if >3 consecutive failures
### 17.4 Ollama Service Restart
If Ollama becomes unresponsive:
1. Check status: `systemctl status qwythos-gpu0.service`
2. View logs: `journalctl -u qwythos-gpu0.service --since "today"`
3. Restart service: `systemctl restart qwythos-gpu0.service`
4. Verify health: `curl http://localhost:8081/v1/health` — should return `{ "status": "ready" }`
5. Check GPU: `nvidia-smi` — ensure service actually loaded
**Escalation**: If issue persists after restart, notify project lead.
**Runbook maintenance**: Update runbooks when features change. Document changes in `runbook_changes.md`.
## 18. Risk Register and Assumptions
### 18.1 Risk Register
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Source API changes (rate limits, endpoint shifts) | High | High | Auto-retry with exponential backoff; monitor adapter health; log API changes for review |
| Model quality drift (summarization accuracy degrades) | Medium | Medium | Weekly golden sample comparison; rollback if hallucination rate >10% or summary <50 tokens |
| Disk space exhaustion (backups, logs) | High | High | Automated 30-day retention; alert at 85% usage; compress old backups |
| Single operator bottleneck (manual theme review) | Medium | Medium | Documented theme governance; quarterly review cycle; catch-all "other" bucket |
| Network outage (all adapters fail) | Low | High | Graceful degradation: store raw data without summaries; resume when network returns |
| SQLite corruption during schema migration | Low | Critical | Pre-migration backup; atomic migration scripts; integrity check after each migration |
### 18.2 Key Assumptions
- All 6 source APIs remain stable for at least 6 months (no breaking changes)
- GPU memory available for Qwythos inference (~15GB free on GPU0)
- Network connectivity to all sources is available during pipeline runs
- Single operator can complete manual theme review within 24 hours
- Disk space sufficient for 30-day backup retention (~10GB)
**Risk review**: Quarterly review of this register; update mitigations when new risks emerge.
**Assumption tracking**: If an assumption proves false, document the deviation in `assumption_deviations.md` and reassess risks.
## 19. Dependencies and Tooling
### 19.1 Python Runtime Dependencies
All dependencies pinned to exact versions in `requirements.txt`:
- **requests**: HTTP client for all source APIs
- **beautifulsoup4**: HTML parsing for RSS feeds
- **feedparser**: RSS/Atom feed handling (primary)
- **tenacity**: Retry logic with exponential backoff (used by all adapters)
- **flask**: Internal REST API for Phase 4+ (served on port 8081)
No external database dependencies — SQLite is built into Python.
### 19.2 Runtime System Dependencies
- **Ollama**: Host-level inference service (~2GB RAM, GPU acceleration)
- **systemd**: Service management for pipeline and inference (`oracle-pipeline.service`, `qwythos-gpu0.service`)
- **cron**: Daily schedule trigger (Phase 1); systemd timers for Phase 2
- **sqlite3**: Database CLI for backup/recovery commands
### 19.3 Development Tooling
- **ruff**: Linting and formatting (Python)
- **pytest**: Unit and integration test framework
- **docker**: Containerization for pipeline
- **pip audit / safety**: Dependency vulnerability scanning (CI checks)
### 19.4 Dependency Update Policy
- Critical security updates: Apply within 7 days of CVE disclosure
- Minor version updates: Test in staging before production deployment
- Major version upgrades: Require migration scripts and rollback plan
**Tooling policy**: No new dependencies without approval from project lead; document rationale in `dependency_justifications.md`.
**Lock file**: `requirements.txt` pinned to exact versions for reproducible builds.
--- ---