update README

2026-09-07 17:45:59 +00:00
parent 5c9cb9bcd5
commit e706c2e009
1 changed files with 44 additions and 1 deletions
@@ -1,3 +1,46 @@
 # ClearFeed

-Story layer for any-news: feed -> dedup/cluster -> rank -> LLM top-k. Signal, not noise.
+A story layer for **any** type of news: pull every feed, collapse the syndicated noise into unique stories, rank by coverage + freshness, and LLM-summarize only the top few.
+
+**Signal, not noise.**
+
+```
+sources → fetch → dedup → cluster → rank → [LLM top-k] → out/index.html
+```
+
+- **Dedup/cluster**: exact (normalized title / canonical URL) then MinHash → connected components. Two reworded copies of one wire story = one cluster.
+- **Rank**: `score = authority × (coverage_count + 1) / (age_h + 2)^1.8` — how many sources echo a story is the importance signal.
+- **LLM, hard-capped**: only `--topk` (default 10) stories get summarized, each input = headline + first ~300 chars. Never the long tail.
+- **Two models on purpose**: the coding agent may use DeepSeek; story summaries hit your local Hermes/llama.cpp via `LLM_BASE_URL` (default `http://127.0.0.1:8080/v1`, model `hermes`). Never point summaries at DeepSeek unless you change `LLM_BASE_URL` on purpose.
+
+## Run
+
+`prototype/` — pure Python on the host (no Docker needed to see it work):
+
+```bash
+cd prototype
+python -m venv .venv && ./.venv/Scripts/pip install -r requirements.txt
+./.venv/Scripts/python cluster.py --opml feeds.opml     # 8 starter feeds → out/
+./.venv/Scripts/python cluster.py --serve :8765         # serve the generated page
+./.venv/Scripts/python -m pytest -q                     # "2 rewrites → 1 cluster" is green
+```
+
+With Hermes up on :8080:
+
+```bash
+LLM_BASE_URL=http://127.0.0.1:8080/v1 LLM_MODEL=hermes python cluster.py --llm
+```
+
+FreshRSS + RSSHub run in Docker (`docker compose up -d`): FreshRSS on `:8280` (first visit → installer → SQLite → one admin), RSSHub on `:1200`. Hermes stays on the host; containers reach it at `http://host.docker.internal:8080/v1`.
+
+## Layout
+
+| File | Why |
+|---|---|
+| `docker-compose.yml` | FreshRSS + RSSHub |
+| `prototype/cluster.py` | fetch → cluster → rank → optional LLM |
+| `prototype/test_cluster.py` | rewrite-dup tests |
+| `prototype/feeds.opml` | 8 starter feeds |
+| `AGENTS.md` | project law |
+| `.agents/skills/storyline/SKILL.md` | agent skill |
+| `prompts/DSH_FIRST_PROMPT.md` | paste-this prompt for dsh |