diff --git a/README.md b/README.md index fedaa0d..f8bc5d7 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,46 @@ # ClearFeed -Story layer for any-news: feed -> dedup/cluster -> rank -> LLM top-k. Signal, not noise. \ No newline at end of file +A story layer for **any** type of news: pull every feed, collapse the syndicated noise into unique stories, rank by coverage + freshness, and LLM-summarize only the top few. + +**Signal, not noise.** + +``` +sources → fetch → dedup → cluster → rank → [LLM top-k] → out/index.html +``` + +- **Dedup/cluster**: exact (normalized title / canonical URL) then MinHash → connected components. Two reworded copies of one wire story = one cluster. +- **Rank**: `score = authority × (coverage_count + 1) / (age_h + 2)^1.8` — how many sources echo a story is the importance signal. +- **LLM, hard-capped**: only `--topk` (default 10) stories get summarized, each input = headline + first ~300 chars. Never the long tail. +- **Two models on purpose**: the coding agent may use DeepSeek; story summaries hit your local Hermes/llama.cpp via `LLM_BASE_URL` (default `http://127.0.0.1:8080/v1`, model `hermes`). Never point summaries at DeepSeek unless you change `LLM_BASE_URL` on purpose. + +## Run + +`prototype/` — pure Python on the host (no Docker needed to see it work): + +```bash +cd prototype +python -m venv .venv && ./.venv/Scripts/pip install -r requirements.txt +./.venv/Scripts/python cluster.py --opml feeds.opml # 8 starter feeds → out/ +./.venv/Scripts/python cluster.py --serve :8765 # serve the generated page +./.venv/Scripts/python -m pytest -q # "2 rewrites → 1 cluster" is green +``` + +With Hermes up on :8080: + +```bash +LLM_BASE_URL=http://127.0.0.1:8080/v1 LLM_MODEL=hermes python cluster.py --llm +``` + +FreshRSS + RSSHub run in Docker (`docker compose up -d`): FreshRSS on `:8280` (first visit → installer → SQLite → one admin), RSSHub on `:1200`. Hermes stays on the host; containers reach it at `http://host.docker.internal:8080/v1`. + +## Layout + +| File | Why | +|---|---| +| `docker-compose.yml` | FreshRSS + RSSHub | +| `prototype/cluster.py` | fetch → cluster → rank → optional LLM | +| `prototype/test_cluster.py` | rewrite-dup tests | +| `prototype/feeds.opml` | 8 starter feeds | +| `AGENTS.md` | project law | +| `.agents/skills/storyline/SKILL.md` | agent skill | +| `prompts/DSH_FIRST_PROMPT.md` | paste-this prompt for dsh | \ No newline at end of file