46 lines
2.1 KiB
Markdown
46 lines
2.1 KiB
Markdown
# ClearFeed
|
||
|
||
A story layer for **any** type of news: pull every feed, collapse the syndicated noise into unique stories, rank by coverage + freshness, and LLM-summarize only the top few.
|
||
|
||
**Signal, not noise.**
|
||
|
||
```
|
||
sources → fetch → dedup → cluster → rank → [LLM top-k] → out/index.html
|
||
```
|
||
|
||
- **Dedup/cluster**: exact (normalized title / canonical URL) then MinHash → connected components. Two reworded copies of one wire story = one cluster.
|
||
- **Rank**: `score = authority × (coverage_count + 1) / (age_h + 2)^1.8` — how many sources echo a story is the importance signal.
|
||
- **LLM, hard-capped**: only `--topk` (default 10) stories get summarized, each input = headline + first ~300 chars. Never the long tail.
|
||
- **Two models on purpose**: the coding agent may use DeepSeek; story summaries hit your local Hermes/llama.cpp via `LLM_BASE_URL` (default `http://127.0.0.1:8080/v1`, model `hermes`). Never point summaries at DeepSeek unless you change `LLM_BASE_URL` on purpose.
|
||
|
||
## Run
|
||
|
||
`prototype/` — pure Python on the host (no Docker needed to see it work):
|
||
|
||
```bash
|
||
cd prototype
|
||
python -m venv .venv && ./.venv/Scripts/pip install -r requirements.txt
|
||
./.venv/Scripts/python cluster.py --opml feeds.opml # 8 starter feeds → out/
|
||
./.venv/Scripts/python cluster.py --serve :8765 # serve the generated page
|
||
./.venv/Scripts/python -m pytest -q # "2 rewrites → 1 cluster" is green
|
||
```
|
||
|
||
With Hermes up on :8080:
|
||
|
||
```bash
|
||
LLM_BASE_URL=http://127.0.0.1:8080/v1 LLM_MODEL=hermes python cluster.py --llm
|
||
```
|
||
|
||
FreshRSS + RSSHub run in Docker (`docker compose up -d`): FreshRSS on `:8280` (first visit → installer → SQLite → one admin), RSSHub on `:1200`. Hermes stays on the host; containers reach it at `http://host.docker.internal:8080/v1`.
|
||
|
||
## Layout
|
||
|
||
| File | Why |
|
||
|---|---|
|
||
| `docker-compose.yml` | FreshRSS + RSSHub |
|
||
| `prototype/cluster.py` | fetch → cluster → rank → optional LLM |
|
||
| `prototype/test_cluster.py` | rewrite-dup tests |
|
||
| `prototype/feeds.opml` | 8 starter feeds |
|
||
| `AGENTS.md` | project law |
|
||
| `.agents/skills/storyline/SKILL.md` | agent skill |
|
||
| `prompts/DSH_FIRST_PROMPT.md` | paste-this prompt for dsh | |