ClearFeed/README.md

ClearFeed

A story layer for any type of news: pull every feed, collapse the syndicated noise into unique stories, rank by coverage + freshness, and LLM-summarize only the top few.

Signal, not noise.

sources → fetch → dedup → cluster → rank → [LLM top-k] → out/index.html

• Dedup/cluster: exact (normalized title / canonical URL) then MinHash → connected components. Two reworded copies of one wire story = one cluster.
• Rank: score = authority × (coverage_count + 1) / (age_h + 2)^1.8 — how many sources echo a story is the importance signal.
• LLM, hard-capped: only --topk (default 10) stories get summarized, each input = headline + first ~300 chars. Never the long tail.
• Two models on purpose: the coding agent may use DeepSeek; story summaries hit your local Hermes/llama.cpp via LLM_BASE_URL (default http://127.0.0.1:8080/v1, model hermes). Never point summaries at DeepSeek unless you change LLM_BASE_URL on purpose.

Run

prototype/ — pure Python on the host (no Docker needed to see it work):

cd prototype
python -m venv .venv && ./.venv/Scripts/pip install -r requirements.txt
./.venv/Scripts/python cluster.py --opml feeds.opml     # 8 starter feeds → out/
./.venv/Scripts/python cluster.py --serve :8765         # serve the generated page
./.venv/Scripts/python -m pytest -q                     # "2 rewrites → 1 cluster" is green

With Hermes up on :8080:

LLM_BASE_URL=http://127.0.0.1:8080/v1 LLM_MODEL=hermes python cluster.py --llm

FreshRSS + RSSHub run in Docker (docker compose up -d): FreshRSS on :8280 (first visit → installer → SQLite → one admin), RSSHub on :1200. Hermes stays on the host; containers reach it at http://host.docker.internal:8080/v1.

Layout

File	Why
docker-compose.yml	FreshRSS + RSSHub
prototype/cluster.py	fetch → cluster → rank → optional LLM
prototype/test_cluster.py	rewrite-dup tests
prototype/feeds.opml	8 starter feeds
AGENTS.md	project law
.agents/skills/storyline/SKILL.md	agent skill
prompts/DSH_FIRST_PROMPT.md	paste-this prompt for dsh