# ClearFeed

A story layer for **any** type of news: pull every feed, collapse the syndicated noise into unique stories, rank by coverage + freshness, and LLM-summarize only the top few.

**Signal, not noise.**

```
sources → fetch → dedup → cluster → rank → [LLM top-k] → out/index.html
```

- **Dedup/cluster**: exact (normalized title / canonical URL) then MinHash → connected components. Two reworded copies of one wire story = one cluster.
- **Rank**: `score = authority × (coverage_count + 1) / (age_h + 2)^1.8` — how many sources echo a story is the importance signal.
- **LLM, hard-capped**: only `--topk` (default 10) stories get summarized, each input = headline + first ~300 chars. Never the long tail.
- **Two models on purpose**: the coding agent may use DeepSeek; story summaries hit your local Hermes/llama.cpp via `LLM_BASE_URL` (default `http://127.0.0.1:8080/v1`, model `hermes`). Never point summaries at DeepSeek unless you change `LLM_BASE_URL` on purpose.

## Run

`prototype/` — pure Python on the host (no Docker needed to see it work):

```bash
cd prototype
python -m venv .venv && ./.venv/Scripts/pip install -r requirements.txt
./.venv/Scripts/python cluster.py --opml feeds.opml     # 8 starter feeds → out/
./.venv/Scripts/python cluster.py --serve :8765         # serve the generated page
./.venv/Scripts/python -m pytest -q                     # "2 rewrites → 1 cluster" is green
```

With Hermes up on :8080:

```bash
LLM_BASE_URL=http://127.0.0.1:8080/v1 LLM_MODEL=hermes python cluster.py --llm
```

FreshRSS + RSSHub run in Docker (`docker compose up -d`): FreshRSS on `:8280` (first visit → installer → SQLite → one admin), RSSHub on `:1200`. Hermes stays on the host; containers reach it at `http://host.docker.internal:8080/v1`.

## Layout

| File | Why |
|---|---|
| `docker-compose.yml` | FreshRSS + RSSHub |
| `prototype/cluster.py` | fetch → cluster → rank → optional LLM |
| `prototype/test_cluster.py` | rewrite-dup tests |
| `prototype/feeds.opml` | 8 starter feeds |
| `AGENTS.md` | project law |
| `.agents/skills/storyline/SKILL.md` | agent skill |
| `prompts/DSH_FIRST_PROMPT.md` | paste-this prompt for dsh |