Hermes Pipeline
83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
---
|
|
name: local-document-research-with-traceable-citations
|
|
version: 1.0.0
|
|
description: Enable users to import local documents, asynchronously process them into
|
|
an indexed knowledge base, and obtain AI-generated answers that cite specific page
|
|
locations and OCR evidence.
|
|
inputs:
|
|
- Local document files (PDF, DOCX, PPTX, XLSX, images, TXT)
|
|
- Configured LLM API endpoint and keys (via .env or settings)
|
|
- Optional web search service config if enabled
|
|
- Local OCR model cache (downloaded on first use)
|
|
steps:
|
|
- Import documents into a project; files are queued for asynchronous processing.
|
|
- Convert non-PDF formats (DOCX, PPTX, XLSX) to PDF using native Office or LibreOffice
|
|
fallback.
|
|
- Run OCR (PaddleOCR) on PDF pages/images to extract text, page numbers, polygons,
|
|
and confidence scores.
|
|
- Chunk text and generate embeddings; publish to LanceDB hybrid index (dense vector
|
|
+ full-text) only when fully processed.
|
|
- User starts a research session or branch; Leader agent analyzes query.
|
|
- Hybrid RAG retrieves candidate chunks from ready documents; dynamic material scope
|
|
ensures no half-indexed docs.
|
|
- Leader delegates tasks to sub-agents (researcher, reviewer, writer) via constrained
|
|
task capability; each delegation logs start, completion, duration, and evidence.
|
|
- Generate answer that references only actually used evidence; citations include document
|
|
ID, page, coordinates.
|
|
- User clicks citation to open original document and view highlighted OCR location.
|
|
outputs:
|
|
- Project with indexed document library (SQLite metadata + LanceDB vectors)
|
|
- AI answers with verifiable citations to source pages
|
|
- Evidence preview with page image and OCR highlight polygons
|
|
- Persistent session history, branches, and long-term memory
|
|
tags: []
|
|
metadata:
|
|
source_repo: https://github.com/0verL1nk/PaperSage.git
|
|
extracted_at: ''
|
|
confidence: 0.85
|
|
---
|
|
|
|
# local-document-research-with-traceable-citations
|
|
|
|
Enable users to import local documents, asynchronously process them into an indexed knowledge base, and obtain AI-generated answers that cite specific page locations and OCR evidence.
|
|
|
|
## Steps
|
|
|
|
1. Import documents into a project; files are queued for asynchronous processing.
|
|
2. Convert non-PDF formats (DOCX, PPTX, XLSX) to PDF using native Office or LibreOffice fallback.
|
|
3. Run OCR (PaddleOCR) on PDF pages/images to extract text, page numbers, polygons, and confidence scores.
|
|
4. Chunk text and generate embeddings; publish to LanceDB hybrid index (dense vector + full-text) only when fully processed.
|
|
5. User starts a research session or branch; Leader agent analyzes query.
|
|
6. Hybrid RAG retrieves candidate chunks from ready documents; dynamic material scope ensures no half-indexed docs.
|
|
7. Leader delegates tasks to sub-agents (researcher, reviewer, writer) via constrained task capability; each delegation logs start, completion, duration, and evidence.
|
|
8. Generate answer that references only actually used evidence; citations include document ID, page, coordinates.
|
|
9. User clicks citation to open original document and view highlighted OCR location.
|
|
|
|
## Inputs
|
|
|
|
- Local document files (PDF, DOCX, PPTX, XLSX, images, TXT)
|
|
- Configured LLM API endpoint and keys (via .env or settings)
|
|
- Optional web search service config if enabled
|
|
- Local OCR model cache (downloaded on first use)
|
|
|
|
## Outputs
|
|
|
|
- Project with indexed document library (SQLite metadata + LanceDB vectors)
|
|
- AI answers with verifiable citations to source pages
|
|
- Evidence preview with page image and OCR highlight polygons
|
|
- Persistent session history, branches, and long-term memory
|
|
|
|
## Failure Modes
|
|
|
|
- OCR quality low for scanned images leading to poor extraction
|
|
- Missing Office/LibreOffice causes conversion failure for Office docs
|
|
- Interrupted indexing leaves documents unpublished and excluded from retrieval
|
|
- LLM API outage or misconfiguration yields no answer
|
|
- Citation coordinates mismatch due to chunk drift
|
|
- Sub-agent recursion if constraints not enforced
|
|
|
|
## Source
|
|
|
|
Extracted from: [https://github.com/0verL1nk/PaperSage.git](https://github.com/0verL1nk/PaperSage.git)
|
|
Confidence: 0.85
|