Epictetus
271f79610d
New skills: - blacknode-graph-workflow - multi-agent-workflow-execution - langgraph-agent-workflow - langgraph-multi-agent-router - three-tier-evaluation-pipeline Config: LLM pipeline uses LFM on llama.cpp (8080)
95 lines
3.4 KiB
Markdown
95 lines
3.4 KiB
Markdown
---
|
|
name: three-tier-evaluation-pipeline
|
|
version: 1.0.0
|
|
description: Run tasks through three evaluation tiers (Run, Trace, Thread) to produce
|
|
comprehensive reports with human-in-the-loop validation
|
|
inputs:
|
|
- query/input text for the task
|
|
- search results (for trace tier evaluation)
|
|
- evaluation criteria and thresholds
|
|
steps:
|
|
- 'Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph
|
|
engine) to generate initial outputs and results'
|
|
- 'Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined
|
|
criteria, generating detailed analysis and scoring'
|
|
- 'Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion,
|
|
approval, and iterative refinement of the output'
|
|
outputs:
|
|
- Final consolidated report combining results from all three tiers
|
|
- Detailed scores and metrics per tier
|
|
- Threaded discussion logs for human review and approval
|
|
tags: []
|
|
metadata:
|
|
source_repo: https://github.com/itszhaoziyan-n/AgentKit.git
|
|
extracted_at: ''
|
|
confidence: 0.95
|
|
---
|
|
|
|
# three-tier-evaluation-pipeline
|
|
|
|
Run tasks through three evaluation tiers (Run, Trace, Thread) to produce comprehensive reports with human-in-the-loop validation
|
|
|
|
## Setup
|
|
|
|
**Dependencies:**
|
|
|
|
```text
|
|
pip install langgraph>=0.3 langchain-core>=0.3 langchain-anthropic>=0.3 langfuse>=2.0 mcp[server]>=1.24 tenacity>=9.0 fastapi>=0.115 psycopg[binary]>=3.1
|
|
```
|
|
|
|
**Setup steps:**
|
|
|
|
1. Install dependencies with pip install -e .[dev]
|
|
1. Start infrastructure: docker compose up -d (PostgreSQL, Langfuse, MCP server)
|
|
1. Configure environment variables (DATABASE_URL, MCP_API_KEY, etc.)
|
|
1. Run the pipeline: python -m eval.runner --tiers run,thread,trace
|
|
|
|
## Key Files
|
|
|
|
- `eval/ - contains the three-tier evaluation logic`
|
|
- `scripts/ci_gate.py - threshold update and benchmark validation`
|
|
- `agentkit/runtime/ - LangGraph engine for state management and graph execution`
|
|
|
|
## Steps
|
|
|
|
1. Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results
|
|
2. Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring
|
|
3. Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output
|
|
|
|
## Implementation Details
|
|
|
|
```python
|
|
The eval/ directory implements Run, Trace, and Thread stages with configurable tiers
|
|
```
|
|
|
|
```python
|
|
Benchmark suite (40 test cases) validates the pipeline's reliability
|
|
```
|
|
|
|
```python
|
|
CI/CD workflows (ci.yml, eval-fast.yml, eval-trace.yml) orchestrate the evaluation pipeline
|
|
```
|
|
|
|
## Inputs
|
|
|
|
- query/input text for the task
|
|
- search results (for trace tier evaluation)
|
|
- evaluation criteria and thresholds
|
|
|
|
## Outputs
|
|
|
|
- Final consolidated report combining results from all three tiers
|
|
- Detailed scores and metrics per tier
|
|
- Threaded discussion logs for human review and approval
|
|
|
|
## Failure Modes
|
|
|
|
- If the Run tier fails (e.g., code execution error), the pipeline can retry but may produce incomplete outputs
|
|
- If the Trace tier LLM-as-Judge produces low-quality evaluations, the Thread tier may need additional human intervention
|
|
- Threshold mismatches between tiers could cause the pipeline to exit early or require manual adjustment
|
|
|
|
## Source
|
|
|
|
Extracted from: [https://github.com/itszhaoziyan-n/AgentKit.git](https://github.com/itszhaoziyan-n/AgentKit.git)
|
|
Confidence: 0.95
|