---
name: three-tier-evaluation-pipeline
version: 1.0.0
description: Run tasks through three evaluation tiers (Run, Trace, Thread) to produce
  comprehensive reports with human-in-the-loop validation
inputs:
- query/input text for the task
- search results (for trace tier evaluation)
- evaluation criteria and thresholds
steps:
- 'Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph
  engine) to generate initial outputs and results'
- 'Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined
  criteria, generating detailed analysis and scoring'
- 'Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion,
  approval, and iterative refinement of the output'
outputs:
- Final consolidated report combining results from all three tiers
- Detailed scores and metrics per tier
- Threaded discussion logs for human review and approval
tags: []
metadata:
  source_repo: https://github.com/itszhaoziyan-n/AgentKit.git
  extracted_at: ''
  confidence: 0.95
---

# three-tier-evaluation-pipeline

Run tasks through three evaluation tiers (Run, Trace, Thread) to produce comprehensive reports with human-in-the-loop validation

## Setup

**Dependencies:**

```text
pip install langgraph>=0.3 langchain-core>=0.3 langchain-anthropic>=0.3 langfuse>=2.0 mcp[server]>=1.24 tenacity>=9.0 fastapi>=0.115 psycopg[binary]>=3.1
```

**Setup steps:**

1. Install dependencies with pip install -e .[dev]
1. Start infrastructure: docker compose up -d (PostgreSQL, Langfuse, MCP server)
1. Configure environment variables (DATABASE_URL, MCP_API_KEY, etc.)
1. Run the pipeline: python -m eval.runner --tiers run,thread,trace

## Key Files

- `eval/ - contains the three-tier evaluation logic`
- `scripts/ci_gate.py - threshold update and benchmark validation`
- `agentkit/runtime/ - LangGraph engine for state management and graph execution`

## Steps

1. Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results
2. Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring
3. Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output

## Implementation Details

```python
The eval/ directory implements Run, Trace, and Thread stages with configurable tiers
```

```python
Benchmark suite (40 test cases) validates the pipeline's reliability
```

```python
CI/CD workflows (ci.yml, eval-fast.yml, eval-trace.yml) orchestrate the evaluation pipeline
```

## Inputs

- query/input text for the task
- search results (for trace tier evaluation)
- evaluation criteria and thresholds

## Outputs

- Final consolidated report combining results from all three tiers
- Detailed scores and metrics per tier
- Threaded discussion logs for human review and approval

## Failure Modes

- If the Run tier fails (e.g., code execution error), the pipeline can retry but may produce incomplete outputs
- If the Trace tier LLM-as-Judge produces low-quality evaluations, the Thread tier may need additional human intervention
- Threshold mismatches between tiers could cause the pipeline to exit early or require manual adjustment

## Source

Extracted from: [https://github.com/itszhaoziyan-n/AgentKit.git](https://github.com/itszhaoziyan-n/AgentKit.git)
Confidence: 0.95