agent-skills/skills/three-tier-evaluation-pipeline/SKILL.md

name, version, description, inputs, steps, outputs, tags, metadata

name	version	description	inputs	steps	outputs	tags	metadata
three-tier-evaluation-pipeline	1.0.0	Run tasks through three evaluation tiers (Run, Trace, Thread) to produce comprehensive reports with human-in-the-loop validation	query/input text for the task search results (for trace tier evaluation) evaluation criteria and thresholds	Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output	Final consolidated report combining results from all three tiers Detailed scores and metrics per tier Threaded discussion logs for human review and approval		source_repo extracted_at confidence https://github.com/itszhaoziyan-n/AgentKit.git 0.95

three-tier-evaluation-pipeline

Run tasks through three evaluation tiers (Run, Trace, Thread) to produce comprehensive reports with human-in-the-loop validation

Setup

Dependencies:

pip install langgraph>=0.3 langchain-core>=0.3 langchain-anthropic>=0.3 langfuse>=2.0 mcp[server]>=1.24 tenacity>=9.0 fastapi>=0.115 psycopg[binary]>=3.1

Setup steps:

1. Install dependencies with pip install -e .[dev]
2. Start infrastructure: docker compose up -d (PostgreSQL, Langfuse, MCP server)
3. Configure environment variables (DATABASE_URL, MCP_API_KEY, etc.)
4. Run the pipeline: python -m eval.runner --tiers run,thread,trace

Key Files

• eval/ - contains the three-tier evaluation logic
• scripts/ci_gate.py - threshold update and benchmark validation
• agentkit/runtime/ - LangGraph engine for state management and graph execution

Steps

1. Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results
2. Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring
3. Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output

Implementation Details

The eval/ directory implements Run, Trace, and Thread stages with configurable tiers

Benchmark suite (40 test cases) validates the pipeline's reliability

CI/CD workflows (ci.yml, eval-fast.yml, eval-trace.yml) orchestrate the evaluation pipeline

Inputs

• query/input text for the task
• search results (for trace tier evaluation)
• evaluation criteria and thresholds

Outputs

• Final consolidated report combining results from all three tiers
• Detailed scores and metrics per tier
• Threaded discussion logs for human review and approval

Failure Modes

• If the Run tier fails (e.g., code execution error), the pipeline can retry but may produce incomplete outputs
• If the Trace tier LLM-as-Judge produces low-quality evaluations, the Thread tier may need additional human intervention
• Threshold mismatches between tiers could cause the pipeline to exit early or require manual adjustment

Source

Extracted from: https://github.com/itszhaoziyan-n/AgentKit.git Confidence: 0.95