Add Skill: three-tier-evaluation-pipeline

Extracted from: https://github.com/itszhaoziyan-n/AgentKit.git
Score: 1.0

skills/langgraph-multi-agent-router/SKILL.md

@@ -1,96 +0,0 @@
----
-name: langgraph-multi-agent-router
-version: 1.0.0
-description: Orchestrate a multi-agent workflow where specialized agents collaborate
-  sequentially to gather information, structure it, and generate a final response
-inputs:
-- User query string (e.g., destination location)
-- BedrockModel configuration (model_id, temperature, top_p)
-- Pre-configured agents with specific system prompts and tool sets
-steps:
-- Researcher agent executes with system prompt to gather raw destination facts (places,
-  history, accommodations, food, web pages) using BedrockModel and available tools
-  (calculator, current_time)
-- Travel guide agent receives raw research output and structures it into labeled sections
-  (Must-See Attractions, Historical Highlights, Accommodation Areas, Culinary Delights,
-  Suggested Web Pages)
-- Writer agent receives the structured guide and synthesizes it into a professional
-  client-facing response with clear formatting and emphasis on the suggested web pages
-outputs:
-- Raw research data (JSON string containing gathered facts)
-- Structured guide content (markdown-formatted travel guide with labeled sections)
-- Final client response (professional formatted response ready for delivery)
-tags: []
-metadata:
-  source_repo: https://github.com/omerbsezer/Fast-LLM-Agent-MCP.git
-  extracted_at: ''
-  confidence: 0.95
----
-
-# langgraph-multi-agent-router
-
-Orchestrate a multi-agent workflow where specialized agents collaborate sequentially to gather information, structure it, and generate a final response
-
-## Setup
-
-**Dependencies:**
-
-```text
-pip install langchain langgraph bedrock-model pydantic
-```
-
-**Setup steps:**
-
-1. Install langchain and langgraph packages
-1. Configure BedrockModel with desired parameters (model_id, temperature, top_p)
-1. Create three Agent instances with specific system prompts and tool sets
-1. Initialize LangGraph with the agent chain and run the workflow
-
-## Key Files
-
-- `agents/langchain_langgraph/00-basic-agent/agent.py`
-- `agents/langchain_langgraph/02-agent-with-tools-structured-output/agent.py`
-- `agents/langchain_langgraph/08-langgraph-multi-agents-sequential-pattern/agent.py`
-
-## Steps
-
-1. Researcher agent executes with system prompt to gather raw destination facts (places, history, accommodations, food, web pages) using BedrockModel and available tools (calculator, current_time)
-2. Travel guide agent receives raw research output and structures it into labeled sections (Must-See Attractions, Historical Highlights, Accommodation Areas, Culinary Delights, Suggested Web Pages)
-3. Writer agent receives the structured guide and synthesizes it into a professional client-facing response with clear formatting and emphasis on the suggested web pages
-
-## Implementation Details
-
-```python
-Researcher agent uses BedrockModel with temperature=0.7, top_p=0.9 to gather destination facts
-```
-
-```python
-Travel guide agent receives raw output and formats into 5 labeled sections
-```
-
-```python
-Writer agent takes structured guide and writes professional client response
-```
-
-## Inputs
-
-- User query string (e.g., destination location)
-- BedrockModel configuration (model_id, temperature, top_p)
-- Pre-configured agents with specific system prompts and tool sets
-
-## Outputs
-
-- Raw research data (JSON string containing gathered facts)
-- Structured guide content (markdown-formatted travel guide with labeled sections)
-- Final client response (professional formatted response ready for delivery)
-
-## Failure Modes
-
-- Researcher agent fails to gather sufficient data or returns incomplete results
-- Travel guide agent fails to structure information correctly or produces unreadable output
-- Writer agent fails to format the final response properly or loses key information from the guide
-
-## Source
-
-Extracted from: [https://github.com/omerbsezer/Fast-LLM-Agent-MCP.git](https://github.com/omerbsezer/Fast-LLM-Agent-MCP.git)
-Confidence: 0.95

skills/langgraph-multi-agent-router/commands.md

@@ -1,6 +0,0 @@
-# Commands: langgraph-multi-agent-router
-
-## Available Commands
-
-- `/skill langgraph-multi-agent-router` — Load this skill
-- `/run langgraph-multi-agent-router` — Execute workflow

skills/langgraph-multi-agent-router/examples.md

@@ -1,10 +0,0 @@
-# Examples: langgraph-multi-agent-router
-
-## Usage Example
-
-```python
-# How to use this skill
-# Inputs: User query string (e.g., destination location), BedrockModel configuration (model_id, temperature, top_p), Pre-configured agents with specific system prompts and tool sets
-# Process: Researcher agent executes with system prompt to gather raw destination facts (places, history, accommodations, food, web pages) using BedrockModel and available tools (calculator, current_time) → Travel guide agent receives raw research output and structures it into labeled sections (Must-See Attractions, Historical Highlights, Accommodation Areas, Culinary Delights, Suggested Web Pages) → Writer agent receives the structured guide and synthesizes it into a professional client-facing response with clear formatting and emphasis on the suggested web pages
-# Outputs: Raw research data (JSON string containing gathered facts), Structured guide content (markdown-formatted travel guide with labeled sections), Final client response (professional formatted response ready for delivery)
-```

skills/langgraph-multi-agent-router/metadata.json

@@ -1,29 +0,0 @@
-{
-  "name": "langgraph-multi-agent-router",
-  "version": "1.0.0",
-  "goal": "Orchestrate a multi-agent workflow where specialized agents collaborate sequentially to gather information, structure it, and generate a final response",
-  "inputs": [
-    "User query string (e.g., destination location)",
-    "BedrockModel configuration (model_id, temperature, top_p)",
-    "Pre-configured agents with specific system prompts and tool sets"
-  ],
-  "steps": [
-    "Researcher agent executes with system prompt to gather raw destination facts (places, history, accommodations, food, web pages) using BedrockModel and available tools (calculator, current_time)",
-    "Travel guide agent receives raw research output and structures it into labeled sections (Must-See Attractions, Historical Highlights, Accommodation Areas, Culinary Delights, Suggested Web Pages)",
-    "Writer agent receives the structured guide and synthesizes it into a professional client-facing response with clear formatting and emphasis on the suggested web pages"
-  ],
-  "outputs": [
-    "Raw research data (JSON string containing gathered facts)",
-    "Structured guide content (markdown-formatted travel guide with labeled sections)",
-    "Final client response (professional formatted response ready for delivery)"
-  ],
-  "failure_modes": [
-    "Researcher agent fails to gather sufficient data or returns incomplete results",
-    "Travel guide agent fails to structure information correctly or produces unreadable output",
-    "Writer agent fails to format the final response properly or loses key information from the guide"
-  ],
-  "confidence": 0.95,
-  "explanation": "This workflow demonstrates a reusable multi-stage agent pattern where specialized agents collaborate in sequence. The Researcher agent gathers raw information using a domain-specific model, the Travel Guide agent structures that information into a consistent format, and the Writer agent synthesizes the final output. This pattern can be adapted to other domains (e.g., code generation, data analysis, research workflows) by swapping the agent types and system prompts while maintaining the same three-step structure.",
-  "source_repo": "https://github.com/omerbsezer/Fast-LLM-Agent-MCP.git",
-  "score": 1.0
-}

skills/three-tier-evaluation-pipeline/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: three-tier-evaluation-pipeline
+version: 1.0.0
+description: Run tasks through three evaluation tiers (Run, Trace, Thread) to produce
+  comprehensive reports with human-in-the-loop validation
+inputs:
+- query/input text for the task
+- search results (for trace tier evaluation)
+- evaluation criteria and thresholds
+steps:
+- 'Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph
+  engine) to generate initial outputs and results'
+- 'Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined
+  criteria, generating detailed analysis and scoring'
+- 'Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion,
+  approval, and iterative refinement of the output'
+outputs:
+- Final consolidated report combining results from all three tiers
+- Detailed scores and metrics per tier
+- Threaded discussion logs for human review and approval
+tags: []
+metadata:
+  source_repo: https://github.com/itszhaoziyan-n/AgentKit.git
+  extracted_at: ''
+  confidence: 0.95
+---
+
+# three-tier-evaluation-pipeline
+
+Run tasks through three evaluation tiers (Run, Trace, Thread) to produce comprehensive reports with human-in-the-loop validation
+
+## Setup
+
+**Dependencies:**
+
+```text
+pip install langgraph>=0.3 langchain-core>=0.3 langchain-anthropic>=0.3 langfuse>=2.0 mcp[server]>=1.24 tenacity>=9.0 fastapi>=0.115 psycopg[binary]>=3.1
+```
+
+**Setup steps:**
+
+1. Install dependencies with pip install -e .[dev]
+1. Start infrastructure: docker compose up -d (PostgreSQL, Langfuse, MCP server)
+1. Configure environment variables (DATABASE_URL, MCP_API_KEY, etc.)
+1. Run the pipeline: python -m eval.runner --tiers run,thread,trace
+
+## Key Files
+
+- `eval/ - contains the three-tier evaluation logic`
+- `scripts/ci_gate.py - threshold update and benchmark validation`
+- `agentkit/runtime/ - LangGraph engine for state management and graph execution`
+
+## Steps
+
+1. Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results
+2. Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring
+3. Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output
+
+## Implementation Details
+
+```python
+The eval/ directory implements Run, Trace, and Thread stages with configurable tiers
+```
+
+```python
+Benchmark suite (40 test cases) validates the pipeline's reliability
+```
+
+```python
+CI/CD workflows (ci.yml, eval-fast.yml, eval-trace.yml) orchestrate the evaluation pipeline
+```
+
+## Inputs
+
+- query/input text for the task
+- search results (for trace tier evaluation)
+- evaluation criteria and thresholds
+
+## Outputs
+
+- Final consolidated report combining results from all three tiers
+- Detailed scores and metrics per tier
+- Threaded discussion logs for human review and approval
+
+## Failure Modes
+
+- If the Run tier fails (e.g., code execution error), the pipeline can retry but may produce incomplete outputs
+- If the Trace tier LLM-as-Judge produces low-quality evaluations, the Thread tier may need additional human intervention
+- Threshold mismatches between tiers could cause the pipeline to exit early or require manual adjustment
+
+## Source
+
+Extracted from: [https://github.com/itszhaoziyan-n/AgentKit.git](https://github.com/itszhaoziyan-n/AgentKit.git)
+Confidence: 0.95

skills/three-tier-evaluation-pipeline/commands.md

@@ -0,0 +1,6 @@
+# Commands: three-tier-evaluation-pipeline
+
+## Available Commands
+
+- `/skill three-tier-evaluation-pipeline` — Load this skill
+- `/run three-tier-evaluation-pipeline` — Execute workflow

skills/three-tier-evaluation-pipeline/examples.md

@@ -0,0 +1,10 @@
+# Examples: three-tier-evaluation-pipeline
+
+## Usage Example
+
+```python
+# How to use this skill
+# Inputs: query/input text for the task, search results (for trace tier evaluation), evaluation criteria and thresholds
+# Process: Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results → Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring → Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output
+# Outputs: Final consolidated report combining results from all three tiers, Detailed scores and metrics per tier, Threaded discussion logs for human review and approval
+```

skills/three-tier-evaluation-pipeline/metadata.json

@@ -0,0 +1,29 @@
+{
+  "name": "three-tier-evaluation-pipeline",
+  "version": "1.0.0",
+  "goal": "Run tasks through three evaluation tiers (Run, Trace, Thread) to produce comprehensive reports with human-in-the-loop validation",
+  "inputs": [
+    "query/input text for the task",
+    "search results (for trace tier evaluation)",
+    "evaluation criteria and thresholds"
+  ],
+  "steps": [
+    "Step 1: Execute the main task using the Run tier of the evaluation pipeline (agentkit/runtime/LangGraph engine) to generate initial outputs and results",
+    "Step 2: Run the Trace tier where an LLM-as-Judge evaluates the output against defined criteria, generating detailed analysis and scoring",
+    "Step 3: Execute the Thread tier which facilitates human-in-the-loop discussion, approval, and iterative refinement of the output"
+  ],
+  "outputs": [
+    "Final consolidated report combining results from all three tiers",
+    "Detailed scores and metrics per tier",
+    "Threaded discussion logs for human review and approval"
+  ],
+  "failure_modes": [
+    "If the Run tier fails (e.g., code execution error), the pipeline can retry but may produce incomplete outputs",
+    "If the Trace tier LLM-as-Judge produces low-quality evaluations, the Thread tier may need additional human intervention",
+    "Threshold mismatches between tiers could cause the pipeline to exit early or require manual adjustment"
+  ],
+  "confidence": 0.95,
+  "explanation": "The AgentKit repository contains a production-ready three-tier evaluation pipeline (Run \u2192 Trace \u2192 Thread) that can be adapted to any task requiring multi-stage validation. This workflow uses LangGraph for orchestration and LangChain for tool integration, making it portable across different agent engineering scenarios. The pattern is reusable because it separates concerns into distinct stages with clear inputs/outputs, allowing teams to plug in different evaluation criteria or human reviewers as needed.",
+  "source_repo": "https://github.com/itszhaoziyan-n/AgentKit.git",
+  "score": 1.0
+}

skills/three-tier-evaluation-pipeline/tests.md

@@ -1,4 +1,4 @@
-# Tests: langgraph-multi-agent-router
+# Tests: three-tier-evaluation-pipeline
 
 ## Test Checklist