Add task reliability ledger for publish-gate and Leonard scoring
This commit is contained in:
@@ -0,0 +1,41 @@
|
|||||||
|
# Task Reliability Ledger
|
||||||
|
|
||||||
|
**Purpose:** Make the ≥100 error-free executions rule and Leonard scoring **measurable**.
|
||||||
|
Until a task type has documented runs, human approval remains required before any client-facing publish for that task type.
|
||||||
|
|
||||||
|
**Error definition (default):** Output that invents facts/numbers, mis-assigns Evidence Tier, includes solution language in Layer 1/2, omits required Snapshot Date / recency, violates template structure, or would be unsafe to show a client without correction.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Publish-gate counters (automation eligibility)
|
||||||
|
|
||||||
|
| task_type | runs | errors | last_error_date | notes |
|
||||||
|
|-----------|------|--------|----------------|-------|
|
||||||
|
| gbp_snapshot_ingest_r1_r3 | 0 | 0 | — | Leonard execution card only |
|
||||||
|
| data_inventory_draft_l1a | 0 | 0 | — | Public baseline |
|
||||||
|
| threat_register_draft_l2 | 0 | 0 | — | Diagnosis only |
|
||||||
|
| review_reply_draft | 0 | 0 | — | Not auto-publish |
|
||||||
|
| gbp_field_edit_draft | 0 | 0 | — | Not auto-publish |
|
||||||
|
| qa_answer_draft | 0 | 0 | — | Not auto-publish |
|
||||||
|
|
||||||
|
**Rule:** A task_type may bypass human approval **only after** ≥100 completed runs with **zero errors** for that exact task_type, documented here. Reliability does not transfer across task types.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Leonard / agent scored runs (rubric 0–2 × 5 criteria)
|
||||||
|
|
||||||
|
| date | agent | client | task_type | score_total | zeros? | pass? | reviewer | notes |
|
||||||
|
|------|-------|--------|-----------|-------------|--------|-------|----------|-------|
|
||||||
|
| — | — | — | — | — | — | — | — | No scored runs yet |
|
||||||
|
|
||||||
|
**Pass:** no zeros and total ≥ 7/10.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to log a run
|
||||||
|
|
||||||
|
1. After human review of an agent draft, add one row to the appropriate table.
|
||||||
|
2. If the draft required material correction for an error (see definition), increment `errors` and set `last_error_date`.
|
||||||
|
3. Do not count unscored exploratory chat as a run.
|
||||||
|
|
||||||
|
**Last updated:** 2026-08-02
|
||||||
Reference in New Issue
Block a user