From ffe401f8b5181cd6a7e57e2ae145763026523fbb Mon Sep 17 00:00:00 2001 From: Ty Date: Mon, 3 Aug 2026 05:13:38 +0000 Subject: [PATCH] Add task reliability ledger for publish-gate and Leonard scoring --- docs/operations/task-reliability-ledger.md | 41 ++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 docs/operations/task-reliability-ledger.md diff --git a/docs/operations/task-reliability-ledger.md b/docs/operations/task-reliability-ledger.md new file mode 100644 index 0000000..a80b3ef --- /dev/null +++ b/docs/operations/task-reliability-ledger.md @@ -0,0 +1,41 @@ +# Task Reliability Ledger + +**Purpose:** Make the ≥100 error-free executions rule and Leonard scoring **measurable**. +Until a task type has documented runs, human approval remains required before any client-facing publish for that task type. + +**Error definition (default):** Output that invents facts/numbers, mis-assigns Evidence Tier, includes solution language in Layer 1/2, omits required Snapshot Date / recency, violates template structure, or would be unsafe to show a client without correction. + +--- + +## Publish-gate counters (automation eligibility) + +| task_type | runs | errors | last_error_date | notes | +|-----------|------|--------|----------------|-------| +| gbp_snapshot_ingest_r1_r3 | 0 | 0 | — | Leonard execution card only | +| data_inventory_draft_l1a | 0 | 0 | — | Public baseline | +| threat_register_draft_l2 | 0 | 0 | — | Diagnosis only | +| review_reply_draft | 0 | 0 | — | Not auto-publish | +| gbp_field_edit_draft | 0 | 0 | — | Not auto-publish | +| qa_answer_draft | 0 | 0 | — | Not auto-publish | + +**Rule:** A task_type may bypass human approval **only after** ≥100 completed runs with **zero errors** for that exact task_type, documented here. Reliability does not transfer across task types. + +--- + +## Leonard / agent scored runs (rubric 0–2 × 5 criteria) + +| date | agent | client | task_type | score_total | zeros? | pass? | reviewer | notes | +|------|-------|--------|-----------|-------------|--------|-------|----------|-------| +| — | — | — | — | — | — | — | — | No scored runs yet | + +**Pass:** no zeros and total ≥ 7/10. + +--- + +## How to log a run + +1. After human review of an agent draft, add one row to the appropriate table. +2. If the draft required material correction for an error (see definition), increment `errors` and set `last_error_date`. +3. Do not count unscored exploratory chat as a run. + +**Last updated:** 2026-08-02