Add Audit Playbook v1 — extracted from Overcome Fitness benchmark for Leonard training

2026-08-02 22:07:26 +00:00
parent d605bdaed0
commit 67d95200f4
1 changed files with 139 additions and 0 deletions
@@ -0,0 +1,139 @@
+# Audit Playbook v1
+
+**Purpose:** Give any agent (especially Leonard) a fixed, Charter-compliant method for performing Layer 1 (Data Inventory) and Layer 2 (Threat Register) work.
+
+**Source:** Extracted from the Overcome Fitness benchmark process (Director + Leonard iterations).
+
+**Governing documents:**
+- `docs/agents/agent-charter-v1.md`
+- `docs/architecture/path-to-poc-sequencing.md`
+- Templates in `docs/templates/`
+
+---
+
+## Non-Negotiables
+
+1. **Evidence Model only** — Every fact is Tier 1 (Verified) or Tier 2 (Indicative). No unmarked claims.
+2. **Agents draft. Humans approve.** Nothing is locked without human sign-off.
+3. **No invented data, numbers, or threats.**
+4. **Order is fixed:** Data Inventory (Layer 1) before Threat Register (Layer 2). Never reverse.
+5. **No solution language in Layer 1 or 2.** Diagnosis only. No “fix this,” “claim that,” or workflow suggestions.
+6. **Stale data must be flagged.** If a snapshot is older than ~30–60 days or its date is unknown, mark it and treat findings as provisional.
+7. **Prefer the more constrained, evidence-bound position** when uncertain.
+
+---
+
+## Layer 1 — Data Inventory Rules
+
+**Goal:** Record what exists, where it lives, and verification status. Nothing else.
+
+### Required sources (minimum)
+- Google Business Profile (or clear statement that it is missing/unclear)
+- Website
+- Primary social channel(s)
+- Booking / scheduling system (or explicit “not visible”)
+- Reviews / reputation surfaces if observable
+
+### Required fields per source
+- Access Status
+- Data Range Available (use date ranges, not vague “12 months”)
+- Snapshot Date
+- Verification Tier (Tier 1 or Tier 2)
+- Notes / Gaps
+- Snapshot Location path
+
+### Rules
+- Public observation = Tier 2 until owner access or stronger verification upgrades it.
+- Do not claim “owner access” unless it is confirmed in the current process.
+- Missing sources are documented as gaps, not ignored.
+- Technical findings (404s, schema, NAP conflicts, etc.) require a dated snapshot. Old findings must be marked for re-verification.
+
+### Exit criteria (human must confirm)
+- [ ] All accessible sources recorded
+- [ ] Every source has Tier + Snapshot Date
+- [ ] Gaps explicitly listed
+- [ ] No analysis or threat language present
+- [ ] Human sign-off obtained
+
+---
+
+## Layer 2 — Threat Register Rules
+
+**Goal:** Identify discrete, evidence-backed failures in customer acquisition or retention. Diagnosis only.
+
+### Required fields per threat
+- Threat ID (T-001, T-002…)
+- Title (short, factual)
+- Signal Source (must map to an inventory row)
+- Evidence Snippet (specific, observable fact)
+- Data Recency (must match or reference inventory snapshot dates)
+- Estimated Severity (Critical / Major / Minor)
+- Verification Tier
+- Notes / Context (optional, still factual)
+- Human Decision (Pending until review)
+
+### Rules
+- One primary fact per threat. Do not bundle multiple unrelated gaps under one ID.
+- Evidence Snippet must be concrete (what was observed), not a general claim.
+- Severity is preliminary and Tier-linked. Do not escalate severity beyond what the evidence supports.
+- Competitor comparisons are secondary context only, never the primary claim.
+- If data is stale, say so in Data Recency or Notes.
+- No recommendations, fixes, or next actions.
+
+### Severity guidance (use conservatively)
+- **Critical:** Directly blocks or severely impairs local discovery / conversion and evidence is clear.
+- **Major:** Material friction or visibility gap with solid observation.
+- **Minor:** Real but lower immediate impact, or evidence is thin.
+
+When unsure, choose the lower severity.
+
+### Exit criteria (human must confirm)
+- [ ] Every threat traces to the inventory
+- [ ] Every threat has Evidence Snippet + Data Recency + Tier
+- [ ] No solution language
+- [ ] No invented threats
+- [ ] Human has recorded decisions (Verified / Downgraded / Rejected)
+
+---
+
+## Operating Pattern for Agents
+
+1. Read the Charter and this Playbook first.
+2. Work only inside the current layer.
+3. Produce drafts that match the templates.
+4. List every change made when revising an existing document.
+5. Surface blockers and stale data explicitly.
+6. Stop at the human review gate. Do not self-lock.
+
+---
+
+## Training / Evaluation Rubric (simple)
+
+Score a Leonard (or any agent) run 0–2 on each:
+
+| Criterion | 0 | 1 | 2 |
+|-----------|---|---|---|
+| Evidence grounding | Invented or vague | Mostly grounded | Fully concrete snippets |
+| Tier discipline | Missing or wrong | Present but soft | Correct and consistent |
+| Scope control | Drifted into solutions/requirements | Minor drift | Strict Layer 1/2 only |
+| Recency honesty | Ignored stale data | Partially flagged | Explicitly flagged |
+| Template fidelity | Structure broken | Mostly followed | Clean match |
+
+**Pass threshold:** No zeros, and total ≥ 7/10.  
+**Benchmark quality:** Total 9–10 with zero scope violations.
+
+---
+
+## Current Benchmark Reference
+
+- Client: Overcome Fitness
+- Files: `docs/clients/overcome-fitness/data-inventory-v1.md` and `threat-register-v1.md`
+- Status: Draft (Benchmark) — not yet Locked; freshness gaps remain
+
+Future Leonard runs should be compared against this package and scored with the rubric above.
+
+---
+
+**Version:** v1  
+**Status:** Active for training  
+**Next upgrade trigger:** After first Locked client Layer 1+2 package exists and at least one Leonard scored run is complete.