From 67d95200f4ebbb5671602e4230a4522eefbf229a Mon Sep 17 00:00:00 2001 From: Ty Date: Sun, 2 Aug 2026 22:07:26 +0000 Subject: [PATCH] =?UTF-8?q?Add=20Audit=20Playbook=20v1=20=E2=80=94=20extra?= =?UTF-8?q?cted=20from=20Overcome=20Fitness=20benchmark=20for=20Leonard=20?= =?UTF-8?q?training?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/agents/audit-playbook-v1.md | 139 +++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 docs/agents/audit-playbook-v1.md diff --git a/docs/agents/audit-playbook-v1.md b/docs/agents/audit-playbook-v1.md new file mode 100644 index 0000000..bef7840 --- /dev/null +++ b/docs/agents/audit-playbook-v1.md @@ -0,0 +1,139 @@ +# Audit Playbook v1 + +**Purpose:** Give any agent (especially Leonard) a fixed, Charter-compliant method for performing Layer 1 (Data Inventory) and Layer 2 (Threat Register) work. + +**Source:** Extracted from the Overcome Fitness benchmark process (Director + Leonard iterations). + +**Governing documents:** +- `docs/agents/agent-charter-v1.md` +- `docs/architecture/path-to-poc-sequencing.md` +- Templates in `docs/templates/` + +--- + +## Non-Negotiables + +1. **Evidence Model only** — Every fact is Tier 1 (Verified) or Tier 2 (Indicative). No unmarked claims. +2. **Agents draft. Humans approve.** Nothing is locked without human sign-off. +3. **No invented data, numbers, or threats.** +4. **Order is fixed:** Data Inventory (Layer 1) before Threat Register (Layer 2). Never reverse. +5. **No solution language in Layer 1 or 2.** Diagnosis only. No “fix this,” “claim that,” or workflow suggestions. +6. **Stale data must be flagged.** If a snapshot is older than ~30–60 days or its date is unknown, mark it and treat findings as provisional. +7. **Prefer the more constrained, evidence-bound position** when uncertain. + +--- + +## Layer 1 — Data Inventory Rules + +**Goal:** Record what exists, where it lives, and verification status. Nothing else. + +### Required sources (minimum) +- Google Business Profile (or clear statement that it is missing/unclear) +- Website +- Primary social channel(s) +- Booking / scheduling system (or explicit “not visible”) +- Reviews / reputation surfaces if observable + +### Required fields per source +- Access Status +- Data Range Available (use date ranges, not vague “12 months”) +- Snapshot Date +- Verification Tier (Tier 1 or Tier 2) +- Notes / Gaps +- Snapshot Location path + +### Rules +- Public observation = Tier 2 until owner access or stronger verification upgrades it. +- Do not claim “owner access” unless it is confirmed in the current process. +- Missing sources are documented as gaps, not ignored. +- Technical findings (404s, schema, NAP conflicts, etc.) require a dated snapshot. Old findings must be marked for re-verification. + +### Exit criteria (human must confirm) +- [ ] All accessible sources recorded +- [ ] Every source has Tier + Snapshot Date +- [ ] Gaps explicitly listed +- [ ] No analysis or threat language present +- [ ] Human sign-off obtained + +--- + +## Layer 2 — Threat Register Rules + +**Goal:** Identify discrete, evidence-backed failures in customer acquisition or retention. Diagnosis only. + +### Required fields per threat +- Threat ID (T-001, T-002…) +- Title (short, factual) +- Signal Source (must map to an inventory row) +- Evidence Snippet (specific, observable fact) +- Data Recency (must match or reference inventory snapshot dates) +- Estimated Severity (Critical / Major / Minor) +- Verification Tier +- Notes / Context (optional, still factual) +- Human Decision (Pending until review) + +### Rules +- One primary fact per threat. Do not bundle multiple unrelated gaps under one ID. +- Evidence Snippet must be concrete (what was observed), not a general claim. +- Severity is preliminary and Tier-linked. Do not escalate severity beyond what the evidence supports. +- Competitor comparisons are secondary context only, never the primary claim. +- If data is stale, say so in Data Recency or Notes. +- No recommendations, fixes, or next actions. + +### Severity guidance (use conservatively) +- **Critical:** Directly blocks or severely impairs local discovery / conversion and evidence is clear. +- **Major:** Material friction or visibility gap with solid observation. +- **Minor:** Real but lower immediate impact, or evidence is thin. + +When unsure, choose the lower severity. + +### Exit criteria (human must confirm) +- [ ] Every threat traces to the inventory +- [ ] Every threat has Evidence Snippet + Data Recency + Tier +- [ ] No solution language +- [ ] No invented threats +- [ ] Human has recorded decisions (Verified / Downgraded / Rejected) + +--- + +## Operating Pattern for Agents + +1. Read the Charter and this Playbook first. +2. Work only inside the current layer. +3. Produce drafts that match the templates. +4. List every change made when revising an existing document. +5. Surface blockers and stale data explicitly. +6. Stop at the human review gate. Do not self-lock. + +--- + +## Training / Evaluation Rubric (simple) + +Score a Leonard (or any agent) run 0–2 on each: + +| Criterion | 0 | 1 | 2 | +|-----------|---|---|---| +| Evidence grounding | Invented or vague | Mostly grounded | Fully concrete snippets | +| Tier discipline | Missing or wrong | Present but soft | Correct and consistent | +| Scope control | Drifted into solutions/requirements | Minor drift | Strict Layer 1/2 only | +| Recency honesty | Ignored stale data | Partially flagged | Explicitly flagged | +| Template fidelity | Structure broken | Mostly followed | Clean match | + +**Pass threshold:** No zeros, and total ≥ 7/10. +**Benchmark quality:** Total 9–10 with zero scope violations. + +--- + +## Current Benchmark Reference + +- Client: Overcome Fitness +- Files: `docs/clients/overcome-fitness/data-inventory-v1.md` and `threat-register-v1.md` +- Status: Draft (Benchmark) — not yet Locked; freshness gaps remain + +Future Leonard runs should be compared against this package and scored with the rubric above. + +--- + +**Version:** v1 +**Status:** Active for training +**Next upgrade trigger:** After first Locked client Layer 1+2 package exists and at least one Leonard scored run is complete.