140 lines
5.3 KiB
Markdown
140 lines
5.3 KiB
Markdown
# Audit Playbook v1
|
||
|
||
**Purpose:** Give any agent (especially Leonard) a fixed, Charter-compliant method for performing Layer 1 (Data Inventory) and Layer 2 (Threat Register) work.
|
||
|
||
**Source:** Extracted from the Overcome Fitness benchmark process (Director + Leonard iterations).
|
||
|
||
**Governing documents:**
|
||
- `docs/agents/agent-charter-v1.md`
|
||
- `docs/architecture/path-to-poc-sequencing.md`
|
||
- Templates in `docs/templates/`
|
||
|
||
---
|
||
|
||
## Non-Negotiables
|
||
|
||
1. **Evidence Model only** — Every fact is Tier 1 (Verified) or Tier 2 (Indicative). No unmarked claims.
|
||
2. **Agents draft. Humans approve.** Nothing is locked without human sign-off.
|
||
3. **No invented data, numbers, or threats.**
|
||
4. **Order is fixed:** Data Inventory (Layer 1) before Threat Register (Layer 2). Never reverse.
|
||
5. **No solution language in Layer 1 or 2.** Diagnosis only. No “fix this,” “claim that,” or workflow suggestions.
|
||
6. **Stale data must be flagged.** If a snapshot is older than ~30–60 days or its date is unknown, mark it and treat findings as provisional.
|
||
7. **Prefer the more constrained, evidence-bound position** when uncertain.
|
||
|
||
---
|
||
|
||
## Layer 1 — Data Inventory Rules
|
||
|
||
**Goal:** Record what exists, where it lives, and verification status. Nothing else.
|
||
|
||
### Required sources (minimum)
|
||
- Google Business Profile (or clear statement that it is missing/unclear)
|
||
- Website
|
||
- Primary social channel(s)
|
||
- Booking / scheduling system (or explicit “not visible”)
|
||
- Reviews / reputation surfaces if observable
|
||
|
||
### Required fields per source
|
||
- Access Status
|
||
- Data Range Available (use date ranges, not vague “12 months”)
|
||
- Snapshot Date
|
||
- Verification Tier (Tier 1 or Tier 2)
|
||
- Notes / Gaps
|
||
- Snapshot Location path
|
||
|
||
### Rules
|
||
- Public observation = Tier 2 until owner access or stronger verification upgrades it.
|
||
- Do not claim “owner access” unless it is confirmed in the current process.
|
||
- Missing sources are documented as gaps, not ignored.
|
||
- Technical findings (404s, schema, NAP conflicts, etc.) require a dated snapshot. Old findings must be marked for re-verification.
|
||
|
||
### Exit criteria (human must confirm)
|
||
- [ ] All accessible sources recorded
|
||
- [ ] Every source has Tier + Snapshot Date
|
||
- [ ] Gaps explicitly listed
|
||
- [ ] No analysis or threat language present
|
||
- [ ] Human sign-off obtained
|
||
|
||
---
|
||
|
||
## Layer 2 — Threat Register Rules
|
||
|
||
**Goal:** Identify discrete, evidence-backed failures in customer acquisition or retention. Diagnosis only.
|
||
|
||
### Required fields per threat
|
||
- Threat ID (T-001, T-002…)
|
||
- Title (short, factual)
|
||
- Signal Source (must map to an inventory row)
|
||
- Evidence Snippet (specific, observable fact)
|
||
- Data Recency (must match or reference inventory snapshot dates)
|
||
- Estimated Severity (Critical / Major / Minor)
|
||
- Verification Tier
|
||
- Notes / Context (optional, still factual)
|
||
- Human Decision (Pending until review)
|
||
|
||
### Rules
|
||
- One primary fact per threat. Do not bundle multiple unrelated gaps under one ID.
|
||
- Evidence Snippet must be concrete (what was observed), not a general claim.
|
||
- Severity is preliminary and Tier-linked. Do not escalate severity beyond what the evidence supports.
|
||
- Competitor comparisons are secondary context only, never the primary claim.
|
||
- If data is stale, say so in Data Recency or Notes.
|
||
- No recommendations, fixes, or next actions.
|
||
|
||
### Severity guidance (use conservatively)
|
||
- **Critical:** Directly blocks or severely impairs local discovery / conversion and evidence is clear.
|
||
- **Major:** Material friction or visibility gap with solid observation.
|
||
- **Minor:** Real but lower immediate impact, or evidence is thin.
|
||
|
||
When unsure, choose the lower severity.
|
||
|
||
### Exit criteria (human must confirm)
|
||
- [ ] Every threat traces to the inventory
|
||
- [ ] Every threat has Evidence Snippet + Data Recency + Tier
|
||
- [ ] No solution language
|
||
- [ ] No invented threats
|
||
- [ ] Human has recorded decisions (Verified / Downgraded / Rejected)
|
||
|
||
---
|
||
|
||
## Operating Pattern for Agents
|
||
|
||
1. Read the Charter and this Playbook first.
|
||
2. Work only inside the current layer.
|
||
3. Produce drafts that match the templates.
|
||
4. List every change made when revising an existing document.
|
||
5. Surface blockers and stale data explicitly.
|
||
6. Stop at the human review gate. Do not self-lock.
|
||
|
||
---
|
||
|
||
## Training / Evaluation Rubric (simple)
|
||
|
||
Score a Leonard (or any agent) run 0–2 on each:
|
||
|
||
| Criterion | 0 | 1 | 2 |
|
||
|-----------|---|---|---|
|
||
| Evidence grounding | Invented or vague | Mostly grounded | Fully concrete snippets |
|
||
| Tier discipline | Missing or wrong | Present but soft | Correct and consistent |
|
||
| Scope control | Drifted into solutions/requirements | Minor drift | Strict Layer 1/2 only |
|
||
| Recency honesty | Ignored stale data | Partially flagged | Explicitly flagged |
|
||
| Template fidelity | Structure broken | Mostly followed | Clean match |
|
||
|
||
**Pass threshold:** No zeros, and total ≥ 7/10.
|
||
**Benchmark quality:** Total 9–10 with zero scope violations.
|
||
|
||
---
|
||
|
||
## Current Benchmark Reference
|
||
|
||
- Client: Overcome Fitness
|
||
- Files: `docs/clients/overcome-fitness/data-inventory-v1.md` and `threat-register-v1.md`
|
||
- Status: Draft (Benchmark) — not yet Locked; freshness gaps remain
|
||
|
||
Future Leonard runs should be compared against this package and scored with the rubric above.
|
||
|
||
---
|
||
|
||
**Version:** v1
|
||
**Status:** Active for training
|
||
**Next upgrade trigger:** After first Locked client Layer 1+2 package exists and at least one Leonard scored run is complete.
|