From be0c0b1cf0fe9daafb60572a8c4ba48234da37cf Mon Sep 17 00:00:00 2001 From: Ty Date: Mon, 3 Aug 2026 05:13:30 +0000 Subject: [PATCH] Playbook: Layer 1a/1b, competitor evidence rules, score ledger pointer --- docs/agents/audit-playbook-v1.md | 63 ++++++++++++++++++++++---------- 1 file changed, 44 insertions(+), 19 deletions(-) diff --git a/docs/agents/audit-playbook-v1.md b/docs/agents/audit-playbook-v1.md index 19c9134..f8259c1 100644 --- a/docs/agents/audit-playbook-v1.md +++ b/docs/agents/audit-playbook-v1.md @@ -1,4 +1,4 @@ -# Audit Playbook v1 +# Audit Playbook v1.1 **Purpose:** Give any agent (especially Leonard) a fixed, Charter-compliant method for performing Layer 1 (Data Inventory) and Layer 2 (Threat Register) work. @@ -14,6 +14,8 @@ - Leonard execution card (minimal reasoning): `docs/agents/leonard-gbp-execution-card.md` - HTML form (fill → copy/paste to Leonard): `tools/gbp-snapshot-form.html` +**Reliability / scores:** Record scored runs and publish-gate counters in `docs/operations/task-reliability-ledger.md`. + --- ## Non-Negotiables @@ -21,45 +23,59 @@ 1. **Evidence Model only** — Every fact is Tier 1 (Verified) or Tier 2 (Indicative). No unmarked claims. 2. **Agents draft. Humans approve.** Nothing is locked without human sign-off. 3. **No invented data, numbers, or threats.** -4. **Order is fixed:** Data Inventory (Layer 1) before Threat Register (Layer 2). Never reverse. -5. **No solution language in Layer 1 or 2.** Diagnosis only. No “fix this,” “claim that,” or workflow suggestions. +4. **Order is fixed:** Data Inventory (Layer 1a, then 1b when available) before Threat Register (Layer 2). Never reverse. +5. **No solution language in Layer 1 or 2.** Diagnosis only. No "fix this," "claim that," or workflow suggestions. 6. **Stale data must be flagged.** If a snapshot is older than ~30–60 days or its date is unknown, mark it and treat findings as provisional. 7. **Prefer the more constrained, evidence-bound position** when uncertain. --- -## Layer 1 — Data Inventory Rules +## Layer 1a — Public Data Inventory Rules (cold audit) -**Goal:** Record what exists, where it lives, and verification status. Nothing else. +**Goal:** Record what is publicly observable, where it lives, and verification status. Nothing else. ### Required sources (minimum) - Google Business Profile / GBP (or clear statement that it is missing/unclear) - Website -- Primary social channel(s) -- Booking / scheduling system (or explicit “not visible”) +- Primary social channel(s) visible without owner login +- Booking / scheduling surface as visible publicly (or explicit "not visible") - Reviews / reputation surfaces if observable ### Required fields per source -- Access Status -- Data Range Available (use date ranges, not vague “12 months”) +- Access Status (public / pending owner / denied) +- Data Range Available (use date ranges, not vague "12 months") - Snapshot Date -- Verification Tier (Tier 1 or Tier 2) +- Verification Tier (almost always Tier 2 for pure public observation) - Notes / Gaps - Snapshot Location path ### Rules -- Public observation = Tier 2 until owner access or stronger verification upgrades it. -- Do not claim “owner access” unless it is confirmed in the current process. +- Public observation = **Tier 2** until owner access or stronger verification upgrades it. +- Do not claim "owner access" unless confirmed in the current process. - Missing sources are documented as gaps, not ignored. - Technical findings (404s, schema, NAP conflicts, etc.) require a dated snapshot. Old findings must be marked for re-verification. - **GBP without owner access:** use human snapshot via `tools/gbp-snapshot-form.html` (or markdown intake). Leonard maps via the execution card only — no free-form GBP reasoning. -### Exit criteria (human must confirm) -- [ ] All accessible sources recorded +### Exit criteria (human must confirm) — Layer 1a +- [ ] All accessible **public** sources recorded - [ ] Every source has Tier + Snapshot Date - [ ] Gaps explicitly listed - [ ] No analysis or threat language present - [ ] Human sign-off obtained +- [ ] Status set to Locked (1a) or Draft Benchmark with blockers listed + +Cold audits may stop here. Do not claim "Layer 1 complete (engagement-ready)" without 1b or explicit acceptance of Tier 2 ceiling. + +--- + +## Layer 1b — Owner-access Inventory (engaged) + +**Goal:** Upgrade inventory with owner-gated sources; enable Tier 1 where justified. + +### Additional rules +- Owner GBP / social / booking access documented per source +- History gaps remain gaps (no invented continuity) +- Exit only when human confirms owner-gated fields and tier upgrades --- @@ -82,11 +98,17 @@ - One primary fact per threat. Do not bundle multiple unrelated gaps under one ID. - Evidence Snippet must be concrete (what was observed), not a general claim. - Severity is preliminary and Tier-linked. Do not escalate severity beyond what the evidence supports. -- Competitor comparisons are secondary context only, never the primary claim. - If data is stale, say so in Data Recency or Notes. - No recommendations, fixes, or next actions. - **GBP-related threats:** only create/update via execution-card trigger rules R1–R3 when ingesting a GBP snapshot. +### Competitor evidence rules (mandatory) +- Competitor comparisons are **secondary context only**, never the primary claim of a threat. +- Any fact about a competitor's business (hours, rating, booking path, review count, etc.) is **Tier 2 by default**. Tier 1 is not available without verification from a source we control (almost never). +- Every competitor claim must include: **Snapshot Date** and **source URL or observation method**. +- A wrong claim about a third party is a **higher reputational risk** than a wrong claim about the client. Prefer omit over weak comparison. +- Client-facing comparative briefs must not treat competitor Tier 2 claims as proven superiority; use relative, provisional language only. + ### Severity guidance (use conservatively) - **Critical:** Directly blocks or severely impairs local discovery / conversion and evidence is clear. - **Major:** Material friction or visibility gap with solid observation. @@ -99,6 +121,7 @@ When unsure, choose the lower severity. - [ ] Every threat has Evidence Snippet + Data Recency + Tier - [ ] No solution language - [ ] No invented threats +- [ ] Competitor facts (if any) follow competitor evidence rules - [ ] Human has recorded decisions (Verified / Downgraded / Rejected) --- @@ -116,7 +139,7 @@ When unsure, choose the lower severity. 1. Human fills `tools/gbp-snapshot-form.html` (or markdown intake). 2. Human pastes output to Leonard with: follow `docs/agents/leonard-gbp-execution-card.md` exactly. 3. Leonard maps fields and fires only R1–R3. -4. Human reviews; Director may score against the rubric below. +4. Human reviews; Director may score against the rubric below and **records the score in the task reliability ledger**. --- @@ -135,18 +158,20 @@ Score a Leonard (or any agent) run 0–2 on each: **Pass threshold:** No zeros, and total ≥ 7/10. **Benchmark quality:** Total 9–10 with zero scope violations. +Record each scored run in `docs/operations/task-reliability-ledger.md`. + --- ## Current Benchmark Reference - Client: Overcome Fitness - Files: `docs/clients/overcome-fitness/data-inventory-v1.md` and `threat-register-v1.md` -- Status: Draft (Benchmark) — not yet Locked; GBP completeness still open +- Status: Draft (Benchmark) — not yet Locked; GBP completeness still open; treat as **Layer 1a path**, not engagement-complete Layer 1 Future Leonard runs should be compared against this package and scored with the rubric above. --- -**Version:** v1 +**Version:** v1.1 **Status:** Active for training -**Next upgrade trigger:** After first Locked client Layer 1+2 package exists and at least one Leonard scored run is complete. +**Next upgrade trigger:** After first Locked client Layer 1a+2 package exists and at least one Leonard scored run is recorded in the ledger.