feat(auditing): pipeline v1.1 — report generation + scripted gate + temporal delta
- report_generate.py: REPORT-final.md + VALIDATION.md from findings + raw capture; per-surface status from summary.surfaces_ok; Data Limitations - report_gate.py: §6 gate scripted (count/evidence/identity/material support); non-zero exit blocks delivery - audit_diff.py: before/after capture comparison -> DELTA.md - audit_pipeline.sh: 2 steps -> 4; collision-safe capture copy (re-runs preserve baseline); relative output dir resolved to absolute - beta-audit-process.md: locked v1.0 -> v1.1 (+ dated decision record) - all 9 runs of the 2026-08-15 batch regenerated + gated (9/9 PASS) - live end-to-end proof: Gilmore re-run, DELTA.md 0 changes (same day)
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# VeriPath Beta Audit Process
|
||||
|
||||
**Status:** Locked v1.0 — 2026-08-15, for beta use
|
||||
**Status:** Locked v1.1 — 2026-08-15, for beta use (v1.1 decision: `docs/decisions/2026-08-15-audit-pipeline-v1-1.md`)
|
||||
**Supersedes:** 2026-08-14 draft lock
|
||||
|
||||
## Purpose
|
||||
@@ -39,9 +39,13 @@ Collect:
|
||||
Run:
|
||||
|
||||
```text
|
||||
multi_scraper → audit_engine
|
||||
multi_scraper → audit_engine → report_generate → report_gate
|
||||
```
|
||||
|
||||
The pipeline writes the full required artifact set itself. `report_gate`
|
||||
fails the run (non-zero exit) if the §6 gate fails — a failing run does
|
||||
not deliver.
|
||||
|
||||
### 3. Human validation (reviewer decision loop)
|
||||
|
||||
The reviewer must resolve every engine finding to exactly one disposition.
|
||||
@@ -74,14 +78,21 @@ language or marketing urgency.
|
||||
|
||||
### 6. Pre-delivery consistency gate
|
||||
|
||||
All three checks pass before the report is sent. Any failure blocks delivery:
|
||||
Executed by `implementation/auditing/report_gate.py` as pipeline step 4;
|
||||
result stamped into `VALIDATION.md`. All four checks pass before the
|
||||
report is sent. Any failure blocks delivery:
|
||||
|
||||
1. **Count match:** the executive summary's stated number of findings equals the
|
||||
number of findings in the Findings section.
|
||||
2. **Evidence present:** every finding's cited evidence exists in the run's
|
||||
archived artifacts (raw capture or `findings.json`).
|
||||
archived artifacts (raw capture or `findings.json`). Absence-type findings
|
||||
(no website, no description, UTM parameters, unreachable surface) are
|
||||
re-derived from the raw capture instead of requiring quoted evidence.
|
||||
3. **Identity match:** report client name, location, and audit date match the
|
||||
intake record.
|
||||
4. **Material support:** each material finding is re-derived against the raw
|
||||
capture (phone cores across surfaces, hours day counts, rating, website
|
||||
presence, description/OG fields).
|
||||
|
||||
### 7. Delivery
|
||||
|
||||
@@ -99,6 +110,11 @@ Every run is archived under `docs/validation/<YYYY-MM-DD>-<slug>/` with:
|
||||
| `findings.md` | engine markdown |
|
||||
| `REPORT-final.md` | client-facing report (post-review) |
|
||||
| `VALIDATION.md` | run notes, surface results, reviewer decisions |
|
||||
| `DELTA.md` | before/after field comparison (re-runs only) |
|
||||
|
||||
Re-runs never overwrite the prior raw capture: the pipeline writes a
|
||||
timestamped sibling (`*__YYYYMMDD-HHMMSS.json`). The prior capture is the
|
||||
temporal baseline; `audit_diff.py` compares the two into `DELTA.md`.
|
||||
|
||||
Runs completed before 2026-08-15 are grandfathered as-is. Every run on or
|
||||
after this date must complete the set before delivery.
|
||||
@@ -129,7 +145,8 @@ implementation/auditing/
|
||||
The standard execution path is:
|
||||
|
||||
```text
|
||||
multi_scraper → audit_engine
|
||||
multi_scraper → audit_engine → report_generate → report_gate
|
||||
audit_diff (re-runs: baseline capture vs. new capture)
|
||||
```
|
||||
|
||||
## Deviations
|
||||
|
||||
Reference in New Issue
Block a user