inference_log must not store base64/binary image payloads #87

New Issue

2026-09-21T05:16:07Z

Ty commented

2026-09-21 05:16:07 +00:00

Problem

inference_log.request_payload stores the full vision chat messages, including base64 data-URLs of processed photos. That is binary image content in the audit log.

Live UAT evidence (5060 `deploy_slab_db`, 2026-09-21)

All three Call 1 rows hit the hard truncate and still never capture the text prompt:

id	slab_id	request_payload length	created_at
1	492ee5b4-…	200000 (cap)	04:17:31
2	f263f2d5-…	200000 (cap)	04:41:31
3	1083cda0-…	200000 (cap)	04:45:24

Images are placed before the text in the message (image_parts + [text]).
_log_call does json.dumps(request)[:200000] (backend/inference.py).
Result: log is mid-base64 only — no Classify / SYNCED TAXONOMY / taxonomy JSON in stored payload. Prompt audit is unusable; DB grows with image bytes.

Photos exist separately in slab_photos (e.g. latest Call 1 slab has photo fd913934-… seq=1, ~292KB processed_blob). Logging base64 duplicates that blob into inference_log.

Root cause

# call1 path
image_parts.append({"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64.b64encode(p[0]).decode()}"}})
messages = [{"role": "user", "content": image_parts + [{"type": "text", "text": text}]}]
_log_call(..., messages, ...)  # dumps full messages incl. data-URLs, then [:200000]

What it should do

Log a redacted / referential request payload suitable for prompt debugging:

Keep full text parts (seed/active prompt, taxonomy JSON, SKU/dims).
For each image part, store a reference only, not bytes/base64. Prefer stable identifiers already on the slab:
- photo_id (slab_photos.id)
- seq
- optional: width_px / height_px, is_calibration, byte length
- human label like photo seq=1 id=fd913934-….png (or original filename if/when that column exists — schema today has no orig_filename)
Never write data:image/...;base64,... into inference_log.
Do not rely on a raw byte cap that silently chops the prompt; after redaction the full text should fit without truncating the useful part.
Same rule for Call 2 if it attaches images.

Scope

backend/inference.py: _log_call and/or a sanitizer applied before insert (Call 1 and Call 2).
Wire photo ids into the call path (today Call 1 SELECTs only processed_blob — need id/seq for the log ref).
Tests: assert logged request has no data:image / base64 image payloads; assert text prompt + taxonomy present; assert image slots are refs.
Docs touch if needed: DATA-MODEL.md §9 / METADATA-66 — "model request" means auditable text + media refs, not binary.

Out of scope

Changing what is sent to the inference API (still needs images at call time).
Deleting historical bloated log rows (optional follow-up cleanup).
Adding original upload filenames (nice-to-have; not required to fix logging).

Acceptance

New Call 1/2 rows: request_payload contains full text prompt + taxonomy; zero base64 image data-URLs.
Each attached photo appears as an explicit ref (id and/or seq, printable label).
Unit test covers sanitizer / log shape.
Live UAT: one Call 1 on 5060; inspect latest inference_log row — text readable, payload size ≪ 200k for typical 1–2 photo slabs.

Filed from live UAT while reviewing Call 1 prompts (#86 follow-on): log was unusable for prompt review because images consumed the entire 200k cap.

## Problem `inference_log.request_payload` stores the full vision chat messages, including **base64 data-URLs of processed photos**. That is binary image content in the audit log. ### Live UAT evidence (5060 `deploy_slab_db`, 2026-09-21) All three Call 1 rows hit the hard truncate and still never capture the text prompt: | id | slab_id | request_payload length | created_at | |----|---------|------------------------|------------| | 1 | 492ee5b4-… | **200000** (cap) | 04:17:31 | | 2 | f263f2d5-… | **200000** (cap) | 04:41:31 | | 3 | 1083cda0-… | **200000** (cap) | 04:45:24 | - Images are placed **before** the text in the message (`image_parts + [text]`). - `_log_call` does `json.dumps(request)[:200000]` (`backend/inference.py`). - Result: log is mid-base64 only — **no** `Classify` / `SYNCED TAXONOMY` / taxonomy JSON in stored payload. Prompt audit is unusable; DB grows with image bytes. Photos exist separately in `slab_photos` (e.g. latest Call 1 slab has photo `fd913934-…` seq=1, ~292KB processed_blob). Logging base64 duplicates that blob into `inference_log`. ### Root cause ```python # call1 path image_parts.append({"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64.b64encode(p[0]).decode()}"}}) messages = [{"role": "user", "content": image_parts + [{"type": "text", "text": text}]}] _log_call(..., messages, ...) # dumps full messages incl. data-URLs, then [:200000] ``` ### What it should do Log a **redacted / referential** request payload suitable for prompt debugging: - Keep full **text** parts (seed/active prompt, taxonomy JSON, SKU/dims). - For each image part, store a **reference only**, not bytes/base64. Prefer stable identifiers already on the slab: - `photo_id` (slab_photos.id) - `seq` - optional: `width_px` / `height_px`, `is_calibration`, byte length - human label like `photo seq=1 id=fd913934-….png` (or original filename **if/when** that column exists — schema today has no orig_filename) - Never write `data:image/...;base64,...` into `inference_log`. - Do not rely on a raw byte cap that silently chops the prompt; after redaction the full text should fit without truncating the useful part. - Same rule for Call 2 if it attaches images. ### Scope - `backend/inference.py`: `_log_call` and/or a sanitizer applied before insert (Call 1 and Call 2). - Wire photo ids into the call path (today Call 1 SELECTs only `processed_blob` — need id/seq for the log ref). - Tests: assert logged request has no `data:image` / base64 image payloads; assert text prompt + taxonomy present; assert image slots are refs. - Docs touch if needed: `DATA-MODEL.md` §9 / METADATA-66 — "model request" means auditable text + media refs, not binary. ### Out of scope - Changing what is **sent** to the inference API (still needs images at call time). - Deleting historical bloated log rows (optional follow-up cleanup). - Adding original upload filenames (nice-to-have; not required to fix logging). ### Acceptance 1. New Call 1/2 rows: `request_payload` contains full text prompt + taxonomy; **zero** base64 image data-URLs. 2. Each attached photo appears as an explicit ref (id and/or seq, printable label). 3. Unit test covers sanitizer / log shape. 4. Live UAT: one Call 1 on 5060; inspect latest `inference_log` row — text readable, payload size ≪ 200k for typical 1–2 photo slabs. Filed from live UAT while reviewing Call 1 prompts (#86 follow-on): log was unusable for prompt review because images consumed the entire 200k cap.

grokbot_ty referenced this issue from a commit

2026-09-21 06:01:10 +00:00

fix(inference): #87 redact image bytes from inference_log

grokbot_ty referenced this issue from a commit

2026-09-21 06:01:41 +00:00

fix(inference): #87 redact image bytes from inference_log

grokbot_ty commented

2026-09-21 06:02:02 +00:00

Fixed in 41401a7.

_redact_request_for_log: data-URLs → image_ref (photo_id/seq/bytes/label); full text kept
Call 1/2 SELECT id+seq; wire path unchanged
Live UAT: new log row has LISTS + expert prompt, zero data:image, length ≪ 200k
Tests: 79 backend passed

Closing.

Fixed in `41401a7`. - `_redact_request_for_log`: data-URLs → `image_ref` (photo_id/seq/bytes/label); full text kept - Call 1/2 SELECT id+seq; wire path unchanged - Live UAT: new log row has LISTS + expert prompt, zero `data:image`, length ≪ 200k - Tests: 79 backend passed Closing.

grokbot_ty closed this issue

2026-09-21 06:02:02 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Ty_Tech/SlabUploader#87

inference_log must not store base64/binary image payloads #87