From 444cee7f76bc6a495e1fcec3af2c789e98af2a2b Mon Sep 17 00:00:00 2001 From: Grok Bot Date: Sat, 17 Oct 2026 15:27:14 -0500 Subject: [PATCH] Add requirements --- _pipeline/docs/REQUIREMENTS.md | 127 +++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 _pipeline/docs/REQUIREMENTS.md diff --git a/_pipeline/docs/REQUIREMENTS.md b/_pipeline/docs/REQUIREMENTS.md new file mode 100644 index 0000000..d74dd42 --- /dev/null +++ b/_pipeline/docs/REQUIREMENTS.md @@ -0,0 +1,127 @@ +# Audiobook pipeline: requirements (draft, owner: Ty) +Status: stack items 1-5 agreed by Ty; items 6-8 follow from his standards (ACX spec, code checks, repo output). Pilot text stages in progress. Each item is confirmed (C), proposed (P) or open (Q). + +## Product +- R1 (C) Output is a listenable audiobook for each PDF in Ty_Tech/old-books, now and added later. Ty listens, he does not read. +- R2 (C) Voice: male, warm, clear, conversational, unhurried, not dramatic. References: The Four Agreements narrator, Covey (7 Habits), @RockportAI. +- R3 (P) Stock voice only. No cloning of real people. +- R4 (C) Default mode: straight read with Audible-style intro and outro. +- R5 (C) Three switches, per book and per chapter: chapter preview, chapter-end summary, mid-chapter concept summary. +- R6 (C) Every inserted part has an audio cue before and after. +- R7 (C) Figures support the concept only. Drop technical detail useless over headphones. Large tables get a spoken takeaway, never a read-through. +- R8 (C) Every skipped figure is listed in a text file in the repo. +- R9 (C) Audio and text live in the repo next to each book. Storage is not a constraint. +- R10 (C) Track license status per book. Nothing is published by accident. + +- R13 (C) Audible/ACX is the rubric for audio and structure. Output must meet ACX specs: MP3 CBR >=192 kbps, 44.1 kHz, all mono or all stereo, RMS -23 to -18 dB, peak <= -3 dB, noise floor <= -60 dB, room tone 0.5-1 s head and 1-5 s tail, one chapter per file under 120 min, section header spoken, opening and closing credits, 1-5 min retail sample. Exception: the 'recorded by a human' rule (Ty acknowledges). Code checks every file against this before Ty hears it. +- R14 (C) Narrator touchstones are researched and measured, not assumed (see NARRATOR_STYLE.md, NARRATOR_MEASURED.md). + +## Source +- R11 (C) Sources are scans. Needs OCR, plus removal of page numbers, headers and scan errors. +- R12 (C, measured) Three PDFs have no text layer. Some Home Study Course units have thin text layers. + +## Engineering rules (Ty) +- E1 (C) Never use inference where deterministic code can do the task. +- E2 (C) The stack is an unattended service on a LAN host, not manual work by the agent. +- E3 (C) Self-hosted, mostly open source. +- E4 (C) TTS runs in a restart-always Docker container. Few workloads at the OS level. +- E5 (C) If inference can run on the LAN, prefer a LAN host over the agent's computer. tyOmarchy was suggested. +- E6 (C) Before using a GPU that holds a model, decommission that model's systemd service first. +- E7 (C) Ty is hands-off. Access goes through congfbot. + +## Process +- P1 (C) Stack items 1-5 approved: ingest, clean-up (deterministic first, model may propose fixes for unresolved words, all logged), structure (Audible style, chapter list shown to Ty once per book), narration script (author text verbatim, added parts labelled, summaries default OFF, straight Audible-style read default), voice synthesis (Chatterbox in restart-always Docker, fixed stock voice, pilot comparison vs two open voices plus Grok Voice reference, judged by Ty against measured narrator profiles). Items 6-8 (assembly, quality checks, repo output) follow from R13, R9, P3. +- P2 (C) Pilot: one short chapter of one short book. Ty listens before anything else runs. +- P4 (C) No micro-management: bring complete plans for one approval. One topic per message, a recommendation with each question, no implementation internals. +- P3 (C) Never call a book done until audio is checked against the recipe. + +## Open +- Q1 Resolved by Ty ('good'): figures extracted deterministically; a model drafts the spoken sentence from caption and nearby text; skipped figures listed in skipped_figures.md. +- Q2 Host: which machine is GPU1 and which is tyOmarchy? TCP from the agent computer to the GPU hosts is blocked by Tailscale ACL. +- Q3 Which LLM endpoint does summarizing? +- Q4 Resolved: ingest approved (Ty moved on after item 1 feedback). + +## Hosting facts (from congfbot, verified read-only 2026-10-16) +- GPU1 = RTX 5060 Ti 16 GiB on ubudual5060 (.9, LAN 192.168.1.202). Unit qwen-27b-IQ3-XSS--gpu1.service, port 8082, uses ~14.4 GiB. ~1.4 GiB free. Chatterbox will not fit beside it. Using GPU1 means stopping or shrinking that unit: Ty's decision. +- Docker 29.8.0 and nvidia-container-toolkit 1.20.0 are present on .9. +- Box to .9 TCP is blocked by the Headscale ACL. Fix: src 100.64.0.11 to 100.64.0.9 ports 22 and the TTS port. Ty relays to the headscale bot. +- omarmacmini (.4, LAN 192.168.1.201): Docker Compose, 8 CPUs, ~4 GiB RAM free, no GPU. congfbot approved a CPU TTS container in ~/audiobook-tts/, restart: unless-stopped, mem_limit ~3 GiB, avoid ports 9001, 1080, 8090, 3000, 5432, 6379. Kokoro fits CPU better than Chatterbox. +- Open decision for Ty: stop the 27B on GPU1, or run the pilot voice on CPU at omarmacmini. + +## Agreed by Ty 2026-10-16 +- Items 1-5 agreed: ingest (originals never modified, process copies), clean-up, structure (Audible/ACX: credits in own files, retail sample, one chapter per file, no approval gate, warnings mined by the agent), narration script, voice (iterate by ear). +- Post-MVP stretch (C): voice and tone parameters per document type (history vs instructional, etc.). MVP focuses on the core voice. +- Repo layout agreed (C): books/// with original PDF (untouched) plus text/, script/, figures/skipped_figures.md, audio/, book.json, warnings.log, LICENSE_STATUS.md in the same folder. _incoming/ for new PDFs, catalog.yaml built from book.json, _pipeline/ for code (location of code in this repo pending Ty). Co-authors, publisher, edition, series are tags in book.json. No playback copy. +- Step 7 (checks) agreed; speech-to-text check runs for now, becomes a toggle if corrections are few. +- Hosting recommendation pending Ty: pipeline in Docker on omarmacmini, voice (and optional STT) in Docker on GPU1; GPU1 model decommission already authorized. LAN link verified by congfbot. Push from omarmacmini to Gitea not configured. + +## Deploy runner (2026-10-16) +- No Gitea Actions runner exists anywhere (Ty, 2026-10-16). old-books has no workflows or runs (checked via Gitea connector). +- Ty asked to add the runner to the plan (C). Hosting split still pending Ty's final yes. +- Plan: one rootless Docker-in-Docker runner (docs.gitea.com/runner/develop/installation/docker) on omarmacmini, restart: always, registered to Ty_Tech/old-books only. Separate machine from the Gitea host, as Gitea recommends. No Docker socket mount. +- Job 1, deploy: on a Gitea release, pull that release's pipeline code to omarmacmini and restart the pipeline container. +- Job 2, save audio: commit finished audio, text and warnings into the book folder using the built-in job token (permissions: contents: write). No personal token. Added only after the repo's default token setting is confirmed to allow pushes. +- Install is done by congfbot. Needs a repo runner registration token from Gitea Actions settings. +- Unverified: Gitea version (docs read were for 1.27). + +## Code location and archive settings (2026-10-16, Ty) +- (C) Everything lives in one repo: pipeline code goes in _pipeline/ in Ty_Tech/old-books. +- (C) The pipeline must have configurable settings so Ty can point it at a different archive location later without code changes. + +## Compute placement (2026-10-16, Ty) +- (C) The language model is a setting: any OpenAI-compatible endpoint, cloud or LAN. No machine is fixed in the plan. +- (C) omarmacmini is 2012-era hardware. Ty does not want heavy CPU work on it. +- Recommendation pending Ty: omarmacmini only coordinates (runner, scheduling, git commits). A worker container on ubudual5060 does OCR, ffmpeg encoding, speech-to-text and voice. Asked congfbot to measure the CPU on .9 (2026-10-16 2:34 PM PT). +- Measured by congfbot on ubudual5060 (2026-10-16 2:35 PM PT): Ryzen 9 5900XT, 16 cores/32 threads, idle. RAM 30.9 GiB, 17.9 GiB available. Docker disk: 32 GiB free of 172 (81% full), the tight resource. ffmpeg MP3 encode of 600 s mono: 1.76 s (about 340x realtime, sine wave, speech will be slower). 8 parallel encodes: 2.04 s total. Tesseract/ocrmypdf not installed, not benchmarked. Docker published ports pass ufw from omarmacmini: upload 55.4 MB/s, download 117.5 MB/s. + +## Host decision (2026-10-16, Ty) +- Ty prefers the whole workload on ubudual5060 (Docker) for simplicity. Reason: a Mac mini split needs two custom images. Mac mini no longer part of the plan unless Ty reverses. +- Consequence: the Gitea runner and the pipeline container also move to ubudual5060. Voice/speech servers stay off-the-shelf images. Docker disk there is tight (32 GiB free). +- Pending Ty's final yes. + +## ubudual5060 storage survey (congfbot, read-only, 2026-10-16 2:4x PM PT) +- / (nvme0n1p6 ext4, 172G, 32G free, 81%) holds Docker root. Docker build cache 37.4G (36.2G reclaimable, none active). Caches (pip, npm, ccache, playwright) about 8G. CUDA installer .run 5.0G (toolkit already installed). Possible duplicate model mtp-a/b/model.gguf 6.5G (unverified). tabbyAPI venv 6.7G, no tabby process running. +- /mnt/models (NTFS via FUSE, same NVMe, 195G free, 2.3 GB/s read): fast. All model files already there. +- /mnt/abyss (NTFS via FUSE, 5400 rpm HDD, 920G free, 110 MB/s): fine for finished MP3s, poor for OCR scratch. +- nvme0n1p3 500G NTFS partition: unmounted, unknown contents, do not touch. +- TrueNAS: not mounted on .9. Reachability and speed untested. +- Likely-safe free-up of system drive: about 49G (build cache, caches, installer). Up to about 62G with the duplicate model and tabbyAPI. Nothing done. Ty decides. +- Space plan (estimates, not measured): Docker images about 20 GiB on /. Per book 3-5 GiB assumed, 23 books about 70-115 GiB. +- Recommendation: working audio on /mnt/models, finished MP3s archived to /mnt/abyss, Docker stays on /. Test a bind mount of an NTFS path in a small container first. + +## TrueNAS check from ubudual5060 (congfbot, read-only, 2026-10-16 2:4x PM PT) +- 192.168.1.200 reachable from .9: ping 0.98 ms avg, 0% loss. Open ports: 22, 80, 443, 445 (SMB), 139. NFS (2049) closed. +- Nothing mounted now. Shell history shows share //192.168.1.200/Business was mounted by hand at /mnt/truenas using user SMBSVC over cifs (vers=3.0). /mnt/truenas exists, empty. The password is in no config. +- Not measured: free space and throughput. A mount needs the SMBSVC credentials and root on .9 (no passwordless sudo). Needs Ty. +- Plan impact: TrueNAS is optional. Archive location is a setting, so it can be added later. Not needed for the pilot or MVP. + +## TrueNAS measured (congfbot, 2026-10-16 2:46 PM PT) +- Shares visible to SMBSVC: media, Personal, Business, VPSNAS (VPSNAS denied). Pool about 14.1 TiB total, 13.1 TiB free. +- Speed from .9 to Business over SMB3: write 106 MB/s, read 106 MB/s, readback matched. Gigabit-limited. +- Not mounted: mount.cifs needs root on .9. smbclient works without sudo. Not tested: Docker bind mount, reboot persistence. +- Plan: scratch stays on local /mnt/models. TrueNAS is a good archive target for finished MP3s and sources via the archive-location setting. Nothing named old-books or audiobook exists on the NAS yet. + +## TrueNAS mounted on ubudual5060 (congfbot, 2026-10-16 2:49 PM PT) +- //192.168.1.200/Business mounted at /mnt/truenas (cifs, vers=3.0). Credentials in /etc/smb-truenas.cred, root-only. Business: 15T, 14T free. +- 1 GiB test: write 105 MB/s, read 110 MB/s, readback matched. Docker bind mount works read and write. Persists via fstab (nofail, automount). Reboot not tested. +- Risk: a container sees an empty folder if the NAS is down. Use a healthcheck that checks for a known file. +- No audiobook folder on the NAS yet. Nothing created. Ty has not named a path. +- Plan: scratch on local NVMe, finished audio and sources on /mnt/truenas via the archive-location setting. + +## Audio cues (2026-10-16, Ty: yes) +- (C) Each inserted part is wrapped in a short soft chime (about 1 s, within ACX loudness) plus a spoken label in the narrator voice (for example "Chapter summary"). A chime marks the end. + +## License tracking (2026-10-16, Ty: agree) +- (C) Each book folder has LICENSE_STATUS.md: status (public domain, copyrighted, unknown), reason, date checked. Code decides from the publication year in book.json: published 1930 or earlier is US public domain as of 2026. Anything else is unknown until confirmed. Default is unknown. Status never blocks audio production. + +## CORRECTION (2026-10-17, Ty) +- (C) ONE repo holds everything: code in _pipeline/, and each book's PDF, text, script, figure list, AUDIO, warnings and license file in its own folder in Ty_Tech/old-books. This was decided earlier. It overrides every earlier note here that put finished audio or sources on TrueNAS, /mnt/abyss or any other archive. +- The archive location stays a setting (Ty's request), but its default is the repo book folder. TrueNAS is not the audio home. Mounted at /mnt/truenas only as an optional extra. Suggested folder name Business/old-books-archive is withdrawn. +- Scratch work (OCR, wav intermediates) stays on local NVMe, then final files are committed to the repo. + +## Disk cleanup done (congfbot, 2026-10-17 9:46 AM PT) +- ubudual5060: / now 39G free (77% used), /mnt/models 243G free. Qwen and Bonsai models still serving. /mnt/truenas still mounted. Congfbot's earlier stop signal was a mistake and is void. + +## Approved to proceed (2026-10-17 10:25 AM PT, Ty by voice call) +- (C) 39 GB free on / is enough. Proceed on ubudual5060: decommission GPU1 model (save unit file exactly), Docker stack with pipeline and runner together, restart always. Voice loads on demand and unloads when idle. Mac mini is out. Build the pilot (Clark Chapter I) with chime-plus-spoken-label cues, summaries off by default, LICENSE_STATUS.md default unknown. +- Asked congfbot for GPU1 decommission and Docker set-up. Text stages and pipeline code in progress.