personal-agent-os/memory-systems/agent-os/Video Stack/Full Stack Spec.md

tags, updated, source

tags	updated	source
video-stack spec	2026-10-15	explainer-video-stack/STACK.md

STACK.md — Everything Hermes needs to produce Grok-Bot-quality explainer videos

Audience: the Hermes explainer profile on TonyB (Windows 11, RTX 3060 12 GB), running a cheap model (e.g. gpt-6-luna). This is the single source of truth. When another doc disagrees with this one, this one wins. Repos: tonyjbala/explainer-video-stack (business explainers) and tonyjbala/political-explainer-stack (ballot/political ads). Worked example to copy: C:\video\examples\prop1ad (HyperFrames, 45 s, 9:16). Last updated: 2026-10-15.

---

1. Overview and the quality bar

A "Grok-Bot-quality" video is a 30–90 s explainer or ad (default 45 s, 9:16, 1080x1920) that passes all six tests:

#	Bar	What it means in practice
1	Verified facts	Every number and claim has an opened, primary source (LAO, official voter guide, SOS, agency data, product docs). Opinion is labelled or phrased as opinion. A short source line sits under every stat on screen.
2	Natural VO	Written for the ear (Section 5.1). No misreads (Whisper-checked), no odd pauses, no read-aloud stage directions. Political = authoritative male (Gemini Charon); business = firm, friendly female (Gemini Kore).
3	Music that adds feel	Timed to the beats: pulse/tension under the problem or cost, a swell under the CTA, resolves on the cut. A generic bed that "doesn't do anything" is a fail (Prop 1 lesson).
4	Kinetic but clean design	One idea per scene, huge numbers, staggered entrances, a palette made for the topic, nothing inside the platform UI safe zones, no dead half-screens.
5	Perfect captions	Burned-in, 3-word groups, active word highlighted, timed from Whisper word timestamps, spelled correctly ("$500M", "Prop 1", not "Prop.", not "$5").
6	Broadcast loudness	Music ~12 dB under voice with sidechain ducking; final -16 LUFS integrated, true peak ≤ -1.5 dBTP, AAC 192k 48 kHz; last word ends ~2 s before the cut.

Plus: three approval gates (script, voice, music) before any build, and nothing is ever posted or sent by you.

---

2. Hardware and software inventory (verified on TonyB 2026-10-15 unless marked)

Item	Version / path	Notes
GPU	NVIDIA GeForce RTX 3060, 12288 MiB, driver 610.88	Qwen alone uses ~6.4 GB baseline (≈11.5 GB with KV in use). TTS models need 3.9–5.0 GB.
LLM	C:\llama\llama-server.exe, Qwen3.8-35B-A3B-Q4_K_M on 127.0.0.1:1922	The only model the video work assumes. Health: http://127.0.0.1:1922/health.
Node	v24.18.0 (HyperFrames needs Node ≥ 22)
HyperFrames	npx --yes hyperframes@0.8.162 <cmd> (pinned in hf\ad\package.json)	Default renderer. Set $env:HYPERFRAMES_SKIP_SKILLS="1" so npx doesn't install its bundled agent skills. Render sends anonymous telemetry: run npx hyperframes telemetry disable once. Never run hyperframes publish (it uploads).
Remotion	C:\video\template (16:9 Main + 9:16 MainVertical, 30 fps)	Fallback renderer only. Copy per project with xcopy /E /I.
ffmpeg	9.0.1 essentials (gyan.dev) on PATH	Mixing, loudness, muxing.
Chatterbox	venv C:\tts\chatterbox (uv, Python 3.11, torch 2.6 cu124) → C:\tts\chatterbox\Scripts\python.exe; driver C:\tts\tts.py	Local fallback voice; also runs Whisper check (--check-only). ~1x realtime, 3.9 GB.
Qwen3-TTS	venv C:\tts\qwen3tts (uv, torch 2.6 cu124, qwen-tts) → C:\tts\qwen3tts\Scripts\python.exe; 1.7B VoiceDesign + 1.7B Base	Second local option; voice-from-description. Slow (~2.5–5 min per 15–30 s), 4.3–5.0 GB, no flash-attn on Windows. Runner: C:\tts\vt\vt.py (env VREF/VTAG; engines `cb
Whisper	faster-whisper small.en, int8, CPU	Built into tts.py; produces words.json {"1": [[word,start,end],...]}.
Kokoro	kokoro-onnx (box/legacy)	Fast fallback voice for drafts only. Never use a Kokoro clip as a clone reference (refs in C:\tts\voices\kokoro_*_ref.wav are legacy; don't use them).
Human voice ref	C:\tts\voices\user_announcer_ref.wav (Anthony, recorded with consent; 20.65 s mono 24 kHz)	The only approved clone reference.
Gemini TTS	C:\tts\gemini_tts.py (stdlib only; repo copy scripts/tts/gemini_tts.py)	Default voice engine. Section 2.1.
Music (code)	scripts/music/gen.py (options A–C), gen2.py (D–F) in explainer-video-stack; compose.py presets in C:\video\template	numpy synth, royalty-free. Writes full track + 20 s preview.
Music (paid, optional)	C:\video\template\el_music.py "<prompt>" <seconds> out.mp3	ElevenLabs Music, needs ELEVENLABS_API_KEY; instrumental only. Use if two synth rounds miss.

2.1 Gemini TTS (gemini_tts.py)

python C:\tts\gemini_tts.py script.json out.wav [--voice Charon] [--style "..."] [--gap 0.2] [--each DIR] [--whole] [--model gemini-3.8-flash-tts] [--fallback-model gemini-3.8-flash-lite-tts]

• script.json = JSON list of lines (or a .txt, one sentence per line). Default: one call per sentence, joined with 200 ms gaps. --each DIR also saves DIR\s01.wav….
• --whole = one API call for the whole script (use this by default; it saves quota).
• On a 429 quota error it retries automatically on gemini-3.8-flash-lite-tts (--fallback-model "" disables).
• Presets (user-approved Oct 2026):
  • Political (sample P1): --voice Charon (default style "measured, authoritative newscaster, confident, serious").
  • Business (sample B6): --voice Kore --style "friendly, polished product explainer, conversational, upbeat but credible" --whole.
• Quota (free tier): flash-tts = 10 requests/day; flash-lite-tts = 3 requests/minute. A per-sentence political ad can eat a whole day. Paid tier costs about $0.01–0.03/min.
• Key: GEMINI_API_KEY from the environment. Put GEMINI_API_KEY=... in C:\Users\fight\AppData\Local\hermes\profiles\explainer\.env (or setx GEMINI_API_KEY "..."). Never print it, echo it, or commit it.
• Output: 24 kHz mono 16-bit WAV.

---

3. The end-to-end pipeline (with gates)

Brief → Research/fact-check → Script (2 variants) → [GATE 1 script]
 → Voice sample → [GATE 2 voice] → Music previews x3 → [GATE 3 music]
 → Full VO + Whisper check → Scene design → HyperFrames build + lint + stills
 → Mix → Final QA → Deliver (9:16 mp4, plus 16:9 re-layout if asked) → [GATE 4 final approval]

Project folder: C:\video\<project>\ with brief.md, vo_review.md, script.json, vo\, music\, hf\ad\, out\, snaps\.

Step 1 — Brief → brief.md

Topic, audience, length (default 45 s), aspect (9:16 first), sponsor / "Paid for by" name, stance (political), CTA, tone, deadline. Political: use political-explainer-stack\templates\brief.md. Done when: every field filled or explicitly "ask user".

Step 2 — Research and fact-check → vo_review.md

• Open every source you cite. Prefer primary: LAO ballot analysis, CA SOS Quick Guide / Official Voter Guide, UC Berkeley IGS, agency data, the product's own docs.
• Table columns: beat | original line | suggested accurate line | status (✅/⚠️) + why | source URL. Then a "Sources" list.
• Watch for the Prop 1 traps: a figure that applies to only part of the measure ($500–600M/yr is for the $10B part only), "inflation-adjusted" qualifiers, endorsements you cannot verify (EDC GOP's own slate was not online; attribute only to a guide you actually have).
• Label opinion lines ("doesn't change the rules that limit supply"). Done when: every number has a URL, every ⚠️ has a fix.

Step 3 — Script → script.json (2 variants)

Rules in Section 5.1. Structure (political, 5 beats): hook (0–7 s) → breakdown → cost/stakes → concerns/proof → vote call. Business: hook → problem → how it works → proof/benefit → CTA. ~105 words max for 45 s (≤150 wpm, ~2 s spare). One JSON string per beat/scene. Numbers spelled as spoken. GATE 1: send both variants (plain text, with word counts and estimated seconds). Stop until the user picks or edits.

Step 4 — Voice sample → GATE 2

Render line 1 (or the first ~15 s) with the preset voice (Section 4.4). If the user wants options, render 2–3 voices of the same line. Done when: the user approves one voice by name.

Step 5 — Music previews → GATE 3

Write a timed music brief first (e.g. "0–6 s sparse tension; 15–30 s driving low pulse under the cost; 38–45 s resolute swell, resolve on the cut"). Make 3 distinct options, 20 s previews of the most important section, mp3. One-line description each. Done when: the user picks one (or asks for changes; iterate).

Step 6 — Full VO + Whisper check → vo\v1..vN.wav, vo\words.json

Render the whole script with the approved voice, one file per line (or --whole then Whisper). Check transcript vs script; re-roll any misread or awkward line. Done when: every line matches (digit-vs-word differences are fine), no pause > ~0.6 s inside a sentence, total VO ≤ length − 2 s.

Step 7 — Scene design

Pick palette, type, motion for this topic (Section 5). Plan one scene per VO line, with start = line start − 0.1 s. Safe zones 9:16: top 250 px, bottom 430 px, right 90 px clear. Content box x 80–910, y 380–1260. Captions at y≈1350.

Step 8 — HyperFrames build

Scene-per-<div> pattern from prop1ad\build.py: one Python script writes hf\ad\index.html from words.json + scene HTML. Lint, snapshot stills at key timestamps, look at them, fix, then render silent. Done when: lint clean, stills at hook / each scene / final 2 s look right, out\silent.mp4 is exactly the target length.

Step 9 — Mix

Exact ffmpeg in Section 4.9. Done when: I = -16 ±1 LUFS, TP ≤ -1.5 dBTP, voice clearly on top, music audible in gaps and swelling at the CTA.

Step 10 — Final QA → deliver → GATE 4

Run the checklist (Section 4.10). Deliver out\<project>_45s_vertical.mp4 (H.264 + AAC). If a 16:9 version is wanted, re-lay out the scenes for 1920x1080 (new positions, side-by-side panels); never just crop or pad the vertical. Send the path + report to the user; never post it.

---

4. Exact commands (PowerShell on TonyB)

4.0 Setup per project

$P = "C:\video\myproj"
New-Item -ItemType Directory -Force $P,"$P\vo","$P\music","$P\out","$P\snaps" | Out-Null
Copy-Item -Recurse C:\video\examples\prop1ad\hf "$P\hf"          # HyperFrames project skeleton
Copy-Item C:\video\examples\prop1ad\build.py "$P\build.py"      # then edit scenes/palette
$env:HYPERFRAMES_SKIP_SKILLS = "1"
# Preflight: exactly one llama-server, on 1922
Get-CimInstance Win32_Process -Filter "Name='llama-server.exe'" | % { "$($_.ProcessId) $($_.CommandLine)" }
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Done: one llama-server line (port 1922). If you see more, report it to the user; don't stop anything yourself.

4.1 Long jobs (anything > ~2 min: Qwen3-TTS, Chatterbox full runs, renders)

Remote-shell children die on disconnect, so use a one-time scheduled task:

schtasks /create /tn OneTime_VO /tr "cmd /c C:\video\myproj\run_vo.cmd > C:\video\myproj\vo\log.txt 2>&1" /sc once /st 23:59 /it /f
schtasks /run /tn OneTime_VO
# poll: Get-Content C:\video\myproj\vo\log.txt -Tail 5   (until it prints DONE)
schtasks /delete /tn OneTime_VO /f

4.4 Voice

cd $P
# Political sample (Gemini Charon), line 1 only:
python C:\tts\gemini_tts.py sample.json vo\sample.wav --voice Charon --whole
# Business sample (Gemini Kore):
python C:\tts\gemini_tts.py sample.json vo\sample.wav --voice Kore --style "friendly, polished product explainer, conversational, upbeat but credible" --whole
# Full VO, one file per line (political; costs N requests):
python C:\tts\gemini_tts.py script.json vo\voice.wav --voice Charon --each vo
# Local fallback (no key / quota gone): Chatterbox with the consented human ref, one line per call
C:\tts\chatterbox\Scripts\python.exe C:\tts\tts.py script.json vo --ref C:\tts\voices\user_announcer_ref.wav --ex 0.55 --cfg 0.4
# Re-roll one bad line:
C:\tts\chatterbox\Scripts\python.exe C:\tts\tts.py script.json vo --ref C:\tts\voices\user_announcer_ref.wav --ex 0.55 --cfg 0.4 --only 3 --seed 22

Whisper check / words.json (works on any WAVs named v1.wav..vN.wav; rename Gemini's s01.wav → v1.wav first):

Get-ChildItem vo\s*.wav | % { $n=[int]($_.BaseName.Substring(1)); Copy-Item $_.FullName "vo\v$n.wav" }
C:\tts\chatterbox\Scripts\python.exe C:\tts\tts.py script.json vo --check-only

Done: every HEARD line matches WANT (digits vs words OK). Join lines with 200 ms silence for vo\voice.wav if you rendered per line (gemini_tts.py does this already).

4.5 Music previews

# code-synth options (edit the arc/section times to the brief first)
cd C:\video\myproj\music
python <explainer-video-stack>\scripts\music\gen.py      # A–C
python <explainer-video-stack>\scripts\music\gen2.py     # D–F
# or presets: $env:MUSIC_SECONDS="47"; C:\tts\chatterbox\Scripts\python.exe C:\video\template\compose.py civic_tension cinematic_build
# or ElevenLabs (if key set): C:\tts\chatterbox\Scripts\python.exe C:\video\template\el_music.py "dark low strings, slow heartbeat pulse, builds to resolute swell at 38s" 47 optA.mp3
ffmpeg -y -i optionA_full.wav -ss 25 -t 20 -af "afade=t=in:d=0.5,afade=t=out:st=19:d=1" -b:a 160k optionA_preview.mp3

Music length = video length + ~2 s.

4.8 Build, lint, stills, render (HyperFrames)

cd $P
python build.py                                        # writes hf\ad\index.html
cd hf\ad
npx --yes hyperframes@0.8.162 lint
npx --yes hyperframes@0.8.162 snapshot --at 1,5,12,24,35,43 -o ..\..\snaps
# LOOK at every PNG in snaps\ (Read tool). Fix, rebuild, re-snapshot only what changed.
npx --yes hyperframes@0.8.162 render -o ..\..\out\silent.mp4 -q delivery
ffprobe -v error -show_entries format=duration:stream=width,height -of csv=p=0 ..\..\out\silent.mp4

Done: lint 0 errors; stills OK; 1080x1920; duration = target (e.g. 45.0).

4.9 Mix (exact command; same as scripts/mix.sh)

cd $P
ffmpeg -v error -y -i vo\voice.wav -i music\approved.wav -filter_complex "[1:a]volume=-12dB[m];[0:a]asplit=2[v][sc];[m][sc]sidechaincompress=threshold=0.05:ratio=6:attack=20:release=300[md];[v][md]amix=inputs=2:duration=longest:normalize=0,loudnorm=I=-16:TP=-1.5:LRA=11[a]" -map "[a]" -ar 48000 out\mix.wav
ffmpeg -v error -y -i out\silent.mp4 -i out\mix.wav -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -ar 48000 -shortest out\myproj_45s_vertical.mp4
ffmpeg -hide_banner -i out\myproj_45s_vertical.mp4 -af ebur128=peak=true -f null - 2>&1 | Select-String "I:|LRA:|Peak:" | Select-Object -Last 3

If the VO doesn't start at 0 s, delay it first: -af "adelay=200|200" on the voice or build voice.wav with the exact line offsets used in build.py (ST list). Done: I -16 ±1, Peak ≤ -1.5, listen once end to end.

4.10 Final QA checklist

• Every fact sourced; source line on screen; "Paid for by …" + "Demo, not official" if political/demo.
• Gates 1–3 approved (script, voice, music).
• Whisper matches script; listened end to end; no awkward pauses.
• Captions match audio word for word, 3-word groups, active word highlighted, inside safe zones.
• Exact duration; last word ends ≥ ~2 s before the cut; music resolves on the cut.
• -16 LUFS ±1, TP ≤ -1.5 dBTP; music audible but never masking VO.
• H.264 + AAC, 1080x1920, file name ends in .mp4, plays on a phone.
• Exactly one llama-server (1922) still running; no leftover scheduled tasks.
• Score the run in scorecard.md (copy, voice, music, visuals, speed, cost; 1–5).

---

5. Design system

5.1 Copywriting (write for the ear)

1. Sentences ≤ 12 words. One idea per line; one line = one breath.
2. Numbers as spoken: "ten billion dollars", "twenty twenty-six", "Prop One". Never two numbers in one sentence if avoidable (Prop 1 line 3 had four → awkward).
3. No comma chains or read-aloud lists; split them.
4. Contractions (it's, don't, you'll). Lead with the stake in the first 2 s.
5. 1 s breath before the CTA. ≤150 wpm; ~105 words per 45 s.
6. Two variants every time. Read aloud (TTS) and time against words.json before designing.

5.2 Palette, type, motion — chosen per topic

• Start from the topic's emotion: civic/serious (navy, red, cream), local/warm (greens, sand), tech/fintech (deep ink + one electric accent). One dark base, one light text, one accent, one highlight color for the active caption word. Check contrast (white on base ≥ 7:1).
• Type: one heavy sans ("Liberation Sans", Arial weight 800–900 for numbers/headlines, 700 for subs). Sizes for 1080x1920: kicker 30 px letter-spaced caps; sub 46 px; hero numbers 128–280 px; notes/sources 26–28 px (≥26 px).
• Motion (GSAP): each scene has a1/a2/a3 elements entering at +0.2 / +0.7 / +1.2 s, opacity 0→1, y 40→0, 0.5 s, power2.out. No bounces or spins for political. Business can add a count-up or slide, still short (≤0.6 s).
• Never reuse Prop 1's look unchanged for another topic.

5.3 Captions

Built from Whisper words.json: group 3 words, show from first word start to min(next group start, last word end + 0.4 s); active word tweens to the highlight color (#FFD34D in Prop 1) over 0.08 s at its timestamp. 60 px, weight 800, centered, text-shadow:0 4px 12px #000, box x 80–910 at y 1350. Clean tokens: "Prop." → "Prop", fix currency splits ("$5"+"00" → "$500"), attach "%"/"." to the previous token.

5.4 Kinetic numbers and data callouts

One number per beat, hero size; proportional bars ($10B 820 px vs $1.25B 103 px); stat blocks with a 10 px accent rule; "✕" concern cards (max 3); a source note under each.

5.5 End cards

Held ≥ 5 s. Political: big NO/YES block in the accent color, "ON PROP N", sponsor, close line, bordered box "Paid for by · Demo, not official / Source: …". Business: logo/name, one-line value, CTA (URL), no clutter.

5.6 Political vs business

	Political	Business
Voice	Gemini Charon, authoritative male	Gemini Kore, firm friendly female
Music	dark/urgent or resolute; pulse under cost, swell under vote call	warm/tech pulse, upbeat but not cheesy
Look	civic navy/red/cream, flag stripe, no playful motion	topic-fitted brand palette, livelier motion OK
Compliance	"Paid for by", demo label, FPPC check before any real use	product claims sourced
Repo	political-explainer-stack	explainer-video-stack

---

6. Hard rules

1. One llama-server only: Qwen on 1922. Never start Gemma (1923) or any second model alongside. Don't stop or start models without asking the user. (Don't run C:\llama\restore_llama_servers.bat — the TonyB copy still starts Gemma too.)
2. Long runs via one-time schtasks, then delete the task.
3. Never post, send, publish, upload or share without explicit approval. No hyperframes publish.
4. Gitea: tonyjbala account only. Never use Ty / Ty_Tech credentials.
5. No Kokoro-made clone references. Clone only from C:\tts\voices\user_announcer_ref.wav (consented) or documented public-domain audio. Never clone an identifiable person without written consent.
6. Keep tokens low: reuse templates, read files only as needed, stills instead of test renders, regenerate only the broken line.
7. Never print or commit keys (GEMINI_API_KEY, ELEVENLABS_API_KEY); no media files (mp4/wav/mp3) in git.

---

7. Known pitfalls and fixes

Pitfall	Fix
Blank HyperFrames scenes — scenes render empty because the GSAP selector doesn't match (e.g. targeting .a1 without the scene id, or an id that doesn't exist), so elements stay at the CSS start state / are never animated in.	Every scene div needs a unique id="sN" + class="scene clip" + data-start/data-duration; animate with "#sN .a1". Register window.__timelines["main"]=tl and tl.set({},{},<duration>). Run lint, then snapshot mid-scene to confirm content is visible.
Gemini reads directions aloud ("Say in a calm voice: …" gets spoken)	Text = only words to be spoken. Delivery goes in --style (sent as speech_metadata.style). systemInstruction is rejected.
Gemini quota (429) — 10/day on flash-tts	Use --whole; script auto-falls back to Flash-Lite (3/min). The daily quota resets once a day (observed around 4:50 PM PT). Fallback: Chatterbox with human ref.
Whisper number formatting ("350,000" vs "three hundred fifty thousand")	Not a misread. Compare normalized; for captions show digits (they read better on screen).
Awkward VO (odd pauses, flat delivery)	Shorter sentences, fewer numbers per line; one sentence per call; re-roll seed; never a Kokoro reference.
Tight ending (slower voice runs into the cut)	Budget ~2 s spare; trim words, or speed VO ≤3% (atempo=1.03); never cut the last word.
Windows mp4 extension — Explorer hides extensions, so a file can end up name.mp4.mp4 or without .mp4 and won't play on the phone	Always pass an explicit -o ...\name.mp4; verify with Get-ChildItem out (shows full names) and ffprobe.
HyperFrames npx installs skills / telemetry	$env:HYPERFRAMES_SKIP_SKILLS="1"; npx hyperframes telemetry disable.
Remote shell jobs die on disconnect	schtasks (Rule 2).

---

8. How a cheap model should work

1. One small step at a time. Do one numbered step from Section 3, write its output to a file, then verify it (read the file back, run the check command) before the next.
2. Use the "Done when" lines and the QA checklist literally. If a check fails, fix only that thing.
3. Copy, don't invent. Start from prop1ad\build.py, the templates, and the exact commands above; change values, not structure.
4. Stop at every GATE and ask the user in one short message with the files/paths. Don't build ahead of approvals.
5. Small context. Don't paste whole files or logs; use -Tail, Select-String, and stills.
6. When unsure, ask. If a source can't be opened, a tool errors twice, or VRAM is short: report it and ask; don't improvise around a rule.
7. Report at the end: mp4 path, duration, LUFS/TP, voice + settings, final script, sources, claims softened, scorecard.

---

Appendix — file map

• C:\tts\ — gemini_tts.py, tts.py (Chatterbox + Whisper), gen3060.py, chatterbox\, qwen3tts\, vt\vt.py, voices\user_announcer_ref.wav
• C:\video\ — examples\prop1ad, template (Remotion), voice_tests (B1–B10, P1–P4, NOTES.md), COPYWRITING.md, HERMES_PROMPT.md, STACK.md (this file)
• Hermes profile — C:\Users\fight\AppData\Local\hermes\profiles\explainer\ (SOUL.md, STACK.md, .env, skills\media\explainer-video\SKILL.md)
• Obsidian — C:\Users\fight\Documents\Obsidian Vault\Video Stack\Full Stack Spec.md

Back to Video Stack - Overview