Fix paragraph breaks, drop caps, headers and captions for chapters 2-6 of the Clark book
Root cause: clean-up found paragraph breaks from the layout indent of pdftotext -layout
(columns, relative to the smallest indent in the whole chapter). Chapter 1 happened to
have a clean left margin; later pages are skewed and drift 1-2 columns, drop caps push
the first lines right by varying amounts, and some pages wrap text around a label, so
continuation lines were taken as paragraph starts (a blank line inside a sentence).
Fix (all deterministic, all logged to warnings.log):
- pdftext: read each line's position in points (pdftotext -bbox-layout) and attach it to
the layout line it belongs to (units() now carries 'geom'; merged/partial lines get none).
- cleanup: a paragraph starts when a line sits >= 4.5 pt right of the line before it, or
>= 2.5 pt lower than the page's normal line pitch. A line without a position or starting
in lower case never starts a paragraph. Text wrapped beside a drop cap is not a start.
Without positions the old indent rule is still used.
- cleanup: drop cap + small caps on ONE layout line ('F OR MANY YEARS') is repaired like
the two-line form (chapters 4 and 5).
- pageparts.header_zone: the running header is found even when text follows it without a
blank line, and the right-hand header (chapter title, page number sometimes misread as
't5' or 'a') is recognised from the chapter title.
- cleanup: all-capitals lines below the text are captions (removed); an all-capitals line
above the text is kept as a section heading in title case ('The Deferred Preface');
'Right: MARK TWIN' style captions removed; '5. JOY' labels set inside a sentence removed.
- cleanup: paragraph starting with a lone quote mark + lower-case word joins the paragraph
before it; paragraph-leading ' + capital becomes '"' (not elisions such as 'Tis); a bullet
in place of an opening quote becomes '"'; closing quote with no opening quote, and a '1'
that is probably the letter I, are logged only.
- cleanup: mid-line 'worth- while' joined only when certain, otherwise logged.
- tests: new _pipeline/tests/test_cleanup.py (36 tests, 27 fail on the previous code);
test_pipeline.py helper test_piece renamed make_test_piece so pytest stops collecting it.
Chapter 1 text is byte-identical to the approved pilot text.
Re-run for chapters 1-6: cleanup, figures, script, license. Text, script, warnings and
book.json for chapters 2-6 added.
This commit is contained in:
@@ -20,7 +20,7 @@ import voice
|
||||
CFG = settings.load()
|
||||
|
||||
|
||||
def test_piece(path, hz, seconds, level):
|
||||
def make_test_piece(path, hz, seconds, level):
|
||||
"""TEST signal: speech-like (harmonics, 4 Hz syllable rhythm, a little noise) at amplitude `level`."""
|
||||
expr = "%g*(0.6*sin(2*PI*%g*t)+0.3*sin(4*PI*%g*t)+0.1*sin(6*PI*%g*t))*(0.55+0.45*sin(2*PI*4*t))" % (level, hz, hz, hz)
|
||||
am.ffmpeg(["-f", "lavfi", "-i", "aevalsrc='%s':s=24000:d=%g" % (expr, seconds), "-ar", "24000", "-ac", "1", path])
|
||||
@@ -32,7 +32,7 @@ class AudioTests(unittest.TestCase):
|
||||
items = []
|
||||
for i, (hz, lvl) in enumerate([(120, 0.02), (150, 0.05), (130, 0.03)]):
|
||||
p = os.path.join(d, "t%d.wav" % i)
|
||||
test_piece(p, hz, 4, lvl)
|
||||
make_test_piece(p, hz, 4, lvl)
|
||||
items.append({"wav": p, "gap_after_s": 0.7})
|
||||
if i == 0:
|
||||
items.append({"cue": "chime", "gap_after_s": 0.25})
|
||||
@@ -46,7 +46,7 @@ class AudioTests(unittest.TestCase):
|
||||
def test_checker_fails_bad_files(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
raw = os.path.join(d, "raw.wav")
|
||||
test_piece(raw, 130, 4, 1.4) # hot test signal, clips at full scale
|
||||
make_test_piece(raw, 130, 4, 1.4) # hot test signal, clips at full scale
|
||||
bad = os.path.join(d, "bad.mp3") # stereo, 128k, 22.05 kHz, no room tone
|
||||
am.ffmpeg(["-i", raw, "-ac", "2", "-ar", "22050", "-c:a", "libmp3lame", "-b:a", "128k", bad])
|
||||
failed = {n for n, v, p in acx_check.judge(CFG, bad, acx_check.measure(CFG, bad)) if not p}
|
||||
@@ -70,7 +70,7 @@ class StubHandler(http.server.BaseHTTPRequestHandler):
|
||||
if self.path.endswith("/audio/speech") or self.path == "/tts":
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = os.path.join(d, "x.wav")
|
||||
test_piece(p, 130, 1, 0.1)
|
||||
make_test_piece(p, 130, 1, 0.1)
|
||||
data = open(p, "rb").read()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "audio/wav")
|
||||
|
||||
Reference in New Issue
Block a user