Fix paragraph breaks, drop caps, headers and captions for chapters 2-6 of the Clark book

Root cause: clean-up found paragraph breaks from the layout indent of pdftotext -layout
(columns, relative to the smallest indent in the whole chapter). Chapter 1 happened to
have a clean left margin; later pages are skewed and drift 1-2 columns, drop caps push
the first lines right by varying amounts, and some pages wrap text around a label, so
continuation lines were taken as paragraph starts (a blank line inside a sentence).

Fix (all deterministic, all logged to warnings.log):
- pdftext: read each line's position in points (pdftotext -bbox-layout) and attach it to
  the layout line it belongs to (units() now carries 'geom'; merged/partial lines get none).
- cleanup: a paragraph starts when a line sits >= 4.5 pt right of the line before it, or
  >= 2.5 pt lower than the page's normal line pitch. A line without a position or starting
  in lower case never starts a paragraph. Text wrapped beside a drop cap is not a start.
  Without positions the old indent rule is still used.
- cleanup: drop cap + small caps on ONE layout line ('F     OR MANY YEARS') is repaired like
  the two-line form (chapters 4 and 5).
- pageparts.header_zone: the running header is found even when text follows it without a
  blank line, and the right-hand header (chapter title, page number sometimes misread as
  't5' or 'a') is recognised from the chapter title.
- cleanup: all-capitals lines below the text are captions (removed); an all-capitals line
  above the text is kept as a section heading in title case ('The Deferred Preface');
  'Right: MARK TWIN' style captions removed; '5. JOY' labels set inside a sentence removed.
- cleanup: paragraph starting with a lone quote mark + lower-case word joins the paragraph
  before it; paragraph-leading ' + capital becomes '"' (not elisions such as 'Tis); a bullet
  in place of an opening quote becomes '"'; closing quote with no opening quote, and a '1'
  that is probably the letter I, are logged only.
- cleanup: mid-line 'worth- while' joined only when certain, otherwise logged.
- tests: new _pipeline/tests/test_cleanup.py (36 tests, 27 fail on the previous code);
  test_pipeline.py helper test_piece renamed make_test_piece so pytest stops collecting it.

Chapter 1 text is byte-identical to the approved pilot text.
Re-run for chapters 1-6: cleanup, figures, script, license. Text, script, warnings and
book.json for chapters 2-6 added.

_pipeline/pdftext.py

@@ -4,6 +4,7 @@ import re
 import subprocess
 import tempfile
 import xml.etree.ElementTree as ET
+from html import unescape
 
 
 def _run(cmd):
@@ -42,14 +43,60 @@ def chars_per_page(pdf):
     return out
 
 
+_LINE_RE = re.compile(r'<line xMin="([\d.]+)" yMin="([\d.]+)" xMax="([\d.]+)" yMax="([\d.]+)">(.*?)</line>', re.S)
+
+
+def page_lines(pdf, page):
+    """Text lines of one PDF page with their position in points, top to bottom:
+    [{'x', 'y', 'x2', 'y2', 'text'}]. Coordinates are those of the displayed page (poppler bbox-layout)."""
+    xml = _run(["pdftotext", "-bbox-layout", "-f", str(page), "-l", str(page), pdf, "-"])
+    out = []
+    for m in _LINE_RE.finditer(xml):
+        words = [unescape(w) for w in re.findall(r">([^<]+)</word>", m.group(5))]
+        out.append({"x": float(m.group(1)), "y": float(m.group(2)), "x2": float(m.group(3)),
+                    "y2": float(m.group(4)), "text": " ".join(words)})
+    return sorted(out, key=lambda ln: (ln["y"], ln["x"]))
+
+
+def _key(text):
+    return re.sub(r"[^A-Za-z0-9]", "", text)
+
+
+def align(raw_lines, boxes, window=8):
+    """Give each line of a layout text (pdftotext -layout) the position of the same line from page_lines.
+    Returns a list parallel to raw_lines: the box dict, or None for blank lines and lines that cannot be
+    matched one to one (merged columns, small-caps headers). A line that only partly matches a box is left
+    as None: its position would be wrong."""
+    result = [None] * len(raw_lines)
+    j = 0
+    for i, line in enumerate(raw_lines):
+        k = _key(line)
+        if not k:
+            continue
+        for jj in range(j, min(j + window, len(boxes))):
+            b = _key(boxes[jj]["text"])
+            if b == k:
+                result[i] = boxes[jj]
+                j = jj + 1
+                break
+            if len(b) >= 8 and (k.endswith(b) or k.startswith(b)):
+                j = jj + 1
+                break
+    return result
+
+
 def units(pdf):
-    """Reading units in order: [{'pdf_page', 'half', 'text'}]. Text keeps the layout (indentation, hyphens)."""
+    """Reading units in order: [{'pdf_page', 'half', 'text', 'geom'}]. Text keeps the layout (indentation, hyphens).
+    'geom' is parallel to text.split('\\n'): the position of each line on the page (see align)."""
     result = []
     for p, (w, h) in enumerate(page_geometry(pdf), start=1):
+        boxes = page_lines(pdf, p)
         for half, x, width in halves(w, h):
             text = _run(["pdftotext", "-layout", "-f", str(p), "-l", str(p),
                          "-x", str(int(x)), "-y", "0", "-W", str(int(width)), "-H", str(int(h)), pdf, "-"])
-            result.append({"pdf_page": p, "half": half, "text": text.replace("\f", "")})
+            text = text.replace("\f", "")
+            mine = [b for b in boxes if x <= b["x"] < x + width] if half else boxes
+            result.append({"pdf_page": p, "half": half, "text": text, "geom": align(text.split("\n"), mine)})
     return result