Layout API

The layout module adds protected machine metadata and maps codec frames to physical pages. read_pages() is the inverse validation step.

glyphive.layout

Encoded line stream ⇄ paginated document with protected layout metadata.

This geometry-agnostic module groups codec L/P frames into physical pages. Page 1 starts with a human-readable #!glyphive summary and then fixed-width H machine-header frames. Every page ends in a T machine footer followed by a human PAGE n/total hint on the same line.

Restore trusts only the H/T frames. They use the measured-safe 16-character bootstrap alphabet and CRC-16 checks; the compact H-frame envelope additionally carries its exact length and a digest. Thus the selected payload codec, compression method, page count, and document digest are recoverable without trusting or repairing unrestricted ASCII. The human representations may be corrupted or clipped without changing machine interpretation.

H/T payload lines are capped at the same 60 safe characters used by codec data frames. Footer page identity and the first 8 bytes of the SHA-256 of that page's "\n"-joined data frames live inside the protected T payload. Pages may be read out of order; a missing or integrity-invalid metadata frame fails loud.

COMMENT_PREFIX = '#!' module-attribute

HEADER_PREFIX = '#!glyphive' module-attribute

LAYOUT_VERSION = 1 module-attribute

PAGE_HASH_CHARS = 16 module-attribute

__all__ = ['HEADER_PREFIX', 'PAGE_HASH_CHARS', 'LayoutError', 'MissingPageError', 'Page', 'format_header', 'parse_header', 'format_page_footer', 'verify_page_footer', 'page_data_hash', 'paginate', 'iter_paginate', 'read_pages', 'read_pages_to_spool'] module-attribute

LayoutError

Bases: ValueError

Raised on a malformed header/footer or an unrecoverable page structure.

MissingPageError(missing, total)

Bases: LayoutError

Raised when the footers show a page number is absent from the transcript.

missing lists the 1-based page numbers that were not found.

Source code in src/glyphive/layout.py

def __init__(self, missing: _ty.Sequence[int], total: int) -> None:
    self.missing = list(missing)
    self.total = total
    joined = ", ".join(str(n) for n in self.missing)
    super().__init__(
        f"missing page(s) {joined} of {total}: transcript is incomplete"
    )

missing = list(missing) instance-attribute

total = total instance-attribute

Page

Bases: NamedTuple

One physical page.

Attributes

number: 1-based page number. total: Total page count of the document. text_lines: Every text line that goes on this physical page, in order: the document header first (page 1 only), then the codec-framed lines, then the PAGE footer last. This is what a renderer prints. encoded_lines: Just the raw codec-framed (L/P) lines this page carries — i.e. text_lines without the header/footer. This is what feeds :func:codec.decode.

encoded_lines instance-attribute

number instance-attribute

text_lines instance-attribute

total instance-attribute

format_header(meta)

Render the compact single-line, display-only document header from meta.

The line is display-only — restore reads authoritative metadata from the CRC-protected H frames, never from this summary — so it is kept minimal to waste as few OCR characters as possible:

``#!glyphive v<N> <codec>[,<comp>] files=<f> bytes=<b> pages=<p>[ pgpar=<k>]``

v is a bare positional token (v1); codec and compression collapse to one positional codec[,comp] token (,comp omitted when compression is none/absent). sha256 and meta are deliberately NOT emitted here (they live in the protected header). pgpar is emitted only when non-zero. meta must supply codec, files, bytes, pages (and v, defaulted to :data:LAYOUT_VERSION). No value may contain whitespace. The inverse is :func:parse_header.

Source code in src/glyphive/layout.py

def format_header(meta: _ty.Mapping[str, _ty.Any]) -> str:
    """Render the compact single-line, display-only document header from ``meta``.

    The line is *display-only* — restore reads authoritative metadata from the
    CRC-protected H frames, never from this summary — so it is kept minimal to
    waste as few OCR characters as possible:

        ``#!glyphive v<N> <codec>[,<comp>] files=<f> bytes=<b> pages=<p>[ pgpar=<k>]``

    ``v`` is a bare positional token (``v1``); codec and compression collapse to
    one positional ``codec[,comp]`` token (``,comp`` omitted when compression is
    ``none``/absent). ``sha256`` and ``meta`` are deliberately NOT emitted here
    (they live in the protected header). ``pgpar`` is emitted only when non-zero.
    ``meta`` must supply ``codec``, ``files``, ``bytes``, ``pages`` (and ``v``,
    defaulted to :data:`LAYOUT_VERSION`). No value may contain whitespace. The
    inverse is :func:`parse_header`.
    """
    version = meta.get("v", LAYOUT_VERSION)
    for key in ("codec", "files", "bytes", "pages"):
        if key not in meta:
            raise LayoutError(f"header meta is missing required key {key!r}")

    comp = str(meta.get("comp", "none"))
    codec_token = str(meta["codec"])
    if comp and comp != "none":
        codec_token = f"{codec_token},{comp}"

    tokens: _ty.List[str] = [HEADER_PREFIX, f"v{version}", codec_token]
    tokens.append(f"files={meta['files']}")
    tokens.append(f"bytes={meta['bytes']}")
    tokens.append(f"pages={meta['pages']}")
    if int(meta.get("pgpar", 0)) != 0:
        tokens.append(f"pgpar={meta['pgpar']}")

    for token in tokens[1:]:
        if any(ch.isspace() for ch in token):
            raise LayoutError(f"header token may not contain whitespace: {token!r}")
    return " ".join(tokens)

format_page_footer(n, total, page_lines)

Render the per-page footer for page n of total.

page_lines are the codec-framed data/parity lines on this page (NOT the header, NOT the footer). Grammar: PAGE <n>/<total> sha256=<first16hex>.

Source code in src/glyphive/layout.py

def format_page_footer(
    n: int, total: int, page_lines: _ty.Sequence[str]
) -> str:
    """Render the per-page footer for page ``n`` of ``total``.

    ``page_lines`` are the codec-framed data/parity lines on this page (NOT the
    header, NOT the footer). Grammar: ``PAGE <n>/<total> sha256=<first16hex>``.
    """
    from .codec.engine import nibble_encode

    if not (1 <= n <= total < 2**32):
        raise LayoutError(f"invalid page position {n}/{total}")
    digest = bytes.fromhex(page_data_hash(page_lines)[:PAGE_HASH_CHARS])
    payload = nibble_encode(
        _MACHINE_FOOTER_MAGIC + total.to_bytes(4, "big") + digest
    )
    machine = _format_machine_frame(
        _MACHINE_FOOTER_KIND, n - 1, payload,
        nsym_line=_MACHINE_LINE_PARITY_BYTES,
    )
    return f"{machine} {_PAGE_MARKER} {n}/{total}"

iter_paginate(encoded_lines, n_encoded, meta, *, lines_per_page, parity_pages=0, emit_human_header=True)

Yield pages without retaining the full encoded line or page lists.

parity_pages (K, default 0) requests K additional document-level whole-page-recovery pages, emitted after the D data pages. Each data page's printed lines ("\n".join(page.encoded_lines), the same convention as :func:page_data_hash) form one RS "block", zero-padded to a common size B (the max data-page block length); K parity blocks are computed over the D data blocks (:mod:glyphive.codec.pagers) and printed as K additional pages carrying Q-framed lines. Parity page numbers continue past the data pages (D+1 .. D+K); footers' total becomes D+K. The machine envelope's pages field always stores D (the data page count); K and B are recorded separately as pgpar/ page_block_bytes. K=0 (the default) reproduces the pre-parity-pages byte-for-byte output exactly: no Q frames, no behavior change.

Source code in src/glyphive/layout.py

def iter_paginate(
    encoded_lines: _ty.Iterable[str],
    n_encoded: int,
    meta: _ty.MutableMapping[str, _ty.Any],
    *,
    lines_per_page: int,
    parity_pages: int = 0,
    emit_human_header: bool = True,
) -> _ty.Iterator[Page]:
    """Yield pages without retaining the full encoded line or page lists.

    ``parity_pages`` (K, default 0) requests K additional document-level
    whole-page-recovery pages, emitted after the D data pages. Each data
    page's printed lines (``"\\n".join(page.encoded_lines)``, the same
    convention as :func:`page_data_hash`) form one RS "block", zero-padded to
    a common size B (the max data-page block length); K parity blocks are
    computed over the D data blocks (:mod:`glyphive.codec.pagers`) and printed
    as K additional pages carrying ``Q``-framed lines. Parity page numbers
    continue past the data pages (``D+1 .. D+K``); footers' ``total`` becomes
    ``D+K``. The machine envelope's ``pages`` field always stores D (the data
    page count); K and B are recorded separately as ``pgpar``/
    ``page_block_bytes``. K=0 (the default) reproduces the pre-parity-pages
    byte-for-byte output exactly: no ``Q`` frames, no behavior change.
    """
    if n_encoded < 0:
        raise ValueError("n_encoded must be non-negative")
    if parity_pages < 0:
        raise ValueError("parity_pages must be non-negative")
    # The compact machine envelope uses a fixed-width u32 for ``pages``, so its
    # frame count does not depend on the numeric page count.  A provisional value
    # therefore gives us the exact first-page overhead before pagination.
    meta["pages"] = 1
    meta["pgpar"] = parity_pages
    meta["page_block_bytes"] = 0
    provisional_machine_header = _format_machine_header(meta)
    # Page-1 overhead: (optional human header line) + protected machine header
    # frames + footer. When ``emit_human_header`` is False the ``#!glyphive``
    # line is omitted, so page 1 gains one data-line slot.
    human_header_lines = 1 if emit_human_header else 0
    first_page_overhead = human_header_lines + len(provisional_machine_header) + 1
    data_total = _page_count(
        n_encoded,
        lines_per_page,
        first_page_overhead=first_page_overhead,
    )
    grand_total = data_total + parity_pages

    from .codec import pagers as _pagers
    from .codec.engine import _detect_line_parity_chars, split_frame, split_frame_with_parity

    # Select the page-parity Galois field automatically: GF(2^8) (1 byte/
    # symbol, cap 255 total blocks) while it fits, GF(2^16) (2 bytes/symbol,
    # cap 65535) once the archive is too large for GF(2^8). K=0 records the
    # GF(2^8) default field for byte-for-byte compatibility with pre-GF(2^16)
    # output, though the field is meaningless without parity pages.
    pgpar_field = 8
    if parity_pages:
        pgpar_field = 16 if grand_total > _pagers.MAX_TOTAL_BLOCKS else 8
        limit = _pagers.max_total_blocks(pgpar_field)
        if grand_total > limit:
            raise LayoutError(
                f"parity_pages={parity_pages} with {data_total} data page(s) "
                f"exceeds the {limit}-page Reed-Solomon limit "
                f"(data pages + parity pages must be <= {limit})"
            )
    meta["pgpar_field"] = pgpar_field

    # ``page_block_bytes`` (B) is a protected header field, but it depends on
    # every data page's contents (the max block length across all of them),
    # which are not known until all data lines are chunked. Page 1's header
    # is the very first thing yielded, so when K>0 we chunk all data pages up
    # front (consistent with the project's existing multi-pass, bounded-memory
    # create path) and reuse those chunks for emission below,
    # rather than consuming the encoded-line source twice. When K=0, none of
    # this runs: pages are chunked lazily exactly as before, so the K=0 path
    # is byte-for-byte identical to the pre-parity-pages format.
    precomputed_chunks: _ty.Optional[_ty.List[_ty.List[str]]] = None
    block_bytes = 0
    line_parity_chars = 0
    if parity_pages:
        source = iter(encoded_lines)
        remaining_n = n_encoded
        precomputed_chunks = []
        for page_no in range(1, data_total + 1):
            overhead = first_page_overhead if page_no == 1 else 1
            capacity = lines_per_page - overhead
            take = min(capacity, remaining_n)
            chunk = list(itertools.islice(source, take))
            remaining_n -= len(chunk)
            precomputed_chunks.append(chunk)
        block_bytes = max(
            (len("\n".join(chunk).encode("utf-8")) for chunk in precomputed_chunks),
            default=0,
        )
        if pgpar_field == 16 and block_bytes % 2:
            # GF(2^16) pairs adjacent bytes into symbols; keeping block_bytes
            # even here means every parity block comes out exactly
            # block_bytes long too (see codec.pagers), so the Q-page length
            # check below never has to special-case an odd/even mismatch.
            block_bytes += 1
        # The L/P lines carry an optional per-line Reed-Solomon parity field
        # (base16g v2) whose width is not knowable from line_width alone --
        # detect it structurally (same heuristic codec decode uses) so the
        # data-payload-width measurement below doesn't fold that field into
        # what it treats as "payload" (which would inflate Q parity rows past
        # the 60-char OCR-safe cap; see the unit-mismatch note below). Uses
        # the SELECTED codec's own alphabet/spec (base32g etc. differ from
        # the base16g bootstrap), falling back to base16g if the codec name
        # is unknown (e.g. a plugin not loaded for this invocation).
        from .codec.engine import BASE16G as _BASE16G

        payload_spec = _BASE16G
        codec_name = meta.get("codec")
        if codec_name:
            try:
                from .codec import get as _get_codec

                payload_spec = getattr(_get_codec(str(codec_name)), "_spec", _BASE16G)
            except ValueError:
                pass
        line_parity_chars = _detect_line_parity_chars(
            (line for chunk in precomputed_chunks for line in chunk), payload_spec
        )

    meta["pages"] = data_total
    meta["page_block_bytes"] = block_bytes
    header_line = format_header(meta)
    machine_header = _format_machine_header(meta)
    if len(machine_header) != len(provisional_machine_header):
        raise LayoutError("machine header frame count changed during pagination")

    encoded = iter(()) if precomputed_chunks is not None else iter(encoded_lines)
    cursor = 0
    data_blocks: _ty.List[bytes] = []
    data_payload_width = 0
    for page_no in range(1, data_total + 1):
        if precomputed_chunks is not None:
            chunk = precomputed_chunks[page_no - 1]
            cursor += len(chunk)
        else:
            overhead = first_page_overhead if page_no == 1 else 1
            capacity = lines_per_page - overhead
            chunk = list(itertools.islice(encoded, min(capacity, n_encoded - cursor)))
            cursor += len(chunk)

        text_lines: _ty.List[str] = []
        if page_no == 1:
            if emit_human_header:
                text_lines.append(header_line)
            text_lines.extend(machine_header)
        text_lines.extend(chunk)
        text_lines.append(format_page_footer(page_no, grand_total, chunk))

        if parity_pages:
            block = "\n".join(chunk).encode("utf-8")
            data_blocks.append(block)
            for line in chunk:
                # split_frame_with_parity (NOT split_frame) is required here:
                # split_frame's 3-tuple form re-merges payload+line-parity for
                # callers that don't care about the distinction, which would
                # silently re-inflate this width measurement right back to the
                # bug this block exists to fix.
                split = split_frame_with_parity(
                    line, spec=payload_spec, line_parity_chars=line_parity_chars
                )
                payload = split[1] if split is not None else line
                data_payload_width = max(data_payload_width, len(payload))

        yield Page(
            number=page_no,
            total=grand_total,
            text_lines=text_lines,
            encoded_lines=chunk,
        )

    # Sanity: every encoded line was placed. A mismatch means the budget math
    # and the chunking disagree — fail loud rather than silently drop data.
    if cursor != n_encoded:
        raise LayoutError(
            f"internal pagination error: placed {cursor} of {n_encoded} "
            "encoded lines"
        )

    if parity_pages:
        from .codec.engine import _frame_bytes

        padded_blocks = [b.ljust(block_bytes, b"\x00") for b in data_blocks]
        parity_blocks = _pagers.encode_page_parity(
            padded_blocks, parity_pages, c_exp=pgpar_field
        )
        # A parity line's PAYLOAD width matches the widest data-line payload
        # (not the full framed-line length -- that units bug printed Q rows
        # wider than the 60-char OCR-safe cap). Falls back to the safe default
        # width for an empty archive with no data lines.
        q_line_width = data_payload_width or _MACHINE_PAYLOAD_CHARS
        for offset, parity_block in enumerate(parity_blocks):
            page_no = data_total + 1 + offset
            q_lines = _frame_bytes("Q", parity_block, q_line_width)
            text_lines = list(q_lines)
            text_lines.append(format_page_footer(page_no, grand_total, q_lines))
            yield Page(
                number=page_no,
                total=grand_total,
                text_lines=text_lines,
                encoded_lines=q_lines,
            )

    try:
        next(encoded)
    except StopIteration:
        return
    raise LayoutError("encoded line iterator yielded more than n_encoded lines")

page_data_hash(page_lines)

Full hex SHA-256 of a page's data-line text block.

The block is "\n".join(page_lines) — the codec-framed lines carried by the page, in printed order. The footer keeps only the first :data:PAGE_HASH_CHARS characters of this digest.

Source code in src/glyphive/layout.py

def page_data_hash(page_lines: _ty.Sequence[str]) -> str:
    """Full hex SHA-256 of a page's data-line text block.

    The block is ``"\\n".join(page_lines)`` — the codec-framed lines carried by
    the page, in printed order. The footer keeps only the first
    :data:`PAGE_HASH_CHARS` characters of this digest.
    """
    block = "\n".join(page_lines)
    return hashlib.sha256(block.encode("utf-8")).hexdigest()

paginate(encoded_lines, meta, *, lines_per_page, parity_pages=0, emit_human_header=True)

Group encoded_lines into :class:Page objects with header/footer.

encoded_lines are the framed lines from :func:codec.encode (already RS-interleaved — layout does NOT recompute FEC, it only chunks). meta is the header dict (codec/comp/files/bytes/sha256 …); this function fills in meta["pages"] with the final page count BEFORE the header is formatted, so the printed pages= matches the physical count.

Chunking: each page's line budget is lines_per_page minus its overhead — 2 on page 1 (document header + footer) and 1 on every other page (footer). The document header is the first text_line of page 1; the footer is the last text_line of every page.

parity_pages (K) requests K additional whole-page-recovery pages after the data pages — see :func:iter_paginate.

Returns the list of pages in order. Raises ValueError if lines_per_page is too small to fit header+footer+data.

Source code in src/glyphive/layout.py

def paginate(
    encoded_lines: _ty.Sequence[str],
    meta: _ty.MutableMapping[str, _ty.Any],
    *,
    lines_per_page: int,
    parity_pages: int = 0,
    emit_human_header: bool = True,
) -> _ty.List[Page]:
    """Group ``encoded_lines`` into :class:`Page` objects with header/footer.

    ``encoded_lines`` are the framed lines from :func:`codec.encode` (already
    RS-interleaved — layout does NOT recompute FEC, it only chunks). ``meta`` is
    the header dict (``codec``/``comp``/``files``/``bytes``/``sha256`` …); this
    function fills in ``meta["pages"]`` with the final page count BEFORE the
    header is formatted, so the printed ``pages=`` matches the physical count.

    Chunking: each page's line budget is ``lines_per_page`` minus its overhead —
    2 on page 1 (document header + footer) and 1 on every other page (footer).
    The document header is the first ``text_line`` of page 1; the footer is the
    last ``text_line`` of every page.

    ``parity_pages`` (K) requests K additional whole-page-recovery pages after
    the data pages — see :func:`iter_paginate`.

    Returns the list of pages in order. Raises ``ValueError`` if
    ``lines_per_page`` is too small to fit header+footer+data.
    """
    encoded = list(encoded_lines)
    return list(
        iter_paginate(
            iter(encoded),
            len(encoded),
            meta,
            lines_per_page=lines_per_page,
            parity_pages=parity_pages,
            emit_human_header=emit_human_header,
        )
    )

parse_header(line)

Parse the compact display-only header line (inverse of :func:format_header).

Grammar: #!glyphive v<N> <codec>[,<comp>] files=<f> bytes=<b> pages=<p>. The first two non-prefix tokens are positional: a bare v<N> version and a codec[,comp] token (comp defaults to none when absent). Remaining tokens are k=v; integer keys (files/bytes/pages/pgpar) are coerced to int. Tolerates extra unknown k=v tokens (forward-compat), returned as strings. Raises :class:LayoutError if the #!glyphive prefix, the positional version/codec tokens, or a required k=v key is missing, or if an integer key is non-numeric. Restore never trusts this summary; :func:read_pages uses only the protected H frames.

Source code in src/glyphive/layout.py

def parse_header(line: str) -> _ty.Dict[str, _ty.Any]:
    """Parse the compact display-only header line (inverse of :func:`format_header`).

    Grammar: ``#!glyphive v<N> <codec>[,<comp>] files=<f> bytes=<b> pages=<p>``.
    The first two non-prefix tokens are positional: a bare ``v<N>`` version and a
    ``codec[,comp]`` token (``comp`` defaults to ``none`` when absent). Remaining
    tokens are ``k=v``; integer keys (``files``/``bytes``/``pages``/``pgpar``) are
    coerced to ``int``. Tolerates *extra* unknown ``k=v`` tokens (forward-compat),
    returned as strings. Raises :class:`LayoutError` if the ``#!glyphive`` prefix,
    the positional version/codec tokens, or a required ``k=v`` key is missing, or
    if an integer key is non-numeric. Restore never trusts this summary;
    :func:`read_pages` uses only the protected H frames.
    """
    stripped = line.strip()
    tokens = stripped.split()
    if not tokens or tokens[0] != HEADER_PREFIX:
        raise LayoutError(
            f"not a glyphive header: line must start with {HEADER_PREFIX!r}"
        )

    body = tokens[1:]
    if len(body) < 2 or "=" in body[0] or "=" in body[1]:
        raise LayoutError(
            "glyphive header must begin with positional v<N> and codec[,comp] tokens"
        )
    meta: _ty.Dict[str, _ty.Any] = {}

    version_token = body[0]
    if not version_token.startswith("v"):
        raise LayoutError(
            f"header version token must look like 'v1', got {version_token!r}"
        )
    try:
        meta["v"] = int(version_token[1:])
    except ValueError:
        raise LayoutError(
            f"header version token must look like 'v1', got {version_token!r}"
        ) from None

    codec_name, _, comp_name = body[1].partition(",")
    meta["codec"] = codec_name
    meta["comp"] = comp_name or "none"

    for token in body[2:]:
        if "=" not in token:
            # Bare token (no '='): ignore for forward-compat rather than crash.
            continue
        key, value = token.split("=", 1)
        if key in _INT_KEYS:
            try:
                meta[key] = int(value)
            except ValueError:
                raise LayoutError(
                    f"header key {key!r} must be an integer, got {value!r}"
                ) from None
        else:
            meta[key] = value

    missing = [key for key in _REQUIRED_KEYS if key not in meta]
    if missing:
        raise LayoutError(
            "header is missing required key(s): " + ", ".join(missing)
        )
    if meta["v"] != LAYOUT_VERSION:
        raise LayoutError(
            f"unsupported layout version {meta['v']} "
            f"(this build handles {LAYOUT_VERSION})"
        )
    return meta

read_pages(all_text_lines)

Parse a full transcript back into (header_meta, encoded_lines).

all_text_lines is every text line of a scanned/typed document — pages may be concatenated in any order and may repeat blank lines or OCR noise. This:

1. Finds and parses the #!glyphive header (raises if none is present).
2. Reads every PAGE n/total footer, using them to detect a missing page (raises :class:MissingPageError naming the absent page numbers) and to verify each page's data-block hash.
3. Collects the codec-framed L/P lines and returns them (in transcript order — codec.decode re-sorts by embedded index, so order does not matter).

Page-footer hash mismatches are advisory and collected separately in meta["_footer_hash_notes"] (they fire on essentially every OCR restore, because OCR-inserted spaces change the page-text hash while the L/P lines still decode via CRC/RS). They do NOT raise. Genuine page-integrity issues (reconstructed/missing pages) go in meta["_page_warnings"]. A missing header raises, and a whole missing page raises only when it is unrecoverable (beyond the page-parity budget and no surviving lines).

The returned meta is the parsed header dict plus:

• meta["_page_warnings"] : real page-integrity warnings (missing/ reconstructed pages) — worth surfacing at WARNING.
• meta["_footer_hash_notes"] : advisory per-page footer-hash mismatches — expected on OCR input, surfaced quietly.
• meta["_pages_seen"] : sorted list of page numbers found.

Source code in src/glyphive/layout.py

def read_pages(
    all_text_lines: _ty.Iterable[str],
) -> _ty.Tuple[_ty.Dict[str, _ty.Any], _ty.List[str]]:
    """Parse a full transcript back into ``(header_meta, encoded_lines)``.

    ``all_text_lines`` is every text line of a scanned/typed document — pages may
    be concatenated in any order and may repeat blank lines or OCR noise. This:

    1. Finds and parses the ``#!glyphive`` header (raises if none is present).
    2. Reads every ``PAGE n/total`` footer, using them to detect a *missing*
       page (raises :class:`MissingPageError` naming the absent page numbers) and
       to verify each page's data-block hash.
    3. Collects the codec-framed ``L``/``P`` lines and returns them (in transcript
       order — codec.decode re-sorts by embedded index, so order does not matter).

    Page-footer hash *mismatches* are advisory and collected separately in
    ``meta["_footer_hash_notes"]`` (they fire on essentially every OCR restore,
    because OCR-inserted spaces change the page-text hash while the L/P lines
    still decode via CRC/RS). They do NOT raise. Genuine page-integrity issues
    (reconstructed/missing pages) go in ``meta["_page_warnings"]``. A missing
    header raises, and a whole missing page raises only when it is unrecoverable
    (beyond the page-parity budget and no surviving lines).

    The returned ``meta`` is the parsed header dict plus:

    - ``meta["_page_warnings"]``     : real page-integrity warnings (missing/
      reconstructed pages) — worth surfacing at WARNING.
    - ``meta["_footer_hash_notes"]`` : advisory per-page footer-hash mismatches —
      expected on OCR input, surfaced quietly.
    - ``meta["_pages_seen"]``        : sorted list of page numbers found.
    """
    spool = io.BytesIO()
    header_meta, _count = read_pages_to_spool(all_text_lines, spool)
    spool.seek(0)
    return header_meta, [line.decode("utf-8").rstrip("\n") for line in spool]

read_pages_to_spool(all_text_lines, sink, *, line_conf=None)

Parse a transcript once and spool normalized codec lines sequentially.

line_conf (plan 3, optional): raw per-character OCR confidence, ONE ENTRY PER ELEMENT OF all_text_lines in the same order (None for a line with no confidence -- e.g. plain-text input, or a shorter line_conf than all_text_lines, padded with None). Only the entries belonging to lines that survive into the L/P codec stream matter; they are carried through page reordering/reconstruction and returned, in the FINAL SPOOL'S OWN LINE ORDER, as header_meta["_line_conf"] -- the shape :meth:Base16GCodec.decode_spool expects. When line_conf is None (the default), this is a no-op and behaves byte-identically to a build without this feature.

Source code in src/glyphive/layout.py

def read_pages_to_spool(
    all_text_lines: _ty.Iterable[str],
    sink: _ty.BinaryIO,
    *,
    line_conf: "_ty.Optional[_ty.Iterable[_ty.Optional[_ty.Sequence[float]]]]" = None,
) -> _ty.Tuple[_ty.Dict[str, _ty.Any], int]:
    """Parse a transcript once and spool normalized codec lines sequentially.

    ``line_conf`` (plan 3, optional): raw per-character OCR confidence,
    ONE ENTRY PER ELEMENT OF ``all_text_lines`` in the same order (``None``
    for a line with no confidence -- e.g. plain-text input, or a shorter
    ``line_conf`` than ``all_text_lines``, padded with ``None``). Only the
    entries belonging to lines that survive into the ``L``/``P`` codec
    stream matter; they are carried through page reordering/reconstruction
    and returned, in the FINAL SPOOL'S OWN LINE ORDER, as
    ``header_meta["_line_conf"]`` -- the shape :meth:`Base16GCodec.decode_spool`
    expects. When ``line_conf`` is ``None`` (the default), this is a no-op
    and behaves byte-identically to a build without this feature.
    """

    # --- Pass 1: recover the authoritative protected header. ----------------
    # The unrestricted ``#!glyphive ...`` line is retained for humans and old
    # tooling, but the restore path never trusts it.  In particular, there is no
    # OCR-repair guessing of a garbled codec name (e.g. a misread character in
    # ``base16g-crc16-rs``): codec selection comes from CRC-checked H frames
    # encoded entirely in the measured-safe bootstrap alphabet.
    header_frames: _ty.List[_ParsedMachineFrame] = []
    warnings: _ty.List[str] = []
    # Footer-hash mismatches are ADVISORY and fire on essentially every OCR
    # restore (OCR inserts interior spaces that change the page-text hash while
    # the L/P lines still decode byte-for-byte via CRC/RS). They are kept
    # separate from real page-integrity warnings so the CLI can log them quietly
    # instead of crying wolf on a clean restore.
    footer_hash_notes: _ty.List[str] = []
    pages_seen: _ty.Dict[int, int] = {}
    block_hash = hashlib.sha256()
    block_count = 0
    # Per-page encoded lines (data ``L``/``P`` AND parity ``Q``), keyed by page
    # number, in the order encountered on that page. Needed so a missing data
    # page's block can be reconstructed from parity and re-injected at the
    # correct spool position -- writing straight to ``sink`` as lines are read
    # (the pre-parity-pages behavior) cannot do that, since a page might need
    # to be rebuilt only after every page has been seen.
    # ``page_lines``/``current_page_lines`` carry ``(stripped_text, conf)``
    # pairs, not bare strings, so a survivor's raw OCR confidence (plan 3)
    # rides along through page reordering/reconstruction. ``conf`` is ``None``
    # unless ``line_conf`` was supplied.
    page_lines: "_ty.Dict[int, _ty.List[_ty.Tuple[str, _ty.Optional[_ty.Sequence[float]]]]]" = {}
    current_page_lines: "_ty.List[_ty.Tuple[str, _ty.Optional[_ty.Sequence[float]]]]" = []
    # Frame-shaped lines whose index token is unreadable (findings #1/#2): buffer
    # them for the current page block and flush to that page's number on its
    # footer, so the reader gets ``{page, raw}`` detail instead of a silent drop.
    unreadable_lines: _ty.List[_ty.Dict[str, _ty.Any]] = []
    pending_unreadable: _ty.List[str] = []
    payload_spec = None  # resolved lazily from the header once H frames are seen
    conf_source = () if line_conf is None else line_conf
    for line, conf in itertools.zip_longest(all_text_lines, conf_source, fillvalue=None):
        if line is None:
            break  # line_conf longer than all_text_lines (should not happen)
        frame = _parse_machine_frame(line, _MACHINE_HEADER_KIND)
        if frame is not None:
            header_frames.append(frame)
        stripped = line.strip()
        if not stripped:
            continue
        # Trim ``conf`` (aligned to the RAW, unstripped ``line``) by the same
        # leading/trailing whitespace ``.strip()`` just removed, so it stays
        # char-for-char aligned to ``stripped`` -- the text actually spooled
        # and later re-read by the codec. A length mismatch (unexpected
        # input) degrades to "no confidence" rather than guessing.
        line_conf_value = None
        if conf is not None and len(conf) == len(line):
            lead = len(line) - len(line.lstrip())
            line_conf_value = conf[lead:lead + len(stripped)]
        footer = _parse_footer(line)
        if footer is not None:
            expected = block_hash.hexdigest()[:PAGE_HASH_CHARS]
            if footer.digest.lower() != expected.lower():
                footer_hash_notes.append(
                    f"page {footer.n}/{footer.total}: footer hash "
                    f"{footer.digest!r} != computed {expected!r} "
                    f"(over {block_count} line(s))"
                )
            for raw in pending_unreadable:
                unreadable_lines.append({"page": footer.n, "raw": raw})
            pending_unreadable = []
            pages_seen[footer.n] = footer.total
            page_lines[footer.n] = current_page_lines
            current_page_lines = []
            block_hash = hashlib.sha256()
            block_count = 0
            continue
        if stripped.startswith(COMMENT_PREFIX):
            continue  # any '#!' line is a display-only comment, not a data line
        if _parse_machine_frame(line, _MACHINE_HEADER_KIND) is not None:
            continue  # protected machine header is not a payload line
        # The payload L/P frames use the SELECTED codec's alphabet (e.g. base32g
        # adds symbols), which differs from the base16g bootstrap used for the H
        # header frames. All H frames precede the first payload line, so resolve
        # the payload codec's spec once, lazily, before classifying L/P lines.
        if payload_spec is None:
            payload_spec = _resolve_payload_spec(header_frames)
        if _looks_like_encoded(line, payload_spec):
            encoded = stripped.encode("utf-8")
            current_page_lines.append((stripped, line_conf_value))
            if block_count:
                block_hash.update(b"\n")
            block_hash.update(encoded)
            block_count += 1
        elif _is_frame_shaped_but_unreadable(line, payload_spec):
            pending_unreadable.append(stripped)
        # else: OCR noise / blank-ish junk — ignored.

    # Trailing frame-shaped-but-unreadable lines with no footer after them still
    # deserve reporting; their page number is unknown.
    for raw in pending_unreadable:
        unreadable_lines.append({"page": None, "raw": raw})

    # Any trailing encoded lines with no footer after them belong to a page
    # whose footer was itself dropped by OCR; they are not attributable to a
    # known page number, so they cannot be placed by :func:`read_pages_to_spool`'s
    # per-page reconstruction. They are lost from ``page_lines`` here (as they
    # were before parity pages existed) -- the missing-page detection below
    # still catches and reports the gap.

    # --- Missing-page detection via integrity-protected machine metadata. ---
    header_meta = _decode_machine_header(header_frames)
    data_total = header_meta["pages"]  # D: data pages only (machine envelope)
    parity_budget = header_meta.get("pgpar", 0)
    block_bytes = header_meta.get("page_block_bytes", 0)
    pgpar_field = header_meta.get("pgpar_field", 8)
    grand_total = data_total + parity_budget
    inconsistent = sorted(
        n for n, observed_total in pages_seen.items()
        if observed_total != grand_total
    )
    if inconsistent:
        raise LayoutError(
            "machine footer total disagrees with protected header on page(s) "
            + ", ".join(str(n) for n in inconsistent)
        )
    missing = [n for n in range(1, grand_total + 1) if n not in pages_seen]
    missing_data = [n for n in missing if n <= data_total]
    missing_parity = [n for n in missing if n > data_total]

    reconstructed_pages: _ty.Set[int] = set()
    if missing_data and parity_budget and len(missing_data) <= parity_budget:
        # Enough parity budget in principle -- attempt page-level RS recovery.
        # Gather every data/parity page's block bytes (``"\n".join(lines)``,
        # the same convention :func:`page_data_hash`/pagination use), in block
        # order ``0..D+K-1`` (index i == page number i+1); missing pages are
        # ``None`` erasures. Present parity pages must decode as clean ``Q``
        # frames for their block to be trustworthy input to reconstruction.
        from .codec import pagers as _pagers
        from .codec.engine import decode_index, nibble_decode, split_frame

        def _q_block(lines: "_ty.List[_ty.Tuple[str, _ty.Any]]") -> _ty.Optional[bytes]:
            chunks: _ty.List[str] = []
            for line, _conf in lines:
                split = split_frame(line)
                if split is None:
                    return None
                label, payload, _check = split
                if label[:1] != "Q" or decode_index(label[1:]) is None:
                    return None
                chunks.append(payload)
            joined = "".join(chunks)
            if len(joined) % 2:
                return None
            try:
                return nibble_decode(joined, len(joined) // 2)
            except ValueError:
                return None

        blocks: _ty.List[_ty.Optional[bytes]] = []
        recoverable = True
        for n in range(1, grand_total + 1):
            if n in missing:
                blocks.append(None)
                continue
            if n <= data_total:
                block = "\n".join(
                    text for text, _conf in page_lines.get(n, [])
                ).encode("utf-8")
                blocks.append(block.ljust(block_bytes, b"\x00")[:block_bytes] if block_bytes else block)
            else:
                q_block = _q_block(page_lines.get(n, []))
                if q_block is None or len(q_block) != block_bytes:
                    recoverable = False
                    break
                blocks.append(q_block)

        if recoverable:
            try:
                rebuilt = _pagers.reconstruct_pages(
                    blocks, parity_budget, c_exp=pgpar_field
                )
            except _pagers.PageParityError:
                rebuilt = None
            if rebuilt is not None:
                for n in missing_data:
                    block = rebuilt[n - 1]
                    text = block.rstrip(b"\x00").decode("utf-8")
                    lines = text.split("\n") if text else []
                    # Reconstructed lines carry no OCR confidence of their
                    # own (they came from page-parity, not a printed
                    # character) -- decode falls back to whole-line erasure
                    # marking for any of these that still fail CRC, exactly
                    # as it always has.
                    page_lines[n] = [(ln, None) for ln in lines]
                    reconstructed_pages.add(n)
                warnings.append(
                    "reconstructed missing data page(s) "
                    + ", ".join(str(n) for n in missing_data)
                    + " from page-parity"
                )

    still_missing_data = [n for n in missing_data if n not in reconstructed_pages]
    if still_missing_data:
        # Do NOT hard-fail here: a wholly missing page is just a contiguous
        # erasure burst in the encoded-line stream, and the codec's
        # document-wide interleaved Reed-Solomon can recover it outright when
        # the parity budget suffices (user decision 2026-07-17). Record the
        # gap and let codec.decode try; if the budget is exceeded it raises its
        # own named CodecError. Only when NO codec lines survived at all is the
        # transcript genuinely unrecoverable at this layer.
        joined = ", ".join(str(n) for n in still_missing_data)
        warnings.append(
            f"missing page(s) {joined} of {data_total}: relying on codec "
            "Reed-Solomon to recover them from the surviving pages"
        )

    # --- Write the encoded-line spool in page order (1..D), skipping parity
    # pages (D+1..D+K) entirely -- they never reach codec.decode. Writing in
    # page order (rather than transcript order) guarantees a reconstructed
    # interior/last page's lines land at the correct position in the spool,
    # which downstream RS-parameter recovery depends on (a missing *last*
    # page otherwise truncates the stream shape -- a real bug found in early
    # testing).
    # ``spool_conf`` mirrors the written lines 1:1 -- this is the shape
    # :meth:`Base16GCodec.decode_spool` expects for its own ``char_conf``
    # (keyed by PHYSICAL LINE ORDER within the spool it reads).
    encoded_count = 0
    spool_conf: "_ty.List[_ty.Optional[_ty.Sequence[float]]]" = []
    for n in range(1, data_total + 1):
        for stripped, conf in page_lines.get(n, []):
            sink.write(stripped.encode("utf-8") + b"\n")
            spool_conf.append(conf)
            encoded_count += 1

    if encoded_count == 0 and still_missing_data:
        raise MissingPageError(still_missing_data, data_total)
    if missing_parity:
        warnings.append(
            "missing parity page(s) "
            + ", ".join(str(n) for n in missing_parity)
            + f" of {grand_total}: parity pages carry no user data and are not "
            "reconstructed"
        )

    header_meta["_page_warnings"] = warnings
    header_meta["_footer_hash_notes"] = footer_hash_notes
    header_meta["_pages_seen"] = sorted(pages_seen)
    header_meta["_unreadable_lines"] = unreadable_lines
    header_meta["_missing_pages"] = missing
    header_meta["_reconstructed_pages"] = sorted(reconstructed_pages)
    header_meta["_line_conf"] = spool_conf if line_conf is not None else None
    return header_meta, encoded_count

verify_page_footer(footer_line, page_lines)

Return True iff footer_line's hash matches page_lines.

A structurally invalid footer line returns False. Comparison is case-insensitive on the hex digest.

Source code in src/glyphive/layout.py

def verify_page_footer(
    footer_line: str, page_lines: _ty.Sequence[str]
) -> bool:
    """Return True iff ``footer_line``'s hash matches ``page_lines``.

    A structurally invalid footer line returns ``False``. Comparison is
    case-insensitive on the hex digest.
    """
    try:
        parsed = _parse_footer(footer_line)
    except LayoutError:
        return False
    if parsed is None:
        return False
    expected = page_data_hash(page_lines)[:PAGE_HASH_CHARS]
    return parsed.digest.lower() == expected.lower()