Skip to content

Codec API

Use glyphive.codec.get(name) to resolve a fresh codec instance. The built-in wire identifier is base16g-crc16-rs.

glyphive.codec

Named printable codecs. Importing this package registers the whole built-in family (base16g-crc16-rs and the other radix codecs) via the shared engine.

__all__ = ['Codec', 'Base16GCodec', 'Base8GCodec', 'Base16Codec', 'Base32Codec', 'Base32CCodec', 'Base32GCodec', 'Base64Codec', 'Base85Codec', 'Z85Codec', 'Base64GCodec', 'BaseMaxGCodec', 'available', 'get', 'names'] module-attribute

Base16Codec

Bases: RadixCodec

Standard hexadecimal (0-9 A-F), 4 bits/char. Not OCR-tuned (use base16g).

name = 'base16-crc16-rs' class-attribute instance-attribute

Base16GCodec

Bases: RadixCodec

base16g-crc16-rs: the 16-char stock-OCR-safe default codec.

The plain shared engine (:class:~glyphive.codec.engine.RadixCodec) with the :data:~glyphive.codec.engine.BASE16G spec -- 4 bits/char, no trained OCR model needed. Every other codec here is the same engine with a denser or textbook alphabet.

name = 'base16g-crc16-rs' class-attribute instance-attribute

Base32CCodec

Bases: RadixCodec

Crockford base32 (0-9 A-Z minus I L O U), 5 bits/char. Not OCR-tuned.

name = 'base32c-crc16-rs' class-attribute instance-attribute

Base32Codec

Bases: RadixCodec

Standard RFC-4648 base32 (A-Z 2-7), 5 bits/char. Not OCR-tuned (use base32g).

name = 'base32-crc16-rs' class-attribute instance-attribute

Base32GCodec

Bases: RadixCodec

base32g: glyphive's 32-char (5 bits/char) codec — 25% denser than base16g.

Reads at 0.0% CER with a per-font trained model; ~14.8% on stock OCR. Denser than base16g but needs the matching trained model for reliable scan restore.

name = 'base32g-crc16-rs' class-attribute instance-attribute

Base64Codec

Bases: RadixCodec

64-char (6 bits/char) codec — densest; needs a trained model to restore.

name = 'base64-crc16-rs' class-attribute instance-attribute

Base64GCodec

Bases: RadixCodec

base64g: glyphive's curated 64-glyph set (confusion-distinct favoring).

6 bits/char like base64, but the alphabet favors OCR-distinct glyphs. Needs a trained model for reliable restore (as with base32g/base64).

name = 'base64g-crc16-rs' class-attribute instance-attribute

Base85Codec

Bases: RadixCodec

Standard base85 (RFC-1924-ish), group-packed 4->5. Densest; interop only.

name = 'base85-crc16-rs' class-attribute instance-attribute

Base8GCodec

Bases: RadixCodec

Sparse 8-char (3 bits/char) codec — most OCR-robust, least dense.

name = 'base8g-crc16-rs' class-attribute instance-attribute

BaseMaxGCodec

Bases: RadixCodec

base-maxg: glyphive's 43-glyph max-distinct set, group-packed 6->9.

The largest mutually-distinct alphabet measured on stock OCR (~43/font). Needs a trained model for reliable restore, like base32g; denser than 32.

name = 'basemaxg-crc16-rs' class-attribute instance-attribute

Codec

Bases: ABC

Base class for named byte-to-lines codecs.

Concrete subclasses register themselves at class definition time. Registry classes must be no-argument constructible; a future factory contract can relax that restriction without complicating this registry.

name class-attribute

__init_subclass__(**kwargs)

Source code in src/glyphive/codec/_base.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
def __init_subclass__(cls, **kwargs: _ty.Any) -> None:
    super().__init_subclass__(**kwargs)
    if any(
        getattr(getattr(cls, method, None), "__isabstractmethod__", False)
        for method in ("encode", "decode")
    ):
        return
    # An intermediate base that implements the codec protocol but is not
    # itself a selectable codec (e.g. the shared RadixCodec engine) opts out
    # by setting ``abstract = True`` in its OWN body. Checked on the class
    # dict, not via inheritance, so concrete subclasses still register.
    if cls.__dict__.get("abstract", False):
        return
    name = getattr(cls, "name", None)
    if not isinstance(name, str) or not re.fullmatch(r"[a-z][a-z0-9_-]*", name):
        raise ValueError(
            f"codec {cls.__module__}.{cls.__qualname__} must define a valid "
            "lowercase ASCII name"
        )
    if name in Codec._registry:
        existing = Codec._registry[name]
        raise ValueError(
            f"duplicate codec name {name!r}: "
            f"{existing.__module__}.{existing.__qualname__} and "
            f"{cls.__module__}.{cls.__qualname__}"
        )
    Codec._registry[name] = cls

available() classmethod

Return names whose implementation reports itself available.

Source code in src/glyphive/codec/_base.py 
55
56
57
58
59
60
61
62
@classmethod
def available(cls) -> _ty.List[str]:
    """Return names whose implementation reports itself available."""
    return sorted(
        name
        for name, implementation in Codec._registry.items()
        if implementation.is_available()
    )

decode(lines, **options) abstractmethod

Decode printable lines as bytes.

Source code in src/glyphive/codec/_base.py 
105
106
107
@abstractmethod
def decode(self, lines: _ty.Iterable[str], **options: _ty.Any) -> bytes:
    """Decode printable lines as bytes."""

encode(data, **options) abstractmethod

Encode bytes as printable lines.

Source code in src/glyphive/codec/_base.py 
101
102
103
@abstractmethod
def encode(self, data: bytes, **options: _ty.Any) -> _ty.List[str]:
    """Encode bytes as printable lines."""

get(name) classmethod

Return a fresh no-argument codec or raise an actionable error.

Source code in src/glyphive/codec/_base.py 
64
65
66
67
68
69
70
71
72
73
74
@classmethod
def get(cls, name: str) -> "Codec":
    """Return a fresh no-argument codec or raise an actionable error."""
    try:
        implementation = Codec._registry[name]
    except (KeyError, TypeError):
        valid = ", ".join(Codec.names()) or "(none)"
        raise ValueError(
            f"unknown codec {name!r}; available codecs: {valid}"
        ) from None
    return implementation()

is_available() classmethod

Return whether this implementation can be selected.

Source code in src/glyphive/codec/_base.py 
76
77
78
79
@classmethod
def is_available(cls) -> bool:
    """Return whether this implementation can be selected."""
    return True

names() classmethod

Return all registered codec names in stable order.

Source code in src/glyphive/codec/_base.py 
50
51
52
53
@classmethod
def names(cls) -> _ty.List[str]:
    """Return all registered codec names in stable order."""
    return sorted(Codec._registry)

Z85Codec

Bases: RadixCodec

ZeroMQ Z85, group-packed 4->5. Interop only (85 glyphs, not OCR-safe).

name = 'z85-crc16-rs' class-attribute instance-attribute

available()

Return registered codec names currently available to use.

Source code in src/glyphive/codec/__init__.py 
57
58
59
def available() -> _ty.List[str]:
    """Return registered codec names currently available to use."""
    return Codec.available()

get(name)

Return a fresh registered codec implementation by name.

Source code in src/glyphive/codec/__init__.py 
47
48
49
def get(name: str) -> Codec:
    """Return a fresh registered codec implementation by name."""
    return Codec.get(name)

names()

Return all registered codec names in stable order.

Source code in src/glyphive/codec/__init__.py 
52
53
54
def names() -> _ty.List[str]:
    """Return all registered codec names in stable order."""
    return Codec.names()

Codec engine and specs

The shared codec engine (RadixCodec, framing, CRC-16, Reed-Solomon, the machine-frame API) lives in glyphive.codec.engine; every concrete codec (base16g-crc16-rs and the denser family) is a thin spec-bearing subclass in glyphive.codec.radix.

glyphive.codec.engine

glyphive codec base16g-crc16-rs — byte stream ↔ OCR-safe printable text lines.

This module is the core of the format. The alphabet is the measured-safe 16-character set below, every line carries its own CRC check, and Reed-Solomon parity lets small OCR errors be corrected rather than silently propagating. It is pure: bytes in, list-of-str lines out (and back). No file I/O, no pathlib_next, no argparse here.

The alphabet

ALPHABET = "ABCDHKLMPRTVXY34" (16 characters -> 4 bits/character). This is not hand-picked: it is the exact set measured safe on Courier 8pt @ 300 DPI / Tesseract 5.4.0 (tools/ocr_font_report.py) — 16/16 characters read back with 0% error, 0% line-insertion rate, and zero corrupting confusions over ~6,600 sampled characters. The previous 32-character Crockford-Base32 alphabet is not safe on this channel: Crockford keeps Q and J while excluding O/I/L/U, but Q is misread as O (which then alias-maps to 0) and J is misread as I (which then alias-maps to 1) — both are silent, undetectable data corruption, not a recoverable erasure. Trading 5 bits/char for 4 costs 25% more pages; that is the price of a format that actually round-trips. See the public wire-format and OCR guides for the derivation and multi-radix measurement method.

Frame grammar (one printed line)

Every printed line has exactly this shape (single ASCII spaces as separators)::

<kind><idx> <payload> [<line-parity>] #<check>
  • <kind> : L for a data line, P for a Reed-Solomon parity line.
  • <idx> : the line's 0-based index within its stream (data lines and parity lines are indexed independently, each starting at 0), rendered as INDEX_WIDTH (5) alphabet characters with a fixed per-position XOR mask applied before rendering (see encode_index) so the token never prints as a run of identical glyphs.
  • <payload>: exactly line_width alphabet characters, EXCEPT the last data line, which may be shorter (it carries the remainder). Parity lines are always exactly line_width wide.
  • <line-parity>: OPTIONAL, present iff the stream's nsym_line (recorded in the group header) is non-zero. nsym_line Reed-Solomon parity bytes computed over idx_token.encode() + <the raw bytes the payload encodes>, rendered in the line's own alphabet (ceil(nsym_line * 8 / spec.bits) characters). On CRC failure, decode blind-corrects this token+payload codeword using these bytes, re-renders it, and accepts the fix only if the RE-COMPUTED check field then matches the originally printed one (see "Per-line Reed-Solomon" below). NOT covered by the check field.
  • # : a literal # marking the start of the check field.
  • <check> : CHECK_WIDTH (4) alphabet characters (16 bits) encoding a 16-bit CRC-16/CCITT (poly 0x1021, init 0xFFFF) computed over the bytes kind.encode() + idx_token.encode() + payload.encode() — i.e. over the printed kind letter, index token, and payload characters (the optional line-parity field is deliberately excluded), so a human can recompute it from the page by hand. Covering kind means an OCR misread that flips L<->P (or H/T/Q in layout.py) now fails the check instead of silently producing a CRC-valid phantom line at the same index.

Why 4 check characters (not 2), and why 4 is exact now

A CRC-16 is a 16-bit value (0..65535). At 4 bits/character (this alphabet), 2 characters would hold only 8 bits (256 values) — far too few. 4 characters hold exactly 16 bits: CHECK_WIDTH(4) * 4 bits/char = 16 bits, which is the exact width of a CRC-16 with zero waste and, critically, no truncation. (The prior 5-bit/char alphabet needed 4 characters too, but only used 16 of the 20 bits it spent — this alphabet spends exactly what it needs.) This detects any single-character OCR substitution in a line and localizes it to exactly that one line without decoding anything downstream; the embedded index token catches OCR line-merge / line-drop.

Header layout of the encoded stream

The very first bytes of a group's payload carry a 9-byte binary header so decode can reconstruct the exact original byte length (nibble bit-packing pads the final 4-bit group, so the raw length must be carried, never guessed):

b"B1" | version:u8 | nsym:u8 | nsym_line:u8 | orig_len:u32-big-endian

nsym_line (0, 2, or 4) is the per-line Reed-Solomon parity byte count requested at encode time (see "Per-line Reed-Solomon" below); decode needs it before it can even find the payload/check boundary of a line, so it is also detected STRUCTURALLY from the printed line shape (4 whitespace-delimited tokens vs 3) and cross-checked against this header field once the header itself is recovered -- a mismatch is a hard decode error, never silently resolved either way. The header bytes are part of the RS-protected data stream, so they are covered by parity and per-line CRC just like the payload.

Reed-Solomon parity

Parity is computed over the group's data bytes (header + original bytes) with reedsolo.RSCodec(nsym). The data is split into interleaved RS blocks of at most 255 bytes so arbitrarily large documents are supported while each block stays within the GF(2^8) code length. Glyphive chooses one nsym in 2..100 whose aggregate parity bytes across all blocks are closest to protected_bytes * parity_ratio. Because the per-line CRC tells decode exactly which lines are wrong, decode feeds those lines' byte positions to RS as erasures — RS corrects up to nsym erasures per block (twice its blind-error budget).

The document parity BYTE STREAM itself is interleaved symbol-major: parity byte j of block b sits at printed-stream offset j * nblocks + b (not b * nsym + j). A single bad parity LINE therefore only ever costs each block at most ceil(line_width_bytes / nblocks) parity symbols instead of erasing one block's entire parity budget outright -- the same burst-spreading logic the data stream already gets from its own interleaving, just applied to the parity stream too. Pure permutation, zero size cost.

Per-line Reed-Solomon (nsym_line, default 2)

A single misread character normally erases its WHOLE line for the document- level RS tier above (the line's CRC fails, so every byte the line carries becomes an erasure). When nsym_line is non-zero, each line additionally carries nsym_line Reed-Solomon parity bytes computed over idx_token.encode() + <the raw bytes that line's payload encodes> and rendered in the line's own alphabet, between the payload and the check field (the parity field itself is NOT covered by the check). On a CRC failure, decode treats the printed token+payload+line-parity as one small RS codeword, blind-corrects up to nsym_line // 2 byte errors (no known erasure positions -- this tier runs before anything is treated as an erasure), re-renders the corrected token/payload, and RECOMPUTES the check field: only if that recomputed check matches the originally printed one is the correction trusted. This keeps the "recoverable" property of the check field intact -- an RS miscorrection almost never reproduces the original 16-bit CRC by chance -- while turning many single/double-character line errors into a silent, free repair that never touches the document-level RS erasure budget at all. Lines the in-line tier cannot verify fall through to the existing repair/erasure tiers unchanged.

Decode oracle discipline (CRITICAL)

Correctness is judged SOLELY by the per-line CRC check and RS correction. There is deliberately no "try confusable substitutions and keep whatever decompresses further" search anywhere in this module. If a line's CRC fails AND RS cannot correct the resulting erasures, decode RAISES a clear exception naming the exact failing line label. It never guesses or mutates data to make it "work".

OCR-confidence erasure hint (char_conf, does NOT weaken the above)

:meth:RadixCodec.decode_spool optionally accepts char_conf: per-line OCR character confidence, keyed by PHYSICAL LINE ORDER within the encoded spool (not by the printed index, which may itself be corrupt on a CRC-failed line). Today, a CRC-failed line contributes its ENTIRE byte span as document- level Reed-Solomon erasures, even though the typical cause is one or two misread characters -- wasting the erasure budget ~line-width-fold. When char_conf names specific low-confidence character positions within an otherwise CRC-failed line (fewer than max_suspects), only the bytes those characters map to are marked as erasures; the line's other ("soft") bytes enter the RS stream as ordinary, unverified data instead of being zeroed.

This is a HINT about WHERE to erase, never about WHAT the bytes are or whether a line is accepted -- it changes erasure positions only. Acceptance is unchanged: a block still must satisfy Reed-Solomon's erasure/ error budget, and the whole-document SHA-256 gate (restore/decode.py) is still the final oracle. A two-pass, block-local safety valve makes the optimization strictly no-worse than today: if a block still fails RS with the narrower char-level erasures, that block alone is retried with the CRC-failed line(s) it touches erased across their FULL span (today's behaviour) before the block is given up as uncorrectable. See _assemble_to_spool and RadixCodec._decode_hardened_spool.

ALPHABET = 'ABCDHKLMPRTVXY34' module-attribute

BASE16G = _RadixSpec(name='base16g-crc16-rs', alphabet=ALPHABET, bits=4, check_width=4, index_width=5, index_mask=(7, 13, 2, 11, 4)) module-attribute

CHECK_WIDTH = 4 module-attribute

DEFAULT_CONF_THRESHOLD = 0.6 module-attribute

DEFAULT_MAX_SUSPECTS = 6 module-attribute

INDEX_WIDTH = 5 module-attribute

MACHINE_SPEC = BASE16G module-attribute

__all__ = ['ALPHABET', 'CodecError', 'RadixCodec', 'MACHINE_SPEC', 'ParsedMachineFrame', 'machine_frame', 'parse_machine_frame', 'machine_check', 'rs_protect', 'rs_recover', 'crc16_ccitt', 'nibble_encode', 'nibble_decode', 'encoded_line_count', 'describe_line_stream', 'StreamShape', 'align_payload_char_conf', 'DEFAULT_CONF_THRESHOLD', 'DEFAULT_MAX_SUSPECTS'] module-attribute

CodecError

Bases: ValueError

Raised when a line fails its CRC and RS cannot correct it.

Subclasses :class:ValueError so callers may catch either. The message always names the exact failing line label (e.g. L00042).

ParsedMachineFrame

Bases: NamedTuple

One parsed machine frame: idx (None if the label is unreadable), the normalized safe-alphabet payload, and ok (the kind-covered CRC matched). A structurally-broken but index-readable frame is returned with ok=False so the caller can localize it; a wholly unreadable one is None.

idx instance-attribute

ok instance-attribute

payload instance-attribute

RadixCodec

Bases: Codec

Shared codec engine: CRC-16-CCITT per line / Reed-Solomon / spooled I/O.

This is the base every concrete radix codec subclasses (base16g and the denser base8/base32g/base64/... in :mod:glyphive.codec.radix), overriding only name and _spec. All the framing/RS/header/spool machinery is radix-agnostic and driven by self._spec. It is not itself a selectable codec (abstract = True); _spec defaults to :data:BASE16G so the machine-frame bootstrap and internal helpers have a concrete spec to work with.

abstract = True class-attribute instance-attribute

decode(lines)

Decode framed lines using only CRC and RS as correctness oracles.

Source code in src/glyphive/codec/engine.py 
2415
2416
2417
2418
2419
2420
2421
2422
2423
def decode(self, lines: _ty.Iterable[str]) -> bytes:
    """Decode framed lines using only CRC and RS as correctness oracles."""
    encoded = _io.BytesIO()
    for line in lines:
        encoded.write(line.encode("utf-8") + b"\n")
    encoded.seek(0)
    output = _io.BytesIO()
    self.decode_spool(encoded, output)
    return output.getvalue()

decode_spool(encoded_source, payload_sink, *, char_conf=None, conf_threshold=DEFAULT_CONF_THRESHOLD, max_suspects=DEFAULT_MAX_SUSPECTS, nsym_line=None, temp_dir=None)

Decode an encoded-line spool with only offsets and RS blocks in RAM.

nsym_line (optional): the AUTHORITATIVE per-line parity byte count, supplied by the caller when it already knows it from the protected machine header. This matters because the printed line-parity field has no delimiter separating it from the payload: on a transcript whose interior spaces were stripped -- which is the NORMAL constrained-OCR whitelist case -- the structural token-count heuristic cannot tell "wide payload, no line-parity" from "narrower payload plus parity", so it falls back to 0 and mis-frames every line. Passing the header's value removes that guess entirely. When None, the heuristic is used (the raw-codec path, where there is no layout header to consult).

char_conf (plan 3, optional): per-line OCR character confidence, keyed by PHYSICAL LINE ORDER within encoded_source (char_conf[i] is the raw per-character confidence of the i-th line as read from the spool, or None) -- never by printed index, which may itself be corrupt on a CRC-failed line. When absent (the default), decode is byte-identical to a build without this feature. See the module docstring's "OCR-confidence erasure hint" section for exactly what this does and does not change about acceptance.

Source code in src/glyphive/codec/engine.py 
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
2440
2441
2442
2443
2444
2445
2446
2447
2448
2449
2450
2451
2452
2453
2454
2455
2456
2457
2458
2459
2460
2461
2462
2463
2464
2465
2466
2467
2468
2469
2470
2471
2472
2473
2474
2475
2476
2477
2478
2479
2480
2481
2482
2483
2484
2485
2486
2487
2488
2489
2490
2491
2492
def decode_spool(
    self,
    encoded_source: _ty.BinaryIO,
    payload_sink: _ty.BinaryIO,
    *,
    char_conf: _ty.Optional[
        _ty.Sequence[_ty.Optional[_ty.Sequence[float]]]
    ] = None,
    conf_threshold: float = DEFAULT_CONF_THRESHOLD,
    max_suspects: int = DEFAULT_MAX_SUSPECTS,
    nsym_line: _ty.Optional[int] = None,
    temp_dir: _ty.Optional[str] = None,
) -> None:
    """Decode an encoded-line spool with only offsets and RS blocks in RAM.

    ``nsym_line`` (optional): the AUTHORITATIVE per-line parity byte count,
    supplied by the caller when it already knows it from the protected
    machine header. This matters because the printed line-parity field has
    no delimiter separating it from the payload: on a transcript whose
    interior spaces were stripped -- which is the NORMAL constrained-OCR
    whitelist case -- the structural token-count heuristic cannot tell
    "wide payload, no line-parity" from "narrower payload plus parity", so
    it falls back to 0 and mis-frames every line. Passing the header's value
    removes that guess entirely. When ``None``, the heuristic is used (the
    raw-codec path, where there is no layout header to consult).

    ``char_conf`` (plan 3, optional): per-line OCR character confidence,
    keyed by PHYSICAL LINE ORDER within ``encoded_source`` (``char_conf[i]``
    is the raw per-character confidence of the i-th line as read from the
    spool, or ``None``) -- never by printed index, which may itself be
    corrupt on a CRC-failed line. When absent (the default), decode is
    byte-identical to a build without this feature. See the module
    docstring's "OCR-confidence erasure hint" section for exactly what
    this does and does not change about acceptance.
    """
    # Plan-1 decode-hardening pre-pass: repair single-char errors, and stop
    # CRC-failed lines with poisoned index tokens from destroying the stream
    # geometry. Runs into a temp spool only when it actually changes lines;
    # a clean transcript pays one streaming scan and reuses the original.
    # ``char_conf`` (raw, per-physical-line) rides along unchanged for kept
    # lines and is dropped along with any line the hardening pass drops --
    # see ``_preprocess_spool``.
    with _tempfile.TemporaryFile(dir=temp_dir) as _hardened:
        changed, hardened_conf = _preprocess_spool(
            encoded_source, _hardened, self._spec, char_conf=char_conf,
            nsym_line=nsym_line,
        )
        if changed:
            _hardened.seek(0)
            self._decode_hardened_spool(
                _hardened,
                payload_sink,
                temp_dir=temp_dir,
                char_conf=hardened_conf,
                conf_threshold=conf_threshold,
                max_suspects=max_suspects,
                nsym_line=nsym_line,
            )
            return
    self._decode_hardened_spool(
        encoded_source,
        payload_sink,
        temp_dir=temp_dir,
        char_conf=char_conf,
        conf_threshold=conf_threshold,
        max_suspects=max_suspects,
        nsym_line=nsym_line,
    )

encode(data, *, line_width=60, parity_ratio=0.12, nsym_line=2)

Encode bytes into OCR-safe framed data and parity lines.

nsym_line (default 2; must be 0, 2, or 4) is the per-line Reed- Solomon parity budget: 0 disables the in-line correction tier (the line-parity field is omitted entirely), 2/4 add that many parity bytes to every printed line so many single/double-character OCR errors self-heal without ever touching the document-level RS erasure budget (see the module docstring's "Per-line Reed-Solomon" section).

Source code in src/glyphive/codec/engine.py 
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
def encode(
    self,
    data: bytes,
    *,
    line_width: int = 60,
    parity_ratio: float = 0.12,
    nsym_line: int = 2,
) -> _ty.List[str]:
    """Encode bytes into OCR-safe framed data and parity lines.

    ``nsym_line`` (default 2; must be 0, 2, or 4) is the per-line Reed-
    Solomon parity budget: 0 disables the in-line correction tier (the
    line-parity field is omitted entirely), 2/4 add that many parity
    bytes to every printed line so many single/double-character OCR
    errors self-heal without ever touching the document-level RS erasure
    budget (see the module docstring's "Per-line Reed-Solomon" section).
    """
    if line_width < 1:
        raise ValueError("line_width must be >= 1")
    if not 0 < parity_ratio < 1:
        raise ValueError("parity_ratio must be in (0, 1)")
    if nsym_line not in (0, 2, 4):
        raise ValueError("nsym_line must be 0, 2, or 4")

    orig_len = len(data)
    unpadded_len = _HEADER_LEN + orig_len
    bytes_per_line = _bytes_per_line(line_width, self._spec)
    pad_bytes = _runt_pad_bytes(unpadded_len, bytes_per_line, line_width, self._spec)
    # ``orig_len`` recorded in the header is the TRUE unpadded length --
    # decode already truncates its output to exactly this many bytes (see
    # ``_assemble_to_spool`` callers), so trailing pad bytes appended here,
    # after the real payload but before RS protection, are transparently
    # dropped on decode with no decoder change: they exist purely to keep
    # the final ``L`` line's printed payload off the runt-line trap (see
    # ``_runt_pad_bytes`` / the 2026-07-21 fourpt-runt-line measurement).
    protected_len = unpadded_len + pad_bytes
    nsym = _select_nsym(protected_len, parity_ratio)
    header = _make_header(nsym, orig_len, nsym_line)
    stream = header + data + b"\x00" * pad_bytes
    data_bytes, parity_bytes, _nblocks = _rs_encode(stream, nsym)

    lines: _ty.List[str] = []
    lines.extend(
        _frame_bytes("L", data_bytes, line_width, self._spec, nsym_line=nsym_line)
    )
    lines.extend(
        _frame_bytes("P", parity_bytes, line_width, self._spec, nsym_line=nsym_line)
    )
    return lines

iter_encode(source, data_len, *, line_width=60, parity_ratio=0.12, nsym_line=2, temp_dir=None)

Encode a seekable payload source with bounded Python allocations.

The existing interleaved RS layout is preserved exactly (document-level data interleave), plus the v2 symbol-major PARITY interleave (see :func:_parity_position). A temporary parity spool avoids retaining parity or framed lines in memory: each block's parity is written contiguously (cheap sequential writes), then re-read through the interleaved position mapping when framing P lines -- the spool never exceeds nsym * nblocks bytes, which stays tiny (RS blocks are capped at 255 bytes each). mmap is used when the source exposes a file descriptor so each RS codeword is at most 255 bytes in Python memory. nsym_line (default 2) is documented on :meth:encode.

Source code in src/glyphive/codec/engine.py 
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
2347
2348
2349
2350
2351
2352
2353
2354
2355
2356
2357
2358
2359
2360
2361
2362
2363
2364
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
2386
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
def iter_encode(
    self,
    source: _ty.BinaryIO,
    data_len: int,
    *,
    line_width: int = 60,
    parity_ratio: float = 0.12,
    nsym_line: int = 2,
    temp_dir: _ty.Optional[str] = None,
) -> _ty.Iterator[str]:
    """Encode a seekable payload source with bounded Python allocations.

    The existing interleaved RS layout is preserved exactly (document-level
    data interleave), plus the v2 symbol-major PARITY interleave (see
    :func:`_parity_position`). A temporary parity spool avoids retaining
    parity or framed lines in memory: each block's parity is written
    contiguously (cheap sequential writes), then re-read through the
    interleaved position mapping when framing ``P`` lines -- the spool
    never exceeds ``nsym * nblocks`` bytes, which stays tiny (RS blocks
    are capped at 255 bytes each). mmap is used when the source exposes a
    file descriptor so each RS codeword is at most 255 bytes in Python
    memory. ``nsym_line`` (default 2) is documented on :meth:`encode`.
    """
    if data_len < 0 or data_len > 0xFFFFFFFF:
        raise ValueError("data length must fit the codec's unsigned 32-bit header")
    if nsym_line not in (0, 2, 4):
        raise ValueError("nsym_line must be 0, 2, or 4")
    (
        bytes_per_line,
        protected_len,
        nsym,
        nblocks,
        _data_lines,
        _parity_lines,
    ) = _encoding_shape(data_len, line_width, parity_ratio, self._spec, nsym_line)
    # ``protected_len`` may exceed ``_HEADER_LEN + data_len`` by a handful
    # of runt-avoidance pad bytes (see ``_runt_pad_bytes``); the header's
    # ``orig_len`` stays the TRUE unpadded length so decode truncates back
    # to it unchanged. Positions at/after ``_HEADER_LEN + data_len`` read
    # as zero bytes below (the source is never asked for more than its
    # real ``data_len`` bytes).
    pad_bytes = protected_len - (_HEADER_LEN + data_len)
    header = _make_header(nsym, data_len, nsym_line)
    source.seek(0, 2)
    actual_len = source.tell()
    if actual_len != data_len:
        direction = "truncated" if actual_len < data_len else "grew"
        raise ValueError(
            f"source {direction} before encoding "
            f"(expected {data_len} bytes, found {actual_len})"
        )
    source.seek(0)
    mapping = None
    if data_len:
        try:
            mapping = _mmap.mmap(source.fileno(), 0, access=_mmap.ACCESS_READ)
        except (AttributeError, OSError, ValueError):
            mapping = None
    try:
        with _tempfile.TemporaryFile(dir=temp_dir) as block_parity_spool:
            rs = _reedsolo.RSCodec(nsym)
            for block_index in range(nblocks):
                block = bytearray()
                for position in range(block_index, protected_len, nblocks):
                    if position < _HEADER_LEN:
                        block.append(header[position])
                    elif position >= _HEADER_LEN + data_len:
                        block.append(0)  # runt-avoidance pad byte
                    elif mapping is not None:
                        block.append(mapping[position - _HEADER_LEN])
                    else:
                        source.seek(position - _HEADER_LEN)
                        value = source.read(1)
                        if not value:
                            raise ValueError("source was truncated during RS encoding")
                        block.extend(value)
                encoded = rs.encode(bytes(block))
                block_parity_spool.write(encoded[len(block):])

            source.seek(0)
            pending = bytearray(header)
            data_remaining = data_len
            pad_remaining = pad_bytes
            line_index = 0
            while pending or data_remaining or pad_remaining:
                needed = bytes_per_line - len(pending)
                if needed and data_remaining:
                    chunk = source.read(min(needed, data_remaining))
                    if not chunk:
                        raise ValueError("source was truncated during data framing")
                    pending.extend(chunk)
                    data_remaining -= len(chunk)
                    needed = bytes_per_line - len(pending)
                if needed and not data_remaining and pad_remaining:
                    # Runt-avoidance pad: zero bytes appended after the real
                    # payload, before RS (see ``_runt_pad_bytes``); decode
                    # truncates its output back to the header's true
                    # ``orig_len`` so these never reach the caller.
                    take = min(needed, pad_remaining)
                    pending.extend(b"\x00" * take)
                    pad_remaining -= take
                if len(pending) < bytes_per_line and (data_remaining or pad_remaining):
                    continue
                chunk = bytes(pending[:bytes_per_line])
                del pending[:bytes_per_line]
                if line_index >= self._spec.max_idx:
                    raise ValueError(f"L stream exceeds {self._spec.max_idx} lines")
                yield _frame(
                    "L", line_index, chunk, self._spec, nsym_line=nsym_line
                )
                line_index += 1
            if source.read(1):
                raise ValueError("source grew during data framing")

            # Re-read the contiguously-written per-block parity through the
            # v2 interleaved (symbol-major) position mapping -- the spool
            # holds nsym*nblocks bytes total (always tiny; RS blocks are
            # capped at 255 bytes), so a full in-memory read is bounded.
            block_parity_spool.seek(0)
            contiguous_parity = block_parity_spool.read()
            interleaved_parity = bytearray(len(contiguous_parity))
            for b in range(nblocks):
                for j in range(nsym):
                    interleaved_parity[_parity_position(j, b, nblocks)] = (
                        contiguous_parity[b * nsym + j]
                    )
            with _io.BytesIO(bytes(interleaved_parity)) as parity:
                yield from _frame_stream(
                    "P",
                    parity,
                    nblocks * nsym,
                    bytes_per_line,
                    self._spec,
                    nsym_line=nsym_line,
                )
    finally:
        if mapping is not None:
            mapping.close()

StreamShape

Bases: NamedTuple

Realized Reed-Solomon shape of an encoded L/P line stream.

nsym/nblocks are None when the stream shape is ambiguous (_candidate_nsym returned other than exactly one candidate) -- never guessed. nsym is the per-interleaved-block erasure budget. nsym_line is detected STRUCTURALLY from the printed line shape (3 vs 4 tokens/line), the same heuristic decode itself uses before the group header is recoverable -- not read from the (possibly-undecoded) header.

data_bytes instance-attribute

data_lines instance-attribute

nblocks instance-attribute

nsym instance-attribute

nsym_line = 0 class-attribute instance-attribute

parity_bytes instance-attribute

parity_lines instance-attribute

align_payload_char_conf(line, char_conf, spec=BASE16G, *, line_parity_chars=0)

Slice a raw line's per-character OCR confidence down to its payload.

char_conf must be the same length as line (one confidence value per printed character, spaces included -- providers give whitespace a confidence of 1.0). This mirrors :func:split_frame_with_parity's POSITIONAL "compact" derivation: label (spec.index_width + 1 chars) then payload+line-parity (everything up to the spec.delimiter) then #check. Both branches of split_frame_with_parity reduce to this same shape once interior OCR whitespace is stripped, so applying it uniformly here (rather than re-running the token-based branch) recovers the same payload boundary for any line that actually has the frame shape. Keep this in sync with :func:split_frame_with_parity if that function's positional math changes.

Returns None (never guesses) if char_conf is the wrong length, the line does not have the frame shape, or the recovered payload region doesn't have a sane width -- callers must treat that as "no confidence available" and fall back to whole-line erasure marking.

Source code in src/glyphive/codec/engine.py 
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
def align_payload_char_conf(
    line: str,
    char_conf: _ty.Sequence[_ty.Optional[float]],
    spec: "_RadixSpec" = BASE16G,
    *,
    line_parity_chars: int = 0,
) -> _ty.Optional[_ty.List[_ty.Optional[float]]]:
    """Slice a raw line's per-character OCR confidence down to its payload.

    ``char_conf`` must be the same length as ``line`` (one confidence value
    per printed character, spaces included -- providers give whitespace a
    confidence of ``1.0``). This mirrors :func:`split_frame_with_parity`'s
    POSITIONAL "compact" derivation: label (``spec.index_width + 1`` chars)
    then payload+line-parity (everything up to the ``spec.delimiter``) then
    ``#check``. Both branches of ``split_frame_with_parity`` reduce to this
    same shape once interior OCR whitespace is stripped, so applying it
    uniformly here (rather than re-running the token-based branch) recovers
    the same payload boundary for any line that actually has the frame
    shape. Keep this in sync with :func:`split_frame_with_parity` if that
    function's positional math changes.

    Returns ``None`` (never guesses) if ``char_conf`` is the wrong length,
    the line does not have the frame shape, or the recovered payload region
    doesn't have a sane width -- callers must treat that as "no confidence
    available" and fall back to whole-line erasure marking.
    """
    if len(char_conf) != len(line):
        return None
    compact_chars: _ty.List[str] = []
    compact_conf: _ty.List[_ty.Optional[float]] = []
    for ch, cf in zip(line, char_conf):
        if not ch.isspace():
            compact_chars.append(ch)
            compact_conf.append(cf)
    compact = "".join(compact_chars)
    label_width = spec.index_width + 1
    if len(compact) < label_width + 1 + spec.check_width:
        return None
    delim = spec.delimiter
    remainder = compact[label_width:]
    if remainder.count(delim) != 1:
        return None
    marker = remainder.index(delim)
    check_end = marker + 1 + spec.check_width
    if marker == 0 or check_end > len(remainder):
        return None
    middle_conf = compact_conf[label_width:label_width + marker]
    if line_parity_chars:
        if len(middle_conf) < line_parity_chars:
            return None
        return middle_conf[:-line_parity_chars]
    return middle_conf

crc16_ccitt(data)

CRC-16/CCITT-FALSE: poly 0x1021, init 0xFFFF, no reflection, no xorout.

Public: this is the single CRC-16 implementation for the whole project. Identical to binascii.crc_hqx(data, 0xFFFF) (verified byte-for-byte); the machine-frame framing in :mod:glyphive.layout uses this rather than a second copy so the per-line and machine-frame checks can never diverge.

Source code in src/glyphive/codec/engine.py 
510
511
512
513
514
515
516
517
518
519
520
521
522
def crc16_ccitt(data: bytes) -> int:
    """CRC-16/CCITT-FALSE: poly 0x1021, init 0xFFFF, no reflection, no xorout.

    Public: this is the single CRC-16 implementation for the whole project.
    Identical to ``binascii.crc_hqx(data, 0xFFFF)`` (verified byte-for-byte); the
    machine-frame framing in :mod:`glyphive.layout` uses this rather than a
    second copy so the per-line and machine-frame checks can never diverge.
    """
    crc = 0xFFFF
    table = _CRC16_TABLE
    for byte in data:
        crc = ((crc << 8) & 0xFFFF) ^ table[((crc >> 8) ^ byte) & 0xFF]
    return crc

decode_index(token)

base16g-bound :func:_decode_index. Public API.

Source code in src/glyphive/codec/engine.py 
660
661
662
def decode_index(token: str) -> _ty.Optional[int]:
    """base16g-bound :func:`_decode_index`. Public API."""
    return _decode_index(token, BASE16G)

describe_line_stream(lines, spec=BASE16G)

Report the realized RS shape of an encoded line stream, read-only.

Mirrors :meth:RadixCodec.decode_spool's modal-width bookkeeping (widest payload among non-last lines sets bytes_per_line) to compute the data and parity byte totals, then derives nsym/nblocks from :func:_candidate_nsym/:func:_num_blocks. It never corrects, decodes, or writes anything -- it exists so callers (e.g. glyphive inspect) can report a document's per-line redundancy without a full decode. If the stream cannot be interpreted (no data lines, or an ambiguous nsym), the RS fields are None rather than a guess.

spec must be the radix spec of the codec that produced the stream (Codec.get(name)._spec); lines framed with a different alphabet or delimiter simply fail to parse and are not counted.

Source code in src/glyphive/codec/engine.py 
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
def describe_line_stream(
    lines: _ty.Iterable[str], spec: "_RadixSpec" = BASE16G
) -> StreamShape:
    """Report the realized RS shape of an encoded line stream, read-only.

    Mirrors :meth:`RadixCodec.decode_spool`'s modal-width bookkeeping (widest
    payload among non-last lines sets ``bytes_per_line``) to compute the data
    and parity byte totals, then derives ``nsym``/``nblocks`` from
    :func:`_candidate_nsym`/:func:`_num_blocks`. It never corrects, decodes, or
    writes anything -- it exists so callers (e.g. ``glyphive inspect``) can
    report a document's per-line redundancy without a full decode. If the
    stream cannot be interpreted (no data lines, or an ambiguous nsym), the
    RS fields are ``None`` rather than a guess.

    ``spec`` must be the radix spec of the codec that produced the stream
    (``Codec.get(name)._spec``); lines framed with a different alphabet or
    delimiter simply fail to parse and are not counted.
    """
    materialized = list(lines)
    line_parity_chars = _detect_line_parity_chars(materialized, spec)
    nsym_line = _nsym_line_for_chars(line_parity_chars, spec)
    data: _ty.Dict[int, "_ParsedLine"] = {}
    parity: _ty.Dict[int, "_ParsedLine"] = {}
    for raw in materialized:
        parsed = _parse_line(raw, spec, line_parity_chars=line_parity_chars)
        if parsed is None:
            continue
        (data if parsed.kind == "L" else parity)[parsed.idx] = parsed

    def _totals(index: _ty.Dict[int, "_ParsedLine"]) -> int:
        if not index:
            return 0
        max_idx = max(index)
        # Modal payload width among non-last lines sets the per-line byte span,
        # exactly as decode does; the last line may legitimately be shorter.
        widths = _collections.Counter(
            len(index[i].payload) for i in index if i != max_idx
        )
        modal = widths.most_common(1)[0][0] if widths else (
            len(index[max_idx].payload)
        )
        bytes_per_line = _bytes_per_line(modal, spec)
        if bytes_per_line < 1:
            return 0
        total = 0
        for i in range(max_idx + 1):
            entry = index.get(i)
            if entry is None:
                total += bytes_per_line
                continue
            total += _payload_byte_len(
                entry.payload, bytes_per_line, i == max_idx, spec
            )
        return total

    data_bytes = _totals(data)
    parity_bytes = _totals(parity)
    nsym: _ty.Optional[int] = None
    nblocks: _ty.Optional[int] = None
    if data:
        candidates = _candidate_nsym(data_bytes, parity_bytes)
        if len(candidates) == 1:
            nsym = candidates[0]
            nblocks = _num_blocks(data_bytes, nsym)
    return StreamShape(
        data_lines=len(data),
        parity_lines=len(parity),
        nsym=nsym,
        nblocks=nblocks,
        data_bytes=data_bytes,
        parity_bytes=parity_bytes,
        nsym_line=nsym_line,
    )

encode_index(idx)

base16g-bound :func:_encode_index. Public API.

0 -> MYCVH, 1 -> MYCVK, 1048575 -> PCYHV.

Source code in src/glyphive/codec/engine.py 
652
653
654
655
656
657
def encode_index(idx: int) -> str:
    """base16g-bound :func:`_encode_index`. Public API.

    ``0`` -> ``MYCVH``, ``1`` -> ``MYCVK``, ``1048575`` -> ``PCYHV``.
    """
    return _encode_index(idx, BASE16G)

encoded_line_count(data_len, *, line_width=60, parity_ratio=0.12, nsym_line=2, spec=BASE16G)

Return the exact number of lines without reading or encoding payload data.

nsym_line (default 2, matching :meth:RadixCodec.encode's default) does not change the returned COUNT (see :func:_encoding_shape); it is accepted and validated here so callers pass the same value they encode with, and so an invalid nsym_line is caught at the same call site a caller would naturally check page planning from.

spec selects the codec: the denser radix codecs pack more bytes per line, so page planning for --codec base32g etc. must pass codec._spec to get the right count instead of assuming base16g's 4 bits/char.

Source code in src/glyphive/codec/engine.py 
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
def encoded_line_count(
    data_len: int,
    *,
    line_width: int = 60,
    parity_ratio: float = 0.12,
    nsym_line: int = 2,
    spec: "_RadixSpec" = BASE16G,
) -> int:
    """Return the exact number of lines without reading or encoding payload data.

    ``nsym_line`` (default 2, matching :meth:`RadixCodec.encode`'s default)
    does not change the returned COUNT (see :func:`_encoding_shape`); it is
    accepted and validated here so callers pass the same value they encode
    with, and so an invalid ``nsym_line`` is caught at the same call site a
    caller would naturally check page planning from.

    ``spec`` selects the codec: the denser radix codecs pack more bytes per line,
    so page planning for ``--codec base32g`` etc. must pass ``codec._spec`` to
    get the right count instead of assuming base16g's 4 bits/char.
    """
    if data_len < 0 or data_len > 0xFFFFFFFF:
        raise ValueError("data length must fit the codec's unsigned 32-bit header")
    return sum(_encoding_shape(data_len, line_width, parity_ratio, spec, nsym_line)[-2:])

line_rs_correct(line, spec=BASE16G, *, nsym_line)

In-line Reed-Solomon correction tier (runs BEFORE :func:repair_line).

For a CRC-failed line carrying a non-empty line-parity field, treat the printed idx_token + payload bytes plus the nsym_line printed parity bytes as one small RS codeword and blind-correct it (no known erasure positions -- the line-parity tier runs before anything is committed as an erasure). On a successful decode, re-render the corrected token/payload and RECOMPUTE the check field: the fix is trusted only if that recomputed check reproduces the ORIGINALLY PRINTED one exactly (an RS miscorrection essentially never also reproduces a 16-bit CRC by chance). Otherwise -- ambiguous RS decode, or a "corrected" line whose check still disagrees -- returns None and the caller falls through to :func:repair_line and finally to whole-line erasure, unchanged.

The payload's own printed width determines how many raw bytes it decodes to (len(payload) * spec.bits // 8, i.e. the line's OWN declared shape -- this tier runs before the modal-width vote, so it cannot rely on stream-wide bookkeeping). Returns None immediately if nsym_line is 0 (no line-parity field to correct with) or the line does not structurally carry one.

Source code in src/glyphive/codec/engine.py 
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
def line_rs_correct(
    line: str, spec: "_RadixSpec" = BASE16G, *, nsym_line: int
) -> _ty.Optional[str]:
    """In-line Reed-Solomon correction tier (runs BEFORE :func:`repair_line`).

    For a CRC-failed line carrying a non-empty line-parity field, treat the
    printed ``idx_token + payload`` bytes plus the ``nsym_line`` printed
    parity bytes as one small RS codeword and blind-correct it (no known
    erasure positions -- the line-parity tier runs before anything is
    committed as an erasure). On a successful decode, re-render the corrected
    token/payload and RECOMPUTE the check field: the fix is trusted only if
    that recomputed check reproduces the ORIGINALLY PRINTED one exactly (an RS
    miscorrection essentially never also reproduces a 16-bit CRC by chance).
    Otherwise -- ambiguous RS decode, or a "corrected" line whose check still
    disagrees -- returns ``None`` and the caller falls through to
    :func:`repair_line` and finally to whole-line erasure, unchanged.

    The payload's own printed width determines how many raw bytes it decodes
    to (``len(payload) * spec.bits // 8``, i.e. the line's OWN declared
    shape -- this tier runs before the modal-width vote, so it cannot rely on
    stream-wide bookkeeping). Returns ``None`` immediately if ``nsym_line`` is
    0 (no line-parity field to correct with) or the line does not
    structurally carry one.
    """
    if nsym_line <= 0:
        return None
    split = split_frame_with_parity(
        line.strip(), spec=spec, line_parity_chars=_line_parity_chars(nsym_line, spec)
    )
    if split is None:
        return None
    label, payload, line_parity, check = split
    kind = label[:1]
    if kind not in ("L", "P"):
        return None
    token = label[1:]
    if len(token) != spec.index_width or not check.startswith(spec.delimiter):
        return None
    want = check[len(spec.delimiter):]
    if len(want) != spec.check_width:
        return None
    payload_bytes = _bytes_per_line(len(payload), spec) if payload else 0
    try:
        chunk = radix_decode(payload, payload_bytes, spec)
        line_parity_bytes = radix_decode(line_parity, nsym_line, spec)
    except ValueError:
        return None
    codeword = bytearray(_line_rs_codeword(token, chunk) + line_parity_bytes)
    codec = _reedsolo.RSCodec(nsym_line)
    try:
        decoded = codec.decode(codeword)[0]
    except (_reedsolo.ReedSolomonError, ValueError):
        return None
    corrected_token_bytes = bytes(decoded[:spec.index_width])
    corrected_chunk = bytes(decoded[spec.index_width:])
    try:
        corrected_token = corrected_token_bytes.decode("ascii")
    except UnicodeDecodeError:
        return None
    fold = spec.case_fold
    token_for_crc = corrected_token.upper() if fold else corrected_token
    corrected_payload = radix_encode(corrected_chunk, spec)
    recomputed = _check_chars(kind, token_for_crc, corrected_payload, spec)
    want_cmp = want.upper() if fold else want
    if recomputed != want_cmp:
        return None  # the RS "fix" doesn't reproduce the printed CRC -- reject
    return f"{kind}{corrected_token} {corrected_payload} {line_parity} {spec.delimiter}{want}"

machine_check(kind, idx_token, payload)

Return the safe-alphabet CRC-16 check field for a machine frame.

Covers kind (H/T/Q) as well as the index token and payload -- the same kind-covered CRC as the per-line check -- so an OCR misread that flips one machine-frame kind letter into another fails the check instead of producing a CRC-valid phantom frame of the wrong kind at the same index.

Source code in src/glyphive/codec/engine.py 
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
def machine_check(kind: str, idx_token: str, payload: str) -> str:
    """Return the safe-alphabet CRC-16 check field for a machine frame.

    Covers ``kind`` (``H``/``T``/``Q``) as well as the index token and payload --
    the same kind-covered CRC as the per-line check -- so an OCR misread that
    flips one machine-frame kind letter into another fails the check instead of
    producing a CRC-valid phantom frame of the wrong kind at the same index.
    """
    crc = crc16_ccitt(
        kind.upper().encode() + idx_token.upper().encode() + payload.upper().encode()
    )
    return nibble_encode(crc.to_bytes(2, "big"))

machine_frame(kind, idx, payload, *, nsym_line=0)

Build one machine frame <kind><token> <payload> [<lparity> ]#<check>.

payload must be MACHINE_SPEC safe-alphabet characters (the caller has already :func:nibble_encode-d its bytes). The CRC covers the kind char and deliberately does NOT cover the line-parity field: a corrupted parity field must not fail an otherwise-good line, since the parity is itself validated by the correct-then-reverify loop in :func:repair_machine_frame.

Source code in src/glyphive/codec/engine.py 
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
def machine_frame(kind: str, idx: int, payload: str, *, nsym_line: int = 0) -> str:
    """Build one machine frame ``<kind><token> <payload> [<lparity> ]#<check>``.

    ``payload`` must be MACHINE_SPEC safe-alphabet characters (the caller has
    already :func:`nibble_encode`-d its bytes). The CRC covers the kind char and
    deliberately does NOT cover the line-parity field: a corrupted parity field
    must not fail an otherwise-good line, since the parity is itself validated by
    the correct-then-reverify loop in :func:`repair_machine_frame`.
    """
    if len(kind) != 1 or not kind.isalpha():
        raise ValueError(f"invalid machine frame kind {kind!r}")
    if any(char not in MACHINE_SPEC.alphabet for char in payload):
        raise ValueError("machine metadata payload contains an unsafe character")
    token = encode_index(idx)
    check = machine_check(kind, token, payload)
    lparity = machine_line_parity(token, payload, nsym_line)
    if lparity:
        return f"{kind}{token} {payload} {lparity} #{check}"
    return f"{kind}{token} {payload} #{check}"

machine_line_parity(idx_token, payload, nsym_line)

Per-line Reed-Solomon parity chars for a machine frame, or "".

Mirrors the payload frames' per-line RS: nsym_line parity bytes over the printed index token plus the payload's decoded bytes, rendered back into the machine (base16g) alphabet. Machine frames previously had NO in-line correction at all -- only a CRC (detect-only), envelope RS, and duplicate copies -- which made them a lower ceiling than the L/P payload frames they introduce: at small font sizes decode failed on the H frames while the data lines themselves were still recoverable.

Source code in src/glyphive/codec/engine.py 
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
def machine_line_parity(idx_token: str, payload: str, nsym_line: int) -> str:
    """Per-line Reed-Solomon parity chars for a machine frame, or ``""``.

    Mirrors the payload frames' per-line RS: ``nsym_line`` parity bytes over the
    printed index token plus the payload's decoded bytes, rendered back into the
    machine (base16g) alphabet. Machine frames previously had NO in-line
    correction at all -- only a CRC (detect-only), envelope RS, and duplicate
    copies -- which made them a *lower* ceiling than the L/P payload frames they
    introduce: at small font sizes decode failed on the H frames while the data
    lines themselves were still recoverable.
    """
    if not nsym_line:
        return ""
    message = idx_token.encode() + nibble_decode(payload, len(payload) // 2)
    parity = bytes(_reedsolo.RSCodec(nsym_line).encode(message)[len(message):])
    return nibble_encode(parity)

nibble_decode(text, byte_len)

base16g-bound :func:radix_decode (4 bits/char). Public API.

Source code in src/glyphive/codec/engine.py 
488
489
490
def nibble_decode(text: str, byte_len: int) -> bytes:
    """base16g-bound :func:`radix_decode` (4 bits/char). Public API."""
    return radix_decode(text, byte_len, BASE16G)

nibble_encode(data)

base16g-bound :func:radix_encode (4 bits/char). Public API.

Source code in src/glyphive/codec/engine.py 
483
484
485
def nibble_encode(data: bytes) -> str:
    """base16g-bound :func:`radix_encode` (4 bits/char). Public API."""
    return radix_encode(data, BASE16G)

parse_machine_frame(line, kind, *, allow_trailing=False, nsym_line=0)

Parse one machine frame of kind without trusting surrounding text.

allow_trailing permits display-only text after the protected frame (a footer's PAGE n/total hint). Interior whitespace in the safe-alphabet payload is stripped, exactly as for payload frames; the kind-covered CRC is the acceptance oracle.

nsym_line is the per-line Reed-Solomon parity byte count the document was written with. It must be supplied by the caller (from the protected header, or the layout constant used to emit the frames) because the parity field carries no delimiter -- exactly the ambiguity that made the payload frames' token-counting heuristic fail on space-stripped OCR output. When it is non-zero and the CRC fails, one in-line RS correction is attempted and the printed CRC is RE-VERIFIED before the result is trusted.

Source code in src/glyphive/codec/engine.py 
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
def parse_machine_frame(
    line: str,
    kind: str,
    *,
    allow_trailing: bool = False,
    nsym_line: int = 0,
) -> _ty.Optional[ParsedMachineFrame]:
    """Parse one machine frame of ``kind`` without trusting surrounding text.

    ``allow_trailing`` permits display-only text after the protected frame (a
    footer's ``PAGE n/total`` hint). Interior whitespace in the safe-alphabet
    payload is stripped, exactly as for payload frames; the kind-covered CRC is
    the acceptance oracle.

    ``nsym_line`` is the per-line Reed-Solomon parity byte count the document
    was written with. It must be supplied by the caller (from the protected
    header, or the layout constant used to emit the frames) because the parity
    field carries no delimiter -- exactly the ambiguity that made the payload
    frames' token-counting heuristic fail on space-stripped OCR output. When it
    is non-zero and the CRC fails, one in-line RS correction is attempted and
    the printed CRC is RE-VERIFIED before the result is trusted.
    """
    lparity_chars = -(-nsym_line * 8 // MACHINE_SPEC.bits) if nsym_line else 0
    split = split_frame_with_parity(
        line, allow_trailing=allow_trailing, line_parity_chars=lparity_chars
    )
    if split is None:
        return None
    label, payload, lparity, check_field = split
    if label[:1].upper() != kind:
        return None
    if len(label) != INDEX_WIDTH + 1:
        return ParsedMachineFrame(idx=None, payload="", ok=False)
    idx_token = label[1:]
    idx = decode_index(idx_token)
    if idx is None:
        return ParsedMachineFrame(idx=None, payload="", ok=False)
    payload = payload.upper()
    check = check_field[1:].upper()
    if machine_check(kind, idx_token, payload) == check:
        return ParsedMachineFrame(idx=idx, payload=payload, ok=True)
    if nsym_line and lparity:
        repaired = _repair_machine_payload(idx_token, payload, lparity, nsym_line)
        if repaired is not None:
            fixed_token, fixed_payload = repaired
            # Re-verify the PRINTED CRC: an RS miscorrection must never be
            # accepted just because RS reported success.
            if machine_check(kind, fixed_token, fixed_payload) == check:
                fixed_idx = decode_index(fixed_token)
                if fixed_idx is not None:
                    return ParsedMachineFrame(
                        idx=fixed_idx, payload=fixed_payload, ok=True
                    )
    return ParsedMachineFrame(idx=idx, payload=payload, ok=False)

radix_decode(text, byte_len, spec=BASE16G)

Decode an alphabet string back to exactly byte_len bytes (bit/group).

Case-insensitive. No confusable aliases are applied (see the comment above _DECODE_MAP): any character outside the alphabet raises ValueError rather than being guessed at. Trailing pad bits beyond byte_len bytes are discarded.

Source code in src/glyphive/codec/engine.py 
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
def radix_decode(text: str, byte_len: int, spec: "_RadixSpec" = BASE16G) -> bytes:
    """Decode an alphabet string back to exactly ``byte_len`` bytes (bit/group).

    Case-insensitive. No confusable aliases are applied (see the comment above
    ``_DECODE_MAP``): any character outside the alphabet raises ``ValueError``
    rather than being guessed at. Trailing pad bits beyond ``byte_len`` bytes
    are discarded.
    """
    if spec.packing == "group":
        return _group_decode(text, byte_len, spec)
    if byte_len == 0:
        return b""
    bits = spec.bits
    decode_map = spec.decode_map
    acc = 0
    nbits = 0
    out = bytearray()
    for ch in text:
        try:
            val = decode_map[ch]
        except KeyError:
            raise ValueError(f"invalid alphabet character {ch!r}") from None
        acc = (acc << bits) | val
        nbits += bits
        if nbits >= 8:
            nbits -= 8
            out.append((acc >> nbits) & 0xFF)
        if len(out) == byte_len:
            break
    if len(out) < byte_len:
        raise ValueError(
            f"payload too short: got {len(out)} bytes, need {byte_len}"
        )
    return bytes(out)

radix_encode(data, spec=BASE16G)

Encode raw bytes to an alphabet string per spec (bit- or group-packing).

Bit-packing: spec.bits bits per char, MSB-first, final group zero-padded. Group-packing: Ascii85-style (see :func:_group_encode). Not self-delimiting; the caller tracks the original byte length (:func:radix_decode).

Source code in src/glyphive/codec/engine.py 
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
def radix_encode(data: bytes, spec: "_RadixSpec" = BASE16G) -> str:
    """Encode raw bytes to an alphabet string per ``spec`` (bit- or group-packing).

    Bit-packing: ``spec.bits`` bits per char, MSB-first, final group zero-padded.
    Group-packing: Ascii85-style (see :func:`_group_encode`). Not self-delimiting;
    the caller tracks the original byte length (:func:`radix_decode`).
    """
    if spec.packing == "group":
        return _group_encode(data, spec)
    if not data:
        return ""
    bits = spec.bits
    mask = spec.mask
    alphabet = spec.alphabet
    out: _ty.List[str] = []
    acc = 0
    nbits = 0
    for byte in data:
        acc = (acc << 8) | byte
        nbits += 8
        while nbits >= bits:
            nbits -= bits
            out.append(alphabet[(acc >> nbits) & mask])
    if nbits:  # flush remaining bits, left-aligned (MSB-first) into one char
        out.append(alphabet[(acc << (bits - nbits)) & mask])
    return "".join(out)

repair_line(line, spec=BASE16G, *, line_parity_chars=0)

Attempt to repair a single misread character in a CRC-failed line.

The per-line CRC is the oracle: we try every single-character substitution of the printed kind + index_token + payload body (kind is included because the CRC now covers it too -- a kind flip is just another single-character error) and the case where the printed check field itself is the corrupted character, recompute the CRC for each candidate, and accept a repair only if exactly one candidate reproduces the printed check (or, for a corrupted check field, the recomputed check is within one character of the printed one). Ambiguity or zero hits leaves the line untouched (the caller keeps it as an erasure).

This does not violate the decode-discipline doctrine ("never mutate a character merely because more compressed bytes become readable"): acceptance is CRC-verified, not decompressibility-driven. A false unique candidate requires a >=2-error line, a spurious CRC match (~2**-16), and no colliding true candidate; such a line enters as a blind error which document RS still corrects within budget, and the whole-document SHA-256 gate backstops it.

line_parity_chars (0 by default): the line-parity field (if present) is NOT covered by the CRC and is not searched for errors here; it is carried through unchanged in the repaired output.

Returns the repaired printed line (which re-parses as CRC-ok), or None. Works for both bit-packed and group-packed specs -- the CRC covers the printed characters, so the procedure is alphabet-agnostic.

Source code in src/glyphive/codec/engine.py 
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
def repair_line(
    line: str, spec: "_RadixSpec" = BASE16G, *, line_parity_chars: int = 0
) -> _ty.Optional[str]:
    """Attempt to repair a single misread character in a CRC-failed line.

    The per-line CRC is the *oracle*: we try every single-character
    substitution of the printed ``kind + index_token + payload`` body (kind is
    included because the CRC now covers it too -- a kind flip is just another
    single-character error) and the case where the printed check field itself
    is the corrupted character, recompute the CRC for each candidate, and
    accept a repair **only if exactly one candidate reproduces the printed
    check** (or, for a corrupted check field, the recomputed check is within
    one character of the printed one). Ambiguity or zero hits leaves the line
    untouched (the caller keeps it as an erasure).

    This does not violate the decode-discipline doctrine ("never mutate a
    character merely because more compressed bytes become readable"): acceptance
    is CRC-verified, not decompressibility-driven. A false unique candidate
    requires a >=2-error line, a spurious CRC match (~2**-16), and no colliding
    true candidate; such a line enters as a blind error which document RS still
    corrects within budget, and the whole-document SHA-256 gate backstops it.

    ``line_parity_chars`` (0 by default): the line-parity field (if present) is
    NOT covered by the CRC and is not searched for errors here; it is carried
    through unchanged in the repaired output.

    Returns the repaired printed line (which re-parses as CRC-ok), or ``None``.
    Works for both bit-packed and group-packed specs -- the CRC covers the
    printed characters, so the procedure is alphabet-agnostic.
    """
    split = split_frame_with_parity(
        line.strip(), spec=spec, line_parity_chars=line_parity_chars
    )
    if split is None:
        return None
    label, payload, line_parity, check = split
    kind = label[:1]
    if kind not in ("L", "P"):
        return None
    token = label[1:]
    if len(token) != spec.index_width or not check.startswith(spec.delimiter):
        return None
    want = check[len(spec.delimiter):]
    if len(want) != spec.check_width:
        return None
    fold = spec.case_fold
    want_cmp = want.upper() if fold else want
    body = kind + token + payload
    alphabet = spec.alphabet
    parity_field = f" {line_parity}" if line_parity_chars else ""
    hits: _ty.List[str] = []
    for pos in range(len(body)):
        original = body[pos]
        # The kind character (position 0) is restricted to the two valid
        # kinds, not the whole payload alphabet -- a "kind" character is never
        # a member of the payload alphabet in general (e.g. base32g), and even
        # when it happens to be, only L/P are ever legal there.
        candidates_here = ("L", "P") if pos == 0 else alphabet
        for sub in candidates_here:
            if sub == original:
                continue
            cand = body[:pos] + sub + body[pos + 1:]
            cand_kind = cand[0]
            cand_token = cand[1:1 + len(token)]
            cand_payload = cand[1 + len(token):]
            token_for_crc = cand_token.upper() if fold else cand_token
            if _check_chars(cand_kind, token_for_crc, cand_payload, spec) == want_cmp:
                hits.append(cand)
                if len(hits) > 1:
                    return None  # ambiguous -- refuse to guess
    # Case: the check FIELD was the corrupted character (body is already right).
    token_for_crc = token.upper() if fold else token
    expected = _check_chars(kind, token_for_crc, payload, spec)
    if not hits and expected != want_cmp and _hamming1(expected, want_cmp):
        # Payload/token/kind are intact; only the printed check drifted.
        return f"{kind}{token} {payload}{parity_field} {spec.delimiter}{expected}"
    if len(hits) == 1:
        cand = hits[0]
        cand_kind = cand[0]
        cand_token = cand[1:1 + len(token)]
        cand_payload = cand[1 + len(token):]
        return f"{cand_kind}{cand_token} {cand_payload}{parity_field} {spec.delimiter}{want}"
    return None

rs_protect(data, nsym)

Reed-Solomon-protect data: return (data, parity, nblocks).

Public wrapper over the interleaved symbol-major block encoder, so callers outside the codec (the machine-header framing in :mod:glyphive.layout) depend on a stable contract rather than a private helper.

Source code in src/glyphive/codec/engine.py 
1534
1535
1536
1537
1538
1539
1540
1541
def rs_protect(data: bytes, nsym: int) -> _ty.Tuple[bytes, bytes, int]:
    """Reed-Solomon-protect ``data``: return ``(data, parity, nblocks)``.

    Public wrapper over the interleaved symbol-major block encoder, so callers
    outside the codec (the machine-header framing in :mod:`glyphive.layout`)
    depend on a stable contract rather than a private helper.
    """
    return _rs_encode(data, nsym)

rs_recover(data, data_erasures, parity, parity_erasures, nsym, nblocks)

Erasure-recover data from parity; return the corrected bytes.

Public wrapper over the interleaved block decoder (see :func:rs_protect). *_erasures are byte positions whose source frame failed its CRC.

Source code in src/glyphive/codec/engine.py 
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
def rs_recover(
    data: bytearray,
    data_erasures: _ty.List[int],
    parity: bytearray,
    parity_erasures: _ty.List[int],
    nsym: int,
    nblocks: int,
) -> bytes:
    """Erasure-recover ``data`` from ``parity``; return the corrected bytes.

    Public wrapper over the interleaved block decoder (see :func:`rs_protect`).
    ``*_erasures`` are byte positions whose source frame failed its CRC.
    """
    return _rs_decode(data, data_erasures, parity, parity_erasures, nsym, nblocks)

split_frame(line, *, allow_trailing=False, spec=BASE16G, line_parity_chars=0)

Structurally split a printed line into (label, payload, check).

OCR (observed: Tesseract) sometimes inserts a spurious space inside the payload (e.g. ...FYWZQH4 6F1IWO0C...), which would turn the intended 3 whitespace tokens into 4+ and cause a naive line.split() shape test to discard an otherwise-perfect line. The frame's actual shape is not "exactly 3 tokens" -- it is "a label, then a payload [then a line-parity run], then a #check field": the label is always the first token and #check is always the last token, so everything in between is payload (+ optionally line-parity).

This is deterministic normalization, not guessing: the payload alphabet contains no whitespace, so any interior space is provably OCR noise, and joining the middle tokens with no separator recovers exactly the printed payload (+ line-parity) characters. The per-line CRC (computed downstream over this recovered payload) is what actually decides correctness -- this function only ensures a noisy-but-readable line reaches that check instead of being silently dropped before it gets the chance.

line_parity_chars (0 by default -- no line-parity field): when > 0, the trailing line_parity_chars characters of the joined middle run are split off as a fourth field; the returned payload is the remainder (positional split from the end, since interior OCR spaces mean the payload/line-parity boundary cannot be found by whitespace alone -- see the module docstring's frame grammar). The returned tuple's second element stays payload only for callers that pass line_parity_chars=0; when it is > 0, use :func:split_frame_with_parity for the 4-tuple form.

Returns None if the line has fewer than 3 tokens or the last token does not start with # (i.e. it does not have the frame shape at all -- e.g. the PAGE 1/1 sha256=... footer, which is 3 tokens but whose last token is not a check field).

Source code in src/glyphive/codec/engine.py 
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
def split_frame(
    line: str,
    *,
    allow_trailing: bool = False,
    spec: "_RadixSpec" = BASE16G,
    line_parity_chars: int = 0,
) -> _ty.Optional[_ty.Tuple[str, str, str]]:
    """Structurally split a printed line into ``(label, payload, check)``.

    OCR (observed: Tesseract) sometimes inserts a spurious space *inside* the
    payload (e.g. ``...FYWZQH4 6F1IWO0C...``), which would turn the intended 3
    whitespace tokens into 4+ and cause a naive ``line.split()`` shape test to
    discard an otherwise-perfect line. The frame's actual shape is not "exactly
    3 tokens" -- it is "a label, then a payload [then a line-parity run], then
    a ``#check`` field": the label is always the *first* token and ``#check``
    is always the *last* token, so everything in between is payload (+
    optionally line-parity).

    This is deterministic normalization, not guessing: the payload alphabet
    contains no whitespace, so any interior space is provably OCR noise, and
    joining the middle tokens with no separator recovers exactly the printed
    payload (+ line-parity) characters. The per-line CRC (computed downstream
    over this recovered payload) is what actually decides correctness -- this
    function only ensures a noisy-but-readable line reaches that check instead
    of being silently dropped before it gets the chance.

    ``line_parity_chars`` (0 by default -- no line-parity field): when > 0,
    the trailing ``line_parity_chars`` characters of the joined middle run are
    split off as a fourth field; the returned ``payload`` is the remainder
    (positional split from the end, since interior OCR spaces mean the
    payload/line-parity boundary cannot be found by whitespace alone -- see
    the module docstring's frame grammar). The returned tuple's second element
    stays ``payload`` only for callers that pass ``line_parity_chars=0``; when
    it is > 0, use :func:`split_frame_with_parity` for the 4-tuple form.

    Returns ``None`` if the line has fewer than 3 tokens or the last token
    does not start with ``#`` (i.e. it does not have the frame shape at all --
    e.g. the ``PAGE 1/1 sha256=...`` footer, which is 3 tokens but whose last
    token is not a check field).
    """
    result = split_frame_with_parity(
        line,
        allow_trailing=allow_trailing,
        spec=spec,
        line_parity_chars=line_parity_chars,
    )
    if result is None:
        return None
    label, payload, line_parity, check = result
    if line_parity_chars:
        return label, payload + line_parity, check
    return label, payload, check

split_frame_with_parity(line, *, allow_trailing=False, spec=BASE16G, line_parity_chars=0)

Like :func:split_frame but returns (label, payload, line_parity, check).

line_parity is "" when line_parity_chars is 0. When positive, the middle glyph run is split positionally from the end: the LAST line_parity_chars characters are the line-parity field, everything before that is payload. This is exact (not a guess) because both fields are drawn from the same alphabet at fixed, known widths -- there is no delimiter between them, by design (one glyph run, split by position).

Source code in src/glyphive/codec/engine.py 
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
def split_frame_with_parity(
    line: str,
    *,
    allow_trailing: bool = False,
    spec: "_RadixSpec" = BASE16G,
    line_parity_chars: int = 0,
) -> _ty.Optional[_ty.Tuple[str, str, str, str]]:
    """Like :func:`split_frame` but returns ``(label, payload, line_parity, check)``.

    ``line_parity`` is ``""`` when ``line_parity_chars`` is 0. When positive,
    the middle glyph run is split positionally from the end: the LAST
    ``line_parity_chars`` characters are the line-parity field, everything
    before that is payload. This is exact (not a guess) because both fields
    are drawn from the same alphabet at fixed, known widths -- there is no
    delimiter between them, by design (one glyph run, split by position).
    """
    delim = spec.delimiter
    parts = line.split()
    middle = None
    if len(parts) >= 3:
        check_positions = [
            index for index, part in enumerate(parts[1:], 1)
            if part.startswith(delim)
        ]
        if len(check_positions) == 1:
            position = check_positions[0]
            if allow_trailing or position == len(parts) - 1:
                middle = "".join(parts[1:position])
                label, check = parts[0], parts[position]

    if middle is None:
        # A constrained Tesseract whitelist can remove both printed separator
        # spaces while preserving every protected glyph.  The label and check
        # have fixed widths, and the delimiter is outside the payload
        # alphabet, so this compact shape remains unambiguous and still flows
        # through the CRC check.
        stripped = line.strip()
        label_width = spec.index_width + 1
        if len(stripped) < label_width + 1 + spec.check_width:
            return None
        label = stripped[:label_width]
        remainder = stripped[label_width:]
        if remainder.count(delim) != 1:
            return None
        marker = remainder.index(delim)
        check_end = marker + 1 + spec.check_width
        if marker == 0 or check_end > len(remainder):
            return None
        trailing = remainder[check_end:]
        if trailing and not allow_trailing:
            return None
        middle = "".join(remainder[:marker].split())
        check = remainder[marker:check_end]

    if line_parity_chars:
        if len(middle) < line_parity_chars:
            return None
        payload = middle[:-line_parity_chars]
        line_parity = middle[-line_parity_chars:]
    else:
        payload = middle
        line_parity = ""
    return label, payload, line_parity, check

glyphive.codec.radix

The radix codec family: base8 / base32g / base64.

Each is the shipped codec engine (codec/engine.py) with a different :class:~glyphive.codec.engine._RadixSpec — a wider alphabet packs more bits per printed character (denser pages), at the cost of stock-OCR reliability.

Density vs. base16g (4 bits/char): - base8 : 3 bits/char (0.75x — sparser, most OCR-robust) - base16g : 4 bits/char (the measured stock-safe default) - base32g : 5 bits/char (1.25x — glyphive's measured 32-glyph set) - base64 : 6 bits/char (1.5x)

The measured stock-OCR-safe ceiling is 16 characters (alphabet/size sweeps 2026-07-18); base32g/base64 are NOT stock-OCR-safe (~14.8% CER stock) but read at 0.0% CER with a per-font fine-tuned model (measured 2026-07-18). They are for the trained-model restore path (opt-in per-font OCR model packages). Codecs are never gated — creation only maps bytes to characters and never needs a model; choosing a denser codec is the user's informed decision. base16g stays the recommendation.

BASE16 = _RadixSpec(name='base16-crc16-rs', alphabet='0123456789ABCDEF', bits=4, check_width=4, index_width=5, index_mask=(7, 13, 2, 11, 4)) module-attribute

BASE32 = _RadixSpec(name='base32-crc16-rs', alphabet='ABCDEFGHIJKLMNOPQRSTUVWXYZ234567', bits=5, check_width=4, index_width=4, index_mask=(7, 26, 13, 21)) module-attribute

BASE32C = _RadixSpec(name='base32c-crc16-rs', alphabet='0123456789ABCDEFGHJKMNPQRSTVWXYZ', bits=5, check_width=4, index_width=4, index_mask=(7, 26, 13, 21)) module-attribute

BASE32G = _RadixSpec(name='base32g-crc16-rs', alphabet='ABCDHKLMPRTVXY34EFGNUW2567?@!&+=', bits=5, check_width=4, index_width=4, index_mask=(7, 26, 13, 21)) module-attribute

BASE64 = _RadixSpec(name='base64-crc16-rs', alphabet='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/', bits=6, check_width=3, index_width=4, index_mask=(11, 46, 23, 58)) module-attribute

BASE64G = _RadixSpec(name='base64g-crc16-rs', alphabet='!"$%&\'(-.0123456789:;>ABCDEGHIJKLMNPRSTUVWXYabcdeghikmnrstyz}#@+', bits=6, check_width=3, index_width=4, index_mask=(11, 46, 23, 58), delimiter=',') module-attribute

BASE85 = _RadixSpec(name='base85-crc16-rs', alphabet='0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~', bits=6, check_width=3, index_width=5, index_mask=(7, 13, 2, 11, 4), packing='group', group_bytes=4, group_chars=5, delimiter=',') module-attribute

BASE8G = _RadixSpec(name='base8g-crc16-rs', alphabet='ABCD34XY', bits=3, check_width=6, index_width=7, index_mask=(1, 6, 3, 5, 2, 7, 4)) module-attribute

BASEMAXG = _RadixSpec(name='basemaxg-crc16-rs', alphabet='!&-012345689;ABCDGIKLMNPRSUVWXY`abdehknrt|~', bits=5, check_width=3, index_width=5, index_mask=(7, 13, 2, 11, 4), packing='group', group_bytes=6, group_chars=9) module-attribute

Z85 = _RadixSpec(name='z85-crc16-rs', alphabet='0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ.-:+=^!/*?&<>()[]{}@%$#', bits=6, check_width=3, index_width=5, index_mask=(7, 13, 2, 11, 4), packing='group', group_bytes=4, group_chars=5, delimiter='\\') module-attribute

__all__ = ['Base16GCodec', 'Base8GCodec', 'Base32GCodec', 'Base64Codec', 'BASE8G', 'BASE32G', 'BASE64', 'Base16Codec', 'Base32Codec', 'Base32CCodec', 'Base85Codec', 'Z85Codec', 'Base64GCodec', 'BaseMaxGCodec', 'BASE16', 'BASE32', 'BASE32C', 'BASE85', 'Z85', 'BASE64G', 'BASEMAXG'] module-attribute

Base16Codec

Bases: RadixCodec

Standard hexadecimal (0-9 A-F), 4 bits/char. Not OCR-tuned (use base16g).

name = 'base16-crc16-rs' class-attribute instance-attribute

Base16GCodec

Bases: RadixCodec

base16g-crc16-rs: the 16-char stock-OCR-safe default codec.

The plain shared engine (:class:~glyphive.codec.engine.RadixCodec) with the :data:~glyphive.codec.engine.BASE16G spec -- 4 bits/char, no trained OCR model needed. Every other codec here is the same engine with a denser or textbook alphabet.

name = 'base16g-crc16-rs' class-attribute instance-attribute

Base32CCodec

Bases: RadixCodec

Crockford base32 (0-9 A-Z minus I L O U), 5 bits/char. Not OCR-tuned.

name = 'base32c-crc16-rs' class-attribute instance-attribute

Base32Codec

Bases: RadixCodec

Standard RFC-4648 base32 (A-Z 2-7), 5 bits/char. Not OCR-tuned (use base32g).

name = 'base32-crc16-rs' class-attribute instance-attribute

Base32GCodec

Bases: RadixCodec

base32g: glyphive's 32-char (5 bits/char) codec — 25% denser than base16g.

Reads at 0.0% CER with a per-font trained model; ~14.8% on stock OCR. Denser than base16g but needs the matching trained model for reliable scan restore.

name = 'base32g-crc16-rs' class-attribute instance-attribute

Base64Codec

Bases: RadixCodec

64-char (6 bits/char) codec — densest; needs a trained model to restore.

name = 'base64-crc16-rs' class-attribute instance-attribute

Base64GCodec

Bases: RadixCodec

base64g: glyphive's curated 64-glyph set (confusion-distinct favoring).

6 bits/char like base64, but the alphabet favors OCR-distinct glyphs. Needs a trained model for reliable restore (as with base32g/base64).

name = 'base64g-crc16-rs' class-attribute instance-attribute

Base85Codec

Bases: RadixCodec

Standard base85 (RFC-1924-ish), group-packed 4->5. Densest; interop only.

name = 'base85-crc16-rs' class-attribute instance-attribute

Base8GCodec

Bases: RadixCodec

Sparse 8-char (3 bits/char) codec — most OCR-robust, least dense.

name = 'base8g-crc16-rs' class-attribute instance-attribute

BaseMaxGCodec

Bases: RadixCodec

base-maxg: glyphive's 43-glyph max-distinct set, group-packed 6->9.

The largest mutually-distinct alphabet measured on stock OCR (~43/font). Needs a trained model for reliable restore, like base32g; denser than 32.

name = 'basemaxg-crc16-rs' class-attribute instance-attribute

Z85Codec

Bases: RadixCodec

ZeroMQ Z85, group-packed 4->5. Interop only (85 glyphs, not OCR-safe).

name = 'z85-crc16-rs' class-attribute instance-attribute