Archive API

glyphive.archive defines the binary-safe archive record stream. Compression is a separate stage.

glyphive.archive

glyphive — file tree ⇄ flat, binary-safe byte stream, plus the ignore filter and the (separate) compression stage.

This module turns a directory tree into one deterministic bytes blob that :mod:glyphive.codec encodes for print and :mod:glyphive.layout lays out on pages; it also provides the inverse parser, iter_records (consumed by :mod:glyphive.restore.unarchive). Serialization is flat and relpath-keyed and restore recreates directories. Every field is length-prefixed binary framing, so the stream survives arbitrary bytes — including a file whose own content happens to contain the magic or any framing bytes — rather than relying on a text delimiter that a payload could collide with.

All filesystem access goes through pathlib_next.Path (never os / pathlib directly).

Wire format (the "archive stream")

Everything is little-endian. struct format codes are given in parentheses.

┌──────────────────────────────────────────────────────────────────────┐
│ MAGIC        8 bytes  = b"GLYPHIV1"                                    │
│ VERSION      1 byte   (B)  = 2                                         │
│ META_FLAGS   1 byte   (B)  0 = none, 1 = basic                        │
│ REC_COUNT    4 bytes  (I)  number of records that follow               │
│ record * REC_COUNT                                                     │
└──────────────────────────────────────────────────────────────────────┘

Each record is:

┌──────────────────────────────────────────────────────────────────────┐
│ REC_TYPE     1 byte   (B)  0 = file, 1 = explicitly-empty directory    │
│ PATH_LEN     2 bytes  (H)  length of the UTF-8 relpath in bytes        │
│ PATH         PATH_LEN bytes   relpath, UTF-8, "/"-separated (POSIX)    │
│ [BASIC] MODE 2 bytes  (H)  permission bits (mode & 0o7777)             │
│ [BASIC] MTIME 8 bytes (q) signed Unix milliseconds                     │
│ CONTENT_LEN  8 bytes  (Q)  content length in bytes (0 for a dir)       │
│ CONTENT      CONTENT_LEN bytes   raw file bytes (absent for a dir)     │
└──────────────────────────────────────────────────────────────────────┘

Version 1 remains readable. Its header is VERSION + REC_COUNT and every record unconditionally contains the historical 4-byte st_mode and 8-byte float st_mtime fields. Version 2 is the default for new archives; none omits optional metadata fields and basic stores only the ordinary permission bits plus mtime rounded to integer milliseconds.

Because every variable-length field is length-prefixed, the content bytes are never scanned for a delimiter and may contain any byte sequence, including the magic or any framing bytes. Records are emitted in ascending relpath order (Python string sort on the POSIX relpath) so the stream is deterministic for a given tree.

Directories are normally reconstructable from the file relpaths, so only explicitly-empty directories (a directory with no files anywhere beneath it) get their own REC_TYPE == 1 record — this is what lets an empty dir round-trip. Non-empty directories are implied by their files' paths.

Ignore filter

When use_ignore is true (the default, matching the CLI's normal mode), a pathspec.PathSpec (gitignore) is built from .gitignore and .ignore found at the tree root only and from any extra_ignore pattern lines. Paths matching the spec (evaluated relative to the root, with a trailing / for directories so dir/ patterns match) are skipped, and the .git/ directory is always skipped regardless. use_ignore=False disables all ignore filtering (the CLI's --no-ignore).

Limitation (v1): only root-level .gitignore / .ignore are read. Nested .gitignore files deeper in the tree are not honored — out of scope for v1.

Compression stage

Compression is a separate stage from serialization. The whole archive_tree output is compressed by the caller, never per file. compress / decompress support "none" (passthrough), "gzip" (stdlib), and "zstd" (via the optional zstandard dependency, imported lazily and only when requested). Which method was used is recorded in the page header by :mod:glyphive.layout, not here.

gzip/deflate has no incremental validation — a single early error invalidates everything downstream — so integrity is provided around the compressed stream by the codec's per-line CRC and layout's per-page hashes, not by dropping compression or adding per-file checks.

FORMAT_VERSION = 2 module-attribute

MAGIC = b'GLYPHIV1' module-attribute

METADATA_PROFILES = ('none', 'basic') module-attribute

PathLike = _ty.Union[str, 'Path', _ty.Any] module-attribute

REC_EMPTY_DIR = 1 module-attribute

REC_FILE = 0 module-attribute

V1_FORMAT_VERSION = 1 module-attribute

__all__ = ['MAGIC', 'FORMAT_VERSION', 'V1_FORMAT_VERSION', 'METADATA_PROFILES', 'REC_FILE', 'REC_EMPTY_DIR', 'Record', 'ArchiveMetadata', 'RecordHeader', 'RecordChunk', 'stream_metadata', 'write_archive', 'archive_tree', 'list_paths', 'iter_records', 'iter_record_events'] module-attribute

ArchiveMetadata

Bases: NamedTuple

Metadata parsed from an archive stream header.

metadata instance-attribute

version instance-attribute

Record

Bases: NamedTuple

One parsed entry from an archive stream.

mode and mtime are zero for version 2 metadata='none' records. Version 1 records retain their historical metadata values. content is b"" for an empty-directory record (type == REC_EMPTY_DIR).

content instance-attribute

mode instance-attribute

mtime instance-attribute

path instance-attribute

type instance-attribute

RecordChunk

Bases: NamedTuple

A bounded piece of the content belonging to the preceding header.

data instance-attribute

RecordHeader

Bases: NamedTuple

Metadata preceding one record's streamed content.

content_length instance-attribute

mode instance-attribute

mtime instance-attribute

path instance-attribute

type instance-attribute

archive_tree(root, *, use_ignore=True, extra_ignore=None, metadata='none')

Serialize the tree at root into one deterministic archive-stream bytes.

root may be a pathlib_next.Path, a str, or any os.PathLike (coerced). The tree is walked with root.walk(); the ignore filter and .git pruning follow the module docstring. Output is byte-for-byte deterministic for a given tree (records sorted by relpath).

metadata is "none" by default and may be "basic" to capture ordinary permission bits and mtime. Compression is not applied here — the caller compresses the whole result via :func:compress.

Source code in src/glyphive/archive.py

def archive_tree(
    root: PathLike,
    *,
    use_ignore: bool = True,
    extra_ignore: _ty.Optional[_ty.Sequence[str]] = None,
    metadata: str = "none",
) -> bytes:
    """Serialize the tree at ``root`` into one deterministic archive-stream ``bytes``.

    ``root`` may be a ``pathlib_next.Path``, a ``str``, or any ``os.PathLike``
    (coerced). The tree is walked with ``root.walk()``; the ignore filter and
    ``.git`` pruning follow the module docstring. Output is byte-for-byte
    deterministic for a given tree (records sorted by relpath).

    ``metadata`` is ``"none"`` by default and may be ``"basic"`` to capture
    ordinary permission bits and mtime. Compression is *not* applied here — the
    caller compresses the whole result via :func:`compress`.
    """
    sink = io.BytesIO()
    write_archive(
        root,
        sink,
        use_ignore=use_ignore,
        extra_ignore=extra_ignore,
        metadata=metadata,
    )
    return sink.getvalue()

iter_record_events(source, *, chunk_size=1024 * 1024, max_content_bytes=None)

Yield record headers and bounded content chunks from a binary source.

A :class:RecordHeader is followed by zero or more :class:RecordChunk events totaling exactly content_length. max_content_bytes rejects a declared per-record size before payload bytes are consumed.

Source code in src/glyphive/archive.py

def iter_record_events(
    source: _ty.BinaryIO,
    *,
    chunk_size: int = 1024 * 1024,
    max_content_bytes: _ty.Optional[int] = None,
) -> _ty.Iterator[_ty.Union[RecordHeader, RecordChunk]]:
    """Yield record headers and bounded content chunks from a binary source.

    A :class:`RecordHeader` is followed by zero or more :class:`RecordChunk`
    events totaling exactly ``content_length``. ``max_content_bytes`` rejects a
    declared per-record size before payload bytes are consumed.
    """
    chunk_size = _validate_chunk_size(chunk_size)
    if max_content_bytes is not None and max_content_bytes < 0:
        raise ValueError("max_content_bytes must be non-negative")
    version, profile, rec_count = _read_stream_header(source)
    for index in range(rec_count):
        prefix = _read_exact(source, _REC_PREFIX.size, f"record {index} (prefix)")
        rec_type, path_len = _REC_PREFIX.unpack(prefix)
        if rec_type not in (REC_FILE, REC_EMPTY_DIR):
            raise ValueError(f"unknown record type {rec_type} at index {index}")
        try:
            relposix = _read_exact(source, path_len, f"record {index} (path)").decode("utf-8")
        except UnicodeDecodeError as exc:
            raise ValueError(f"record {index} path is not valid UTF-8") from exc
        if version == V1_FORMAT_VERSION:
            mode, mtime, content_len = _REC_META_V1.unpack(
                _read_exact(source, _REC_META_V1.size, f"record {index} (metadata)")
            )
        elif profile == "basic":
            mode, millis, content_len = _REC_META_BASIC.unpack(
                _read_exact(source, _REC_META_BASIC.size, f"record {index} (metadata)")
            )
            mtime = millis / 1000.0
        else:
            mode, mtime = 0, 0.0
            content_len = _CONTENT_LEN.unpack(
                _read_exact(source, _CONTENT_LEN.size, f"record {index} (content length)")
            )[0]
        if rec_type == REC_EMPTY_DIR and content_len:
            raise ValueError(
                f"empty-directory record {index} has nonzero content length"
            )
        if max_content_bytes is not None and content_len > max_content_bytes:
            raise ValueError(
                f"record {index} content length {content_len} exceeds limit "
                f"{max_content_bytes}"
            )
        yield RecordHeader(rec_type, relposix, mode, mtime, content_len)
        remaining = content_len
        while remaining:
            chunk = source.read(min(chunk_size, remaining))
            if not chunk:
                raise ValueError(f"truncated record {index} (content)")
            remaining -= len(chunk)
            yield RecordChunk(chunk)
    if source.read(1):
        raise ValueError("archive stream has trailing byte(s) after records")

iter_records(data)

Parse an archive stream, yielding :class:Record in stream order.

Raises :class:ValueError on a truncated stream, unknown version/profile, unknown record type, or trailing bytes. The full stream is validated by restore before any records are materialized on disk.

Source code in src/glyphive/archive.py

def iter_records(data: bytes) -> _ty.Iterator[Record]:
    """Parse an archive stream, yielding :class:`Record` in stream order.

    Raises :class:`ValueError` on a truncated stream, unknown version/profile,
    unknown record type, or trailing bytes. The full stream is validated by
    restore before any records are materialized on disk.
    """
    header = None
    content = []
    for event in iter_record_events(io.BytesIO(data)):
        if isinstance(event, RecordHeader):
            if header is not None:
                yield Record(header.type, header.path, header.mode, header.mtime, b"".join(content))
            header, content = event, []
        else:
            content.append(event.data)
    if header is not None:
        yield Record(header.type, header.path, header.mode, header.mtime, b"".join(content))

list_paths(root, *, use_ignore=True, extra_ignore=None)

Return the sorted POSIX relpaths that :func:archive_tree would archive.

Reuses the same walk + ignore logic, so the manifest :mod:glyphive.layout prints in the page header exactly matches the archived records. Empty-directory entries appear with a trailing / to distinguish them from files.

Source code in src/glyphive/archive.py

def list_paths(
    root: PathLike,
    *,
    use_ignore: bool = True,
    extra_ignore: _ty.Optional[_ty.Sequence[str]] = None,
) -> list[str]:
    """Return the sorted POSIX relpaths that :func:`archive_tree` would archive.

    Reuses the same walk + ignore logic, so the manifest :mod:`glyphive.layout` prints in the
    page header exactly matches the archived records. Empty-directory entries
    appear with a trailing ``/`` to distinguish them from files.
    """
    root = _validate_root(root)
    paths: list[str] = []
    for relposix, _path, is_empty_dir in _walk_entries(
        root, use_ignore=use_ignore, extra_ignore=extra_ignore
    ):
        paths.append(relposix + "/" if is_empty_dir else relposix)
    paths.sort()
    return paths

stream_metadata(data)

Parse and return the archive version and explicit metadata profile.

Version 1 is reported as metadata='basic' because its historical wire format always carried mode and mtime fields.

Source code in src/glyphive/archive.py

def stream_metadata(data: bytes) -> ArchiveMetadata:
    """Parse and return the archive version and explicit metadata profile.

    Version 1 is reported as ``metadata='basic'`` because its historical wire
    format always carried mode and mtime fields.
    """
    _mv, _off, version, profile, _count = _parse_stream_header(data)
    return ArchiveMetadata(version, profile)

write_archive(root, sink, *, use_ignore=True, extra_ignore=None, metadata='none', chunk_size=1024 * 1024)

Serialize root to a binary sink without buffering file contents.

The sorted manifest is retained in memory, but payload bytes are copied in chunks. File length is captured before the header is emitted; a concurrent truncation or growth raises instead of producing a malformed archive.

Source code in src/glyphive/archive.py

def write_archive(
    root: PathLike,
    sink: _ty.BinaryIO,
    *,
    use_ignore: bool = True,
    extra_ignore: _ty.Optional[_ty.Sequence[str]] = None,
    metadata: str = "none",
    chunk_size: int = 1024 * 1024,
) -> None:
    """Serialize ``root`` to a binary sink without buffering file contents.

    The sorted manifest is retained in memory, but payload bytes are copied in
    chunks. File length is captured before the header is emitted; a concurrent
    truncation or growth raises instead of producing a malformed archive.
    """
    metadata = _validate_metadata(metadata)
    chunk_size = _validate_chunk_size(chunk_size)
    root = _validate_root(root)
    entries = sorted(
        _walk_entries(root, use_ignore=use_ignore, extra_ignore=extra_ignore),
        key=lambda item: item[0],
    )
    manifest = []
    for relposix, path, is_empty_dir in entries:
        mode, mtime, length = 0, 0.0, 0
        if not is_empty_dir or metadata == "basic":
            try:
                stat = path.stat()
            except OSError:
                if not is_empty_dir:
                    raise
            else:
                if metadata == "basic":
                    mode, mtime = stat.st_mode, stat.st_mtime
                if not is_empty_dir:
                    length = stat.st_size
        manifest.append((relposix, path, is_empty_dir, mode, mtime, length))

    sink.write(MAGIC)
    sink.write(_HEADER_V2.pack(FORMAT_VERSION, _METADATA_FLAGS[metadata], len(manifest)))
    for relposix, path, is_empty_dir, mode, mtime, length in manifest:
        path_bytes = relposix.encode("utf-8")
        sink.write(_REC_PREFIX.pack(REC_EMPTY_DIR if is_empty_dir else REC_FILE, len(path_bytes)))
        sink.write(path_bytes)
        if metadata == "basic":
            try:
                millis = int(round(float(mtime) * 1000.0))
            except (OverflowError, TypeError, ValueError) as exc:
                raise ValueError(
                    f"mtime for {relposix!r} cannot be represented as milliseconds"
                ) from exc
            if not -(1 << 63) <= millis <= (1 << 63) - 1:
                raise ValueError(
                    f"mtime for {relposix!r} is outside the signed millisecond range"
                )
            sink.write(_REC_META_BASIC.pack(int(mode) & 0o7777, millis, length))
        else:
            sink.write(_CONTENT_LEN.pack(length))
        if is_empty_dir:
            continue
        copied = 0
        with path.open("rb") as source:
            while copied < length:
                chunk = source.read(min(chunk_size, length - copied))
                if not chunk:
                    raise ValueError(
                        f"file {relposix!r} was truncated while being archived "
                        f"(expected {length} bytes, read {copied})"
                    )
                sink.write(chunk)
                copied += len(chunk)
            if source.read(1):
                raise ValueError(
                    f"file {relposix!r} grew while being archived "
                    f"(expected {length} bytes)"
                )