Dump formats (dbdump)

The packaging-manifest format registry and its built-in formats. A third-party format subclasses PerEntryFormat or MultiArtifactFormat with its own NAME and advertises it in the pkgforge.dump_formats entry-point group; see Adding a format. A format that writes files its user edits sets DumpFormat.SKELETON (written with dump(..., skeleton=True)) or DumpFormat.KEEP_EXISTING; such a file is kept, with a warning, unless force=True. The checks every format must pass are in Contract suite.

pkgforge.exceptions.DumpError

Bases: PkgForgeError, ValueError

A DB entry cannot be rendered in the chosen format.

Raised for a path the target format's own tooling cannot represent -- a control character, or (rpm only) a %. Caught by :func:pkgforge.cli.main's error boundary like any :class:~pkgforge.exceptions.PkgForgeError: one stderr line, exit 1. Nothing is written to OUTPUT first: every entry is rendered before OUTPUT is opened, so this leaves no partial file.

pkgforge.exceptions.InvalidEntryError

Bases: UsageError, DumpError

A DB entry is malformed for every format: a key that is not an absolute normalized path, an unknown type, a mode/owner/group outside the recorded grammar, an unknown kind or one that does not fit the type, or caps that are malformed or on anything but a regular file that is not a ghost.

Raised by :func:_validate_entries before any format renders, so it leaves no output behind. A :class:~pkgforge.exceptions.UsageError (exit 2): the DB, not the format, holds the bad value.

pkgforge.exceptions.UnsupportedOutputError

Bases: UsageError, DumpError, NotImplementedError

OUTPUT is the wrong shape for the chosen format: a file for a multi-artifact format, or an existing directory for a per-entry one.

Subclasses :class:~pkgforge.exceptions.UsageError (exit 2, same as this replaces two separate raises with) and :class:DumpError (the format itself is refusing this OUTPUT, not a bad argument in general) and, on top of both, :class:NotImplementedError (this format cannot write this output shape) -- a valid MRO (measured, py3.14).

pkgforge.dbdump.dump(db, fmt, output=None, *, exclude=(), own=None, config=None, config_path=None, skeleton=False, force=False)

Render the live entries of db with the format fmt.

fmt is a registered format name or alias. Removed entries (None) and entries matched by an exclude statement are dropped. With own (/-rooted prefixes), so is every directory entry that is neither a prefix nor below one; entries of other types are untouched. The rest are sorted by path and validated. config names the packages the entries ship in (default: an empty configuration, one unnamed package) and config_path is the file it was read from, for a format that points at it. With output the result is also written whole: a file for a per-entry format (or a directory holding one list per package), a directory (created if missing) for a multi-artifact one.

With skeleton a format that has one (debian) also writes the files its user edits, into a directory output. A skeleton file, or for rpmspec the spec, that already exists is kept with a warning unless force.

Returns the rendered bytes (a bytes value, or {filename: bytes}). Raises :class:~pkgforge.exceptions.UsageError for an unknown format or exclude statement, an entry of a package config does not declare, a format that needs [package] name or other keys, skeleton with a format that has none or without a directory output, or force where nothing is a skeleton; :class:~pkgforge.exceptions.UnsupportedOutputError for an output of the wrong shape (checked before anything is rendered), :class:~pkgforge.exceptions.InvalidEntryError for an entry no format can render and :class:~pkgforge.exceptions.DumpError for one this format cannot.

Source code in src/pkgforge/dbdump/__init__.py

def dump(
    db: typing.Mapping[str, typing.Optional[FileEntry]],
    fmt: str,
    output: typing.Optional[StrPath] = None,
    *,
    exclude: typing.Sequence[str] = (),
    own: typing.Optional[typing.Sequence[str]] = None,
    config: typing.Optional[ProjectConfig] = None,
    config_path: typing.Optional[StrPath] = None,
    skeleton: bool = False,
    force: bool = False,
) -> Rendered:
    """Render the live entries of ``db`` with the format ``fmt``.

    ``fmt`` is a registered format name or alias. Removed entries (``None``)
    and entries matched by an ``exclude`` statement are dropped. With ``own``
    (``/``-rooted prefixes), so is every directory entry that is neither a
    prefix nor below one; entries of other types are untouched. The rest are
    sorted by path and validated. ``config`` names the packages the entries
    ship in (default: an empty configuration, one unnamed package) and
    ``config_path`` is the file it was read from, for a format that points at
    it. With ``output`` the result is also written whole: a file for a
    per-entry format (or a directory holding one list per package), a
    directory (created if missing) for a multi-artifact one.

    With ``skeleton`` a format that has one (``debian``) also writes the files
    its user edits, into a directory ``output``. A skeleton file, or for
    ``rpmspec`` the spec, that already exists is kept with a warning unless
    ``force``.

    Returns the rendered bytes (a ``bytes`` value, or ``{filename: bytes}``).
    Raises :class:`~pkgforge.exceptions.UsageError` for an unknown format or
    exclude statement, an entry of a package ``config`` does not declare, a
    format that needs ``[package] name`` or other keys, ``skeleton`` with a
    format that has none or without a directory ``output``, or ``force`` where
    nothing is a skeleton;
    :class:`~pkgforge.exceptions.UnsupportedOutputError`
    for an ``output`` of the wrong shape (checked before anything is
    rendered), :class:`~pkgforge.exceptions.InvalidEntryError` for an entry no
    format can render and :class:`~pkgforge.exceptions.DumpError` for one this
    format cannot.
    """
    instance, target = _prepare(fmt, output, skeleton=skeleton, force=force)
    entries = _surviving_entries(db, exclude, own)
    _validate_entries(entries)
    context = DumpContext(
        config=config if config is not None else ProjectConfig(),
        config_path=None if config_path is None else Path(os.fspath(config_path)),
        output=target,
    )
    rendered = instance.render(entries, context)
    if target is not None:
        for written in instance.write(rendered, target, force=force):
            _LOGGER.info("Wrote %s", written)
    return rendered

pkgforge.dbdump.DumpContext(config=ProjectConfig(), config_path=None, output=None) dataclass

What a format renders against besides the entries.

config is the project configuration (its [package] and [[packages]] name the packages), config_path the file it was read from when known, and output the path being written (None for stdout or when nothing is written).

config = dataclasses.field(default_factory=ProjectConfig) class-attribute instance-attribute

config_path = None class-attribute instance-attribute

output = None class-attribute instance-attribute

pkgforge.dbdump.DumpFormat(*, skeleton=False)

Bases: Registered, ABC

A packaging-manifest format, selected by dbdump -f NAME.

A subclass registers by declaring its own NAME (plus, optionally, ALIASES); see :class:pkgforge._registry.Registered. Subclass :class:PerEntryFormat or :class:MultiArtifactFormat -- they supply :meth:check_output and :meth:write for their respective output shape -- or this class, supplying both, for a format with another shape.

skeleton (dbdump --skeleton) asks a format with :attr:SKELETON to write the files its user edits as well.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

ALIASES = () class-attribute

KEEP_EXISTING = False class-attribute

NAME = '' class-attribute

SKELETON = False class-attribute

skeleton = skeleton instance-attribute

check_output(output) abstractmethod

Raise :class:UnsupportedOutputError if output is the wrong shape for this format.

Source code in src/pkgforge/dbdump/__init__.py

@abc.abstractmethod
def check_output(self, output: Path) -> None:
    """Raise :class:`UnsupportedOutputError` if ``output`` is the wrong
    shape for this format."""

render(entries, context) abstractmethod

Render the (already validated) entries to bytes, writing nothing.

Source code in src/pkgforge/dbdump/__init__.py

@abc.abstractmethod
def render(self, entries: FileEntries, context: DumpContext) -> Rendered:
    """Render the (already validated) ``entries`` to bytes, writing nothing."""

write(rendered, output, *, force=False) abstractmethod

Write rendered to output, whole or not at all; return the files written. A skeleton file that exists is kept (and not returned) unless force.

Source code in src/pkgforge/dbdump/__init__.py

@abc.abstractmethod
def write(
    self, rendered: Rendered, output: Path, *, force: bool = False
) -> typing.List[Path]:
    """Write ``rendered`` to ``output``, whole or not at all; return the
    files written. A skeleton file that exists is kept (and not returned)
    unless ``force``."""

pkgforge.dbdump.PerEntryFormat(*, skeleton=False)

Bases: DumpFormat

A format that renders one line per DB entry.

Given a file (or stdout) it renders the entries of the one package that has any; more than one package is a :class:~pkgforge.exceptions.UsageError. Given an existing directory it renders :attr:PACKAGE_FILE for every declared package, so a package with no entry still gets its (possibly empty) list.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

EMPTY = b'' class-attribute

PACKAGE_FILE = 'files-{package}.txt' class-attribute

check_output(output)

Any path fits: a file, or a directory for one list per package.

Source code in src/pkgforge/dbdump/__init__.py

def check_output(self, output: Path) -> None:
    """Any path fits: a file, or a directory for one list per package."""

render(entries, context)

Source code in src/pkgforge/dbdump/__init__.py

def render(
    self, entries: FileEntries, context: DumpContext
) -> typing.Union[bytes, typing.Dict[str, bytes]]:
    groups = _packages(entries, context.config)
    if context.output is not None and context.output.is_dir():
        if context.config.package is None:
            raise UsageError(
                f"writing {self.PACKAGE_FILE.replace('{package}', 'PACKAGE')} "
                "needs [package] name"
            )
        return {
            self.PACKAGE_FILE.format(package=name): (
                self._render_lines(group) if group else self.EMPTY
            )
            for name, group in groups
        }
    used = [(name, group) for name, group in groups if group]
    if len(used) > 1:
        names = ", ".join(name for name, _group in used)
        raise UsageError(
            f"dbdump: the DB holds entries of {len(used)} packages ({names}): "
            f"give a directory OUTPUT for one "
            f"{self.PACKAGE_FILE.replace('{package}', 'PACKAGE')} each"
        )
    return self._render_lines(used[0][1]) if used else self._render_lines([])

render_entry(path, entry) abstractmethod

Render one DB entry to its line of bytes.

Source code in src/pkgforge/dbdump/__init__.py

@abc.abstractmethod
def render_entry(self, path: str, entry: FileEntry) -> bytes:
    """Render one DB entry to its line of bytes."""

write(rendered, output, *, force=False)

Source code in src/pkgforge/dbdump/__init__.py

def write(
    self, rendered: Rendered, output: Path, *, force: bool = False
) -> typing.List[Path]:
    if isinstance(rendered, bytes):
        _write_files({output: rendered})
        return [output]
    files = {output / name: data for name, data in rendered.items()}
    output.mkdir(parents=True, exist_ok=True)
    _write_files(files)
    return list(files)

pkgforge.dbdump.MultiArtifactFormat(*, skeleton=False)

Bases: DumpFormat

A format that renders several named artifacts into a directory.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

check_output(output)

Source code in src/pkgforge/dbdump/__init__.py

def check_output(self, output: Path) -> None:
    if output.exists() and not output.is_dir():
        raise UnsupportedOutputError(
            f"{self.NAME} writes several files; OUTPUT must be a "
            f"directory or '-', got existing file {output}"
        )

render(entries, context) abstractmethod

Render every artifact this format produces: {filename: bytes}.

Source code in src/pkgforge/dbdump/__init__.py

@abc.abstractmethod
def render(
    self, entries: FileEntries, context: DumpContext
) -> typing.Dict[str, bytes]:
    """Render every artifact this format produces: ``{filename: bytes}``."""

write(rendered, output, *, force=False)

Source code in src/pkgforge/dbdump/__init__.py

def write(
    self, rendered: Rendered, output: Path, *, force: bool = False
) -> typing.List[Path]:
    artifacts = typing.cast(typing.Dict[str, bytes], rendered)
    files = {output / name: data for name, data in artifacts.items()}
    for directory in {path.parent for path in files}:
        directory.mkdir(parents=True, exist_ok=True)
    _write_files(files)
    return list(files)

Built-in formats

pkgforge.dbdump.RPMSpecFiles(*, skeleton=False)

Bases: PerEntryFormat

RPM %files lines: %attr(mode,owner,group) "path", a %dir prefix for directories, %caps(CAPS) for an entry with file capabilities, and the entry kind's directive first (%config, %config(noreplace), %doc, %license or %ghost; none for file).

The path is quoted for rpm's %files -f parser (targets rpm 4.19+): a backslash and a double quote are escaped, and the whole line is written as UTF-8 with surrogateescape (a non-UTF-8 name round-trips its original bytes). Glob characters are never escaped -- rpm's own quoted-string globbing already matches only the literal name. A % anywhere in the path, or a control character (including a tab), raises :class:~pkgforge.exceptions.DumpError. Below rpm 4.19, use :class:RPMSpecFilesPre419 instead -- this quoting is not safe there.

Into a directory it writes files-PACKAGE.txt for every declared package (%files -n PACKAGE -f .../files-PACKAGE.txt); a package with no entry gets one empty line, which rpm builds as an empty package.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

ALIASES = ('rpm',) class-attribute instance-attribute

EMPTY = b'\n' class-attribute instance-attribute

NAME = 'rpmspecfiles' class-attribute instance-attribute

render_entry(path, entry)

Source code in src/pkgforge/dbdump/_rpm.py

def render_entry(self, path: str, entry: FileEntry) -> bytes:
    return self._render_lines([(path, entry)])

pkgforge.dbdump.RPMSpecFilesPre419(*, skeleton=False)

Bases: RPMSpecFiles

RPM %files lines for rpm older than 4.19 (measured against real rpmbuild runs on 4.14.3, 4.16.1 and 4.18.2).

Inherits :class:RPMSpecFiles's %attr/%dir/kind/caps rendering and batch-then-per-entry validation, overriding only the escaping/rejection hooks: the path is written completely unquoted, and a path containing a space, a glob character (* ? [ ] { }), % (in any form), or a byte that is not valid UTF-8 raises :class:~pkgforge.exceptions.DumpError instead of risking a silently wrong or overmatched package. Use :class:RPMSpecFiles instead when the rpm that builds the package is 4.19 or newer.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

ALIASES = ('rpm-pre419',) class-attribute instance-attribute

NAME = 'rpmspecfiles-pre419' class-attribute instance-attribute

pkgforge.dbdump.RPMSpec(*, skeleton=False)

Bases: DumpFormat

The RPM spec for the packages pkgforge.cfg declares: one Name preamble ([package]), a %package -n block per [[packages]] element, a %install that runs pkgforge stage --init and writes one %files list per package, and one %files -f per package. A %post creates the tmpfiles.d paths of the package that ships /usr/lib/tmpfiles.d/NAME.conf.

A package that records a systemd unit directly in /usr/lib/systemd/system gets %systemd_post, %systemd_preun and %systemd_postun_with_restart for its units. They need systemd-rpm-macros (or EL8's systemd) when the package is built, and expand to nothing without them.

Needs [package] name, version, summary and license, and a summary for each [[packages]] element; an any subpackage of an all main package is refused. Every configuration text has its % doubled, so no macro is read from the file, and a %description line whose first non-blank character is % or # is refused, since rpm reads it as a section, a conditional or a comment. The spec holds no absolute path: build it with rpmbuild -bb --define "pkgforge_config PATH" NAME.spec, PATH being pkgforge.cfg, and pkgforge on PATH.

The package is built from the recorded files unchanged (no brp scripts, no debuginfo). Writes a file or -; a spec that exists is kept with a warning unless force, since it is meant to be edited. rpm 4.19+ creates the accounts from the shipped sysusers.d file; use :class:RPMSpecPre419 for older rpm.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

KEEP_EXISTING = True class-attribute instance-attribute

NAME = 'rpmspec' class-attribute instance-attribute

check_output(output)

Source code in src/pkgforge/dbdump/_rpmspec.py

def check_output(self, output: Path) -> None:
    if output.is_dir():
        raise UnsupportedOutputError(
            f"{self.NAME} writes one spec file; OUTPUT must be a file or "
            f"'-', got directory {output}"
        )

render(entries, context)

Source code in src/pkgforge/dbdump/_rpmspec.py

def render(self, entries: FileEntries, context: DumpContext) -> bytes:
    install = [
        "export PKGFORGE_ROOT=%{buildroot} PKGFORGE_DB=%{_builddir}/pkgforge.jsonl\n",
        'pkgforge -q --config "%{pkgforge_config}" stage --init\n'
        "mkdir -p %{_builddir}/pkgforge-files\n"
        f'pkgforge -q --config "%{{pkgforge_config}}" dbdump -f {self._LISTS} '
        "%{_builddir}/pkgforge-files\n",
    ]
    spec = _spec(
        context.config,
        entries,
        fmt=self.NAME,
        sysusers_pre=self._SYSUSERS_PRE,
        install=install,
        guard=True,
    )
    return spec.encode("utf-8")

write(rendered, output, *, force=False)

Source code in src/pkgforge/dbdump/_rpmspec.py

def write(
    self, rendered: Rendered, output: Path, *, force: bool = False
) -> typing.List[Path]:
    files = _keep_existing({output: typing.cast(bytes, rendered)}, force)
    _write_files(files)
    return list(files)

pkgforge.dbdump.RPMSpecPre419(*, skeleton=False)

Bases: RPMSpec

:class:RPMSpec for rpm older than 4.19: the %files lists are rpmspecfiles-pre419's, and a %pre of the package that ships the sysusers.d file creates the accounts itself.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

NAME = 'rpmspec-pre419' class-attribute instance-attribute

pkgforge.dbdump.DebianFiles(*, skeleton=False)

Bases: MultiArtifactFormat

Debian packaging artifacts built from surviving DB entries, named for the packages pkgforge.cfg declares ([package] and [[packages]]; [package] name is required). For each package P:

• P.install -- dh_install-style lines <src> <dest-dir> (the source is the build-root-relative path, debhelper-escaped; the destination is the entry's parent directory, escaped for $/space only), one per non-directory entry;
• P.dirs -- dh_installdirs-style lines (dest-escaped, one per directory entry, in every case -- an already-populated directory is a harmless duplicate mkdir), so a directory recorded empty (install -d) still reaches the package; never routed through install, which would re-copy any children --exclude dropped;
• P.conffiles -- the config and config-noreplace entries outside /etc/ (debhelper adds those under /etc/ itself), one absolute path per line; whitespace in such a path raises :class:~pkgforge.exceptions.DumpError;
• P.postinst -- a maintainer script with #DEBHELPER#; on configure it runs, for each entry owned by an account the package creates (an owner or group outside the base-passwd set), a guarded dpkg-statoverride --update --add and then, for each entry with caps, a setcap line;
• P.postrm -- the same header; on purge it removes those overrides;
• P.substvars -- pkgforge:Depends=libcap2-bin when P has an entry with caps, else an empty value, for Depends: ${pkgforge:Depends}. dh_prep deletes this file, so it only takes effect when the dump runs after dh_prep (override_dh_auto_install).

And once for the source:

• not-installed -- the ghost entries of every package that are not directories, sorted, in install's source escaping, so dh_missing --fail-missing accepts a ghost file left in the staging directory (debhelper reads only the source-wide file);
• fixperms -- a POSIX sh script, always written, that applies every pinned mode/owner/group directly to the package directories: run it from override_dh_fixperms (after dh_fixperms, which would otherwise strip the very bits this pins) as sh debian/fixperms debian. Per pinned entry: one chown owner:group (or chown/chgrp when only one is pinned), then chmod (chown before chmod, since chown clears a regular file's setuid/setgid bit even to the same owner); a symlink gets chown -h/chgrp -h and never a chmod (POSIX chmod has no -h and would follow the link to a target outside the package tree). A path that dh_compress renamed to <path>.gz is found there; one that is gone (dh_usrlocal) is reported on stderr and skipped. With no pinned entry, the script is just its own header.

With skeleton (dbdump --skeleton) it also writes the files you edit: control (a stanza per package, Depends listing ${pkgforge:Depends}), changelog (dated SOURCE_DATE_EPOCH, else the current time), rules (the debian/tmp recipe, naming the project file, mode 755) and source/format. They need a directory OUTPUT, the project file's path and [package] version, summary and maintainer; one that exists is kept with a warning unless force. A ${NAME} in the summary, description, homepage or maintainer of any package is :class:~pkgforge.exceptions.UsageError: dpkg-gencontrol replaces it and nothing escapes it.

Every artifact is written for every declared package, so no stale one survives. Writing refuses, before it writes anything: an existing install, dirs, not-installed, postinst, postrm or fixperms without the marker line this format puts first (a hand-written file); a nameless install, dirs, conffiles, postinst or postrm in the directory, which debhelper ignores once the package-named file exists; and an existing P.conffiles that lists a path the new one lacks.

The build needs dh $@ --with installsysusers (debhelper 13.10+) or Build-Depends: dh-sequence-installsysusers so the accounts exist before postinst applies the overrides.

Ghost entries are left out of install, dirs and fixperms, and so are entries owned by an account the package creates: dpkg unpacks them before postinst can create the account, so fixperms leaves them root-owned and the override applies them. Such a symlink, or one with no recorded mode, raises :class:~pkgforge.exceptions.DumpError.

A path with a control character raises :class:~pkgforge.exceptions.DumpError while the artifacts are built in memory; they are written only after :meth:render returns, so nothing partial is left behind.

Source code in src/pkgforge/dbdump/__init__.py

def __init__(self, *, skeleton: bool = False) -> None:
    if skeleton and not self.SKELETON:
        raise UsageError(f"{self.NAME} has no skeleton")
    self.skeleton = skeleton

ALIASES = ('deb',) class-attribute instance-attribute

NAME = 'debian' class-attribute instance-attribute

SKELETON = True class-attribute instance-attribute

render(entries, context)

Source code in src/pkgforge/dbdump/_debian.py

def render(
    self, entries: FileEntries, context: DumpContext
) -> typing.Dict[str, bytes]:
    if context.config.package is None:
        raise UsageError(
            "dbdump -f debian names its files after [package] name: "
            "set it in pkgforge.cfg"
        )
    files = self._render_package_files(entries, context)
    if self.skeleton:
        files.update(_skeleton(context))
    return files

write(rendered, output, *, force=False)

As for any multi-artifact format, after refusing a directory whose existing files this dump would replace or shadow (see the class). A skeleton file that exists is kept unless force; rules is made executable.

Source code in src/pkgforge/dbdump/_debian.py

def write(
    self, rendered: Rendered, output: Path, *, force: bool = False
) -> typing.List[Path]:
    """As for any multi-artifact format, after refusing a directory whose
    existing files this dump would replace or shadow (see the class). A
    skeleton file that exists is kept unless ``force``; ``rules`` is made
    executable."""
    artifacts = typing.cast(typing.Dict[str, bytes], rendered)
    _check_existing(artifacts, output)
    files = {
        output / name: data
        for name, data in artifacts.items()
        if name not in _SKELETON_FILES
    }
    files.update(
        _keep_existing(
            {
                output / name: data
                for name, data in artifacts.items()
                if name in _SKELETON_FILES
            },
            force,
        )
    )
    for directory in {path.parent for path in files}:
        directory.mkdir(parents=True, exist_ok=True)
    _write_files(files)
    if output / "rules" in files:
        (output / "rules").chmod(0o755)
    return list(files)